# MFM-ETP: An Embodied Task Planning Benchmark for Multi-modal Foundation
Models

Min Zhang 1, Jianye Hao1 , Xian Fu1, Peilong Han1, Hao Zhang1,
Lei Shi1, Hongyao Tang2, Yan Zheng1
1 College of Intelligence and Computing, Tianjin University
2 Montreal Institute of Learning Algorithms (MILA)
Corresponding author: Jianye Hao<EMAIL_ADDRESS>
# MFE-ETP: A Comprehensive Evaluation Benchmark for Multi-modal Foundation
Models on Embodied Task Planning

Min Zhang 1, Jianye Hao1 , Xian Fu1, Peilong Han1, Hao Zhang1,
Lei Shi1, Hongyao Tang2, Yan Zheng1
1 College of Intelligence and Computing, Tianjin University
2 Montreal Institute of Learning Algorithms (MILA)
Corresponding author: Jianye Hao<EMAIL_ADDRESS>
###### Abstract

In recent years, Multi-modal Foundation Models (MFMs) and Embodied Artificial
Intelligence (EAI) have been advancing side by side at an unprecedented pace.
The integration of the two has garnered significant attention from the AI
research community. In this work, we attempt to provide an in-depth and
comprehensive evaluation of the performance of MFM s on embodied task
planning, aiming to shed light on their capabilities and limitations in this
domain. To this end, based on the characteristics of embodied task planning,
we first develop a systematic evaluation framework, which encapsulates four
crucial capabilities of MFMs: object understanding, spatio-temporal
perception, task understanding, and embodied reasoning. Following this, we
propose a new benchmark, named MFE-ETP, characterized its complex and variable
task scenarios, typical yet diverse task types, task instances of varying
difficulties, and rich test case types ranging from multiple embodied question
answering to embodied task reasoning. Finally, we offer a simple and easy-to-
use automatic evaluation platform that enables the automated testing of
multiple MFMs on the proposed benchmark. Using the benchmark and evaluation
platform, we evaluated several state-of-the-art MFMs and found that they
significantly lag behind human-level performance. The MFE-ETP is a high-
quality, large-scale, and challenging benchmark relevant to real-world tasks.

##  1 Introduction

The impressive progress of Multi-modal Foundation Models (MFMs) [1, 2, 3, 4,
5, 6, 7] has sparked a surge of interest in promoting the application of MFMs
in Embodied Artificial Intelligence (EAI). Currently, the mainstream
approaches can be divided into two branches, one of which is to train specific
models for low-level robotic control based on particular robotics data [8, 9,
10, 11, 12, 13, 14]. The other is to employ off-the-shelf models such as
GPT-4V [4] as a high-level task planner and then use pre-trained skills to
achieve all sub-task goals from high-level task planner, adapting flexibly to
various robotics scenarios in a zero-shot manner [15, 16, 17, 18, 19, 20]. The
main difference between these approaches lies in the design details of the
task planning pipeline, e.g., whether it integrates an affordance analyzer
[15], or incorporates perceptual information and visual feedback [16], etc.
Rooted in data-driven algorithms and specific scenarios, the former only
characterizes abstract features of limited acquisition data, often failing to
derive nuanced scenario understanding and effective causal reasoning for
robotics manipulation. In contrast, the latter approach avoids the high costs
of data collection and leverages the remarkable reasoning and generalization
capabilities of MFMs to enhance model generalization in robotics applications.
However, blindly applying off-the-shelf MFMs without understanding their
capabilities may prevent these methods from achieving optimal performance.
Actually, since the release of GPT-4V[4], a large number of performance
evaluation reports on MFMs in different domains have been published, such as
commonsense tasks[21, 22, 23], vision tasks [24, 21, 25, 26], autonomous
driving [27, 28], and robotics [16, 17, 29], etc. However, a comprehensive
performance evaluation of MFMs for embodied task planning has not been
thoroughly explored.

To better utilize MFMs for embodied task planning, based on the
characteristics of embodied task planning, we first propose a systematic
evaluation framework, which encapsulates four crucial capabilities: object
understanding, spatio-temporal perception, task understanding, and embodied
reasoning, aiming to answer what factors constrain MFMs to output accurate
task plans. Then, we propose a comprehensive evaluation benchmark (MFE-ETP)
that aligns with this framework to quantitatively and qualitatively access the
performance limitations of MFMs on embodied task planning. In addition, we
develop an automated evaluation platform to facilitate the performance
evaluation of various embodied task planners using MFMs on our proposed
benchmark. Compared to existing evaluation efforts, our work has four key
advantages: ① Our evaluation framework and benchmark are specifically designed
for embodied task planning, ensuring accurate performance evaluation of MFMs
on task planning; ② We provide a diverse set of scenarios and tasks, allowing
for an extensive evaluation of MFMs’ task planning capabilities across
different settings; ③ The capability dimensions we evaluate are more
comprehensive and tightly aligned with embodied task planning, covering a wide
range of test case types from embodied question answering to embodied task
planning; ④ We offer an easy-to-use evaluation platform that facilitates the
automatic evaluation of multiple MFMs on our benchmark.

In summary, the main contributions of our work are as follows:

  * •

We propose a systematic evaluation framework tailored for embodied task
planning for the first time. This framework offers effective guidance for the
improvement of MFMs-based task planners.

  * •

We propose a benchmark, MFE-ETP containing over 1100 high-quality test cases
carefully annotated by human, covering 100 embodied tasks.

  * •

We develop an evaluation platform and evaluate six advanced MFMs. The
evaluation results show that object type recognition and spatial perception
are the main constraints for MFMs to generate correct task planning results.

  * •

We will make this benchmark and platform open source111project website
available at https://mfe-etp.github.io/ to foster future research on embodied
task planning and inspire more focused research directions.

##  2 Evaluation Framework

Figure 1: The overall evaluation framework tailored for embodied task planning
encapsulates four crucial capabilities: Object Understanding, Spatio-Temporal
perception, Task Understanding, and Embodied Reasoning.

In this section, we introduce our evaluation framework designed for embodied
task planning, which consists of four levels of crucial capabilities: object
understanding, spatio-temporal perception, task understanding, and embodied
reasoning. Each capability dimension is further decomposed into several sub-
aspects. The overall framework is shown in Fig.1.

Object Understanding (OU) Object understanding involves recognizing the Type
and Property of objects. The object properties include physical properties
(e.g., color, shape, etc.) and functional properties (e.g., openable,
grabbable, etc.). Accurate recognition of object types and properties is a
fundamental requirement for correct task planning. For example, in the task of
preserving food shown in Fig.1, if the food on the dinner table and its
properties (e.g., the apple and pie that can be grasped) are incorrectly or
incompletely recognized by MFMs, their output task plans may fail to enable
the robot to complete the task. In addition, this could raise safety concerns
due to inappropriate actions against misidentified objects. Tab.4 in Appendix
lists the definitions of each considered property along with representative
objects.

Spatio-Temporal Perception (STP) Building upon the understanding of static
objects, the spatial-temporal perception capability of MFMs is vital for
solving embodied task planning problems. In our proposed framework, for
Spatial perception, we evaluate the ability of MFMs to judge spatial
attributes such as distance, geometry, direction, and relative position
between objects. Our benchmark covers five types of spatial relations:
"Inside", "OnTop", "NextTo", and "Under", with further subdivisions of
"NextTo" into "Front", "Back", "Left", and "Right" to accommodate more tasks.
For Temporal perception, we access MFMs’ ability to recognize the
chronological sequence of task progress, and predict the effects of actions on
the environment. As illustrated in Figure 1, MFMs need to perceive the spatial
positions of the apple and pie from a first-person perspective. This
understanding enables them to plan navigation actions and subsequent steps by
observing task progress and changes in the environment, such as the apple
being placed in the freezer and the freezer being opened. Accurate spatial and
temporal perception enhances MFMs’ effectiveness in task planning.

Task Understanding (TU) Compared to object understanding and spatio-temporal
perception, task understanding is a higher-level and more abstract capability
for MFMs. It requires MFMs to recognize task-relevant objects and understand
the operation knowledge of these objects, the sequence of task steps, and the
conditions for task goal completion. To this end, we evaluate the task
understanding capability of MFMs from the above four aspects, which we name as
Relevant Object, Operation Knowledge, Step Sequence, and Goal, respectively.
For the food preserving task in Fig.1, MFMs first need to accurately select
the food on the dinner table and the appropriate storage location, i.e., the
freezer, from various object types within the field of view. Then, MFMs
determine the operation knowledge (e.g., open/place_inside/close) and the
sequence of steps (e.g., open the freezer first, then put the food in the
freezer, and finally close the freezer) based on the usage of the freezer.
Finally, MFMs need to identify whether the task goal has been achieved. If the
task understanding capability of MFMs is insufficient, it may result in
planning errors, such as improper action functions or sequences of steps.

Embodied Reasoning (ER) Towards more realistic, long-horizon, and complex
embodied tasks based on general descriptions, it is evident that relying
solely on the first three capabilities is insufficient for MFMs.
Indispensably, MFMs need to further integrate and reason based on perceived
object information, spatio-temporal information, and task knowledge to
complete task planning. Thus, we evaluate the end-to-end embodied reasoning
capabilities of MFMs. Specifically, we consider two types of tasks. The first
type involves common typical tasks with general task descriptions and
corresponding visual inputs, such as the task example shown in Fig.1. The task
description provides only basic information about preserving food. For the
second type, we have designed specific tasks tailored to test object
understanding, spatio-temporal perception, and task understanding. These tasks
also include general task descriptions and corresponding visual inputs but
primarily focus on investigating the impact of these capability dimensions on
embodiment reasoning. Taking spatial perception as an example, a customized
food preserving task has its general task description replaced with preserving
the rightmost food on the dinner table. This limited information enables a
more focused evaluation of MFMs’ performance limitations in embodied task
planning under realistic conditions. Tab.5 in Appendix lists the definitions
of each considered action in the embodied reasoning results.

Figure 2: Example questions of MFE-ETP benchmark and illustration of the six
types of task formats.

Figure 3: This figure outlines our evaluation workflow. It involves benchmark
construction, data preparation, model deployment, and metric calculation to
derive the final score.

##  3 Evaluation Benchmark

In this section, we describe in detail our benchmark, MFE-ETP, a dataset
comprising video-derived images and task instructions designed to evaluate
MFMs on embodied task planning. This section introduces the data collection
(Sec.3.1), benchmark structure (Sec.3.2), and automatic evaluation method
(Sec.3.3) in sequence, respectively.

###  3.1 Data Collection

The household environment represents a common scenario for robot-human
interaction, where the diversity and complexity of household tasks can
thoroughly test the robot’s perception, reasoning, planning, and action
capabilities. Therefore, in this paper, we select typical household tasks to
build a benchmark for embodied task planning. Specifically, we collect raw
data from representative household task platforms BEHAVIOR-100 [30] and
VirtualHome [31]. More information about these platforms can be found in
Appendix C.1.

To avoid redundancy of similar tasks while ensuring task diversity, we
abstracted 20 typical household task types from all tasks provided by
BEHAVIOR-100 [30] and VirtualHome [31], and generate 100 household task
instances. To illustrate the typicality and diversity of our benchmark tasks,
we provide a mapping table of 20 typical household task types to the specific
tasks from BEHAVIOR-100[30], VirtualHome[31] and our benchmark in Appendix
C.1. For each household task instance from BEHAVIOR-100 [30], we collect
several frames containing all the initial conditions of the task and visual
frames at different moments as the visual information of the task through the
available task video. For each household task instance from VirtualHome [31],
due to the lack of readily available task videos, we first build the task
program and then use the VirtualHome [31] simulation platform to collect the
visual frames containing all the initial conditions of the task and visual
frames at different moments. Furthermore, based on the visual frames of 100
household tasks and the proposed evaluation framework, we manually annotate
the visual frames and design corresponding task instructions for different
capability dimensions. More details are presented in Sec.3.2.

###  3.2 Benchmark Structure

In order to comprehensively evaluate the performance of MFMs on embodied task
planning, we designed six different task instruction forms based on our
proposed evaluation framework: (1) Open-vocabulary Q&A requires MFMs to
identify and answer questions about object types, based on general or task-
related objects. (2) Single-Choice Q&A evaluates MFMs’ spatial perception
ability and task-related object operation knowledge, requiring MFMs to select
the correct answer from multiple options. (3) Multi-Choice Q&A means that
there may not be a single correct answer to the question, considering that
objects can have multiple properties and there may be multiple reasonable task
step sequences. (4) Sorting Q&A tasks MFMs with sorting visual frames from
different moments based on time or spatial distance to evaluate their spatio-
temporal perception. (5) Yes/No Q&A This is a simple judgment question, mainly
used to determine whether the task goal is achieved and the changes in the
embodied environment. (6) Planning Q&A challenges MFMs to generate task plans
based on a general task description to access MFMs’ end-to-end embodied task
planning performance.

Figure 4: Statistics of MFE-ETP benchmark.

Figure 5: Radar chart of MFMs’ scores on MFE-ETP.

Fig.2 shows example question-and-answer pairs of our benchmark on different
capability dimensions and the six types of task instructions used. In
addition, we have carefully designed a total of 156 task descriptions for
different capability dimensions and combined them with specific task
instructions to generate more diverse task text prompts. Fig.5 reports the
dataset statistics for our benchmark, which contains a total of 1,184
evaluation cases. Obviously, our diverse evaluation cases can meet the needs
of evaluating different capability dimensions of MFMs. Moreover, except for
planning Q&A, other embodied Q&A tasks can be automatically and efficiently
evaluated due to their constrained form (see Appendix C.2 for more details).

###  3.3 Automatic Evaluation

As illustrated in Fig.3, the complete evaluation process of MFMs on our
automatic evaluation platform includes data preparation, model deployment, and
evaluation metric calculation.

#### Data Preparation

The main purpose of this phase is to standardize the data format of the
evaluation cases based on diverse task instructions in the proposed benchmark
to enhance the accuracy and stability of MFMs. First, we organize all the
evaluation cases (with different instruction formats) under each capability
dimension into the same JSON file. Each evaluation case in this file has a
unified format and contains keys such as "sample_id", "task_name". We then
carefully customize multiple fixed prompt templates for each capability
dimension to reduce the impact of prompts on the performance of MFMs. Finally,
we structure each evaluation case based on the JSON structure and the
corresponding prompt template as the input for MFMs.

#### Model Delpoyment

For model deployment, our platform provides a unified interface for multiple
different models, through which they can receive input prompts and return
results in the same way. This interface enhances scalability, enabling the
easy integration of new models by simply implementing this interface.

#### Metric Calculation

The typical evaluation methods of MFMs fall into two categories: automatic
evaluation and human evaluation[32]. In this paper, we use the human
evaluation method for cases involving planning Q&A instructions, while other
evaluation cases under the form of embodied Q&A instructions are evaluated
automatically using GPT-3.5[33]. There are two main reasons for this: ①
Compared with embodied Q&A, the output results of planning Q&A are more
uncertain and complex, with results often not being uniquely correct. This
makes it difficult for automatic evaluation to judge the quality of the
outputs accurately. ② GPT-3.5 is capable of performing efficient and accurate
evaluation when dealing with cases where the output results are relatively
stable. This characteristic is very beneficial for large-scale benchmarks as
it enhances both the efficiency and accuracy of the evaluation process. As
shown in Fig.3, our automatic evaluation platform integrates GPT-3.5[33] while
saving all output results of MFMs for convenient human evaluation.

##  4 Experiments

In this section, we report the experimental results of MFMs on embodied Q&A
tasks (Sec. 4.1) and Planning Q&A (Sec. 4.2) tasks according to different
evaluation metrics.

Table 1: Evaluation Results. We report the average scores of each model in each aspect. We highlight the best scores with bold and underline and use bold only to mark the second-best scores. Dimensions | BLIP-2 | MiniGPT-4 | InstructBLIP | GPT-4V | LLaVA-1.5 | MiniCPM
---|---|---|---|---|---|---
Object Understanding | Type | 18.5% | 19.3% | 21.0% | 34.9% | 3.7% | 44.0%
Property | 2.3% | 39.9% | 3.6% | 58.1% | 48.1% | 53.6%
OU.Agg. | 10.4% | 29.6% | 12.3% | 46.5% | 25.9% | 48.8%
Spatio-Temporal Perception | Spatial | 38.9% | 31.4% | 38.6% | 52.3% | 5.5% | 65.5%
Temporal | 22.5% | 23.3% | 20.0% | 30.0% | 25.8% | 26.7%
STP.Agg. | 30.7% | 27.4% | 29.3% | 41.2% | 15.7% | 46.1%
Task Understanding | Relevant | 10.0% | 20.0% | 8.6% | 52.9% | 22.9% | 35.7%
Operation | 67.3% | 25.0% | 71.2% | 71.2% | 23.1% | 82.7%
Sequence | 22.8% | 28.8% | 20.1% | 37.0% | 27.7% | 44.6%
Goal | 42.0% | 24.0% | 26.0% | 83.0% | 27.0% | 63.0%
TU.Agg. | 35.5% | 24.5% | 31.5% | 61.0% | 25.2% | 56.5%
EQA.Agg. | 28.0% | 26.4% | 26.1% | 52.4% | 23.0% | 51.9%

###  4.1 Results on Embodied Q&A Tasks

For the embodied Q&A tasks (object understanding, spatio-temporal perception,
and task understanding), we evaluate the performance of GPT-4V [4] and five
other open-source MFMs, namely BLIP-2 [1], MiniGPT-4 [2], InstructBLIP [3],
LLaVA-1.5 [5], and MiniCPM [6]. The experimental results are presented in Tab.
1 and radar chart (Fig. 5). In Tab. 1, we report the average scores for each
capability dimension across each aspect, the average scores for all aspects of
each capability dimension (denoted OU./STP./TU.Agg.), as well as the overall
average score for all embodied Q&A tasks (denoted EQA.Agg.). From Tab. 1 and
Fig. 5, we can draw the following general conclusions: ❶ GPT4V [4] and MiniCPM
[6]significantly outperform the other four MFMs, with a performance
improvement of over 100% compared to the worst-performing LLaVA-1.5[5]. ❷ The
overall average score differences among the four MFMs, BLIP-2 [1], MiniGPT-4
[2], InstructBLIP [3], LLaVA-1.5 [5] are relatively minor, indicating
comparable performance levels on the embodied Q&A tasks. ❸ MiniCPM [6]
performs close to GPT-4V [4] and surpasses the other open-source MFMs in all
aspects. ❹ Compared with higher-level and more abstract task understanding
capabilities, MFMs generally perform worse in low-level perception
capabilities such as object understanding and spatio-temporal perception.
Therefore, we recommend augmenting MFMs with object detection modules, such as
GroundingDINO [34], and integrating 3D information [35], instead of relying
solely on MFMs. Note that all models occasionally generate output that does
not conform to the specified format requirements, with MiniGPT-4 exhibiting
this issue most frequently. By specifically analyzing the results of the six
MFMs in each capability dimension, we further obtain the following
conclusions:

Object Understanding ❶ MiniCPM [6] excels the other models in recognizing
object types with an average score of 44.0%. ❷ In terms of identifying object
properties, GPT-4V [4] is far ahead with an average score of 58.1%, followed
by MiniCPM[6] with 53.6%. ❸ Overall, the object understanding capability of
MiniCPM [6] and GPT-4V [4] is more impressive compared to other MFMs.
MiniGPT-4 [2] and LLaVA-1.5 [5] have an accuracy of 29.6% and 25.9%,
respectively, while InstructBLIP [3] and BLIP-2 [1] can only succeed in about
11% cases.

Figure 6: Results of GPT-4V on General Description-based embodied reasoning
and object uderstanding, spatio-temporal perception, and task understanding-
oriented embodied reasoning tasks.

Figure 7: Results of gradually information injection. +o. means adding object
information, +s.t. means adding spatio-temporal information, and +k. means
adding task-related knowledge.

Spatio-temporal Perception ❶ Regarding spatio-temporal perception, the average
scores of the top-performing models, GPT-4V [4] and MiniCPM [6], were both
above 40%, and each excels in either temporal or spatial perception.and
spatial perception, respectively. ❷ Compared with spatial perception, except
for LLaVa-1.5 [5], the average scores of all other MFMs in temporal perception
tasks do not exceed 30%. ❸ Overall, spatiotemporal perception is a relatively
common and challenging problem for MFMs.

Task Understanding ❶ Regarding the four aspects of task understanding, the six
MFMs perform significantly better in task-related object operation knowledge
and task goal understanding. In these two dimensions, the average scores of
four MFMs and two MFMs exceeded 60%. ❷ GPT-4V [4] and MiniCPM [6] perform best
in task-related object recognition and task step sequence understanding,
respectively, but these two aspects are still difficult for MFMs. This is
mainly because task-related object recognition and task step sequence require
MFMs to integrate multiple information such as vision, language, and task
knowledge for reasoning.

###  4.2 Results on Planning Q&A Tasks

Based on the proposed evaluation framework, Planning Q&A corresponds to the
highest-level embodied reasoning capability dimension, which includes two
types of tasks: General Description-based embodied reasoning (GD-ER) and
object understanding (OU), spatio-temporal perception (STP), and task
understanding (TU)-oriented embodied reasoning (OU/STP/TU-ER). We evaluate the
performance of MFMs on the two types of tasks, respectively. Specifically, we
first utilize the proposed automatic evaluation platform to obtain the
evaluation result files for all evaluation cases. Then, experienced AI
researchers manually score all the result files,based on whether the task
planning results were reasonable and could guide the completion of the task
goals, assigning a score of 1 for satisfactory task plans and 0 otherwise.

Given that GPT-4V [4] and MiniCPM [6] show state-of-the-art performance on
embodied Q&A tasks and significantly outperform other MFMs, this experiment
evaluates the performance of GPT-4V [4] and MiniCPM [6] on planning Q&A tasks.
However, none of the 270 evaluation cases for MiniCPM [6] in the benchmark
yielded correct task planning results, and its outputs contained extraneous
information to the answers. To ensure reliability, we also evaluated the
output results of MiniCPM [6] on planning Q&A tasks on its official website
demo. The results are consistent with those obtained on our platform, which
shows that MiniCPM [6] has serious limitations in end-to-end embodied task
planning. Therefore, we only show the evaluation results of GPT-4V. From Fig.
7, we can draw the following conclusion: Under the general task description,
the success rate of the most advanced GPT-4V is only 19%. In other words,
among the 100 typical household tasks we proposed, GPT-4V correctly planned
only 19 tasks. Obviously, this performance falls short of human expectations.
Similarly, GPT-4V’s results on OU-ER, STP-ER, and TU-ER tasks were also
unsatisfactory.

Furthermore, we explore the effect of providing additional information in
embodied reasoning. As shown in Fig. 7, we select 50 typical tasks from the
original set. For each task, we gradually inject information related to some
supporting capabilities into GPT-4V [4] via prompts (e.g., informing GPT-4V
that the food on the table consists of an apple and a pie) and detect the
impact of corresponding capabilities on task planning through changes in
success rate. We obtain the following conclusions: ① Injecting object
information, spatio-temporal information, and task knowledge can have a
positive effect on task planning; ② compared to settings where only object
information (+o.) or object, spatial, and task knowledge (+o.+s.t.+k.) are
injected, injecting object and spatial information (+o.+s.t.) notably enhances
the success rate of task planning. Due to space constraints, we have placed
the other additional experiments in Appendix D.

##  5 Conclusion

In this paper, we propose a systematic evaluation framework tailored for
embodied task planning for the first time. Utilizing this framework, we have
developed an evaluation benchmark, called MFE-ETP, striving to explore the
performance limitations of MFMs. Empirical results on our benchmark
consistently indicate that even advanced GPT-4V fails to effectively complete
embodied planning tasks. Notably, object type recognition and spatiao-temporal
perception are critical factors that affect the accuracy of MFMs’ task
planning capabilities. We conduct an in-depth analysis of the experimental
results and put forward many useful insights to promote the application and
development of MFMs in the field of embodied task planning.

Limitations and future work. ① We utilized virtual environments as data
sources for easier data collection, which may raise concerns about the
generalizability of our findings to more realistic scenarios. ② Besides,
evaluating a broader range of models remains a promising direction for future
research. ③ While the Embodied Reasoning task remains highly challenging for
current models, planning all future steps in advance remains advantageous.
Utilizing closed-loop control systems like ViLa [16] for testing task planning
could offer a viable compromise. ④ Furthermore, exploring additional QA
formats such as correcting past behavior and predicting subsequent actions
could further enrich the research.

## References

  * [1] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.  In ICML, 2023.
  * [2] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  arXiv preprint arXiv:2304.10592, 2023.
  * [3] W Dai, J Li, D Li, AMH Tiong, J Zhao, W Wang, B Li, P Fung, and S Hoi.  Instructblip: Towards general-purpose vision-language models with instruction tuning. arxiv 2023.  arXiv preprint arXiv:2305.06500, 2023.
  * [4] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [5] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning.  arXiv:2310.03744, 2023.
  * [6] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun.  Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024.
  * [7] Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, and Yueting Zhuang.  Empowering vision-language models to follow interleaved vision-language instructions.  arXiv preprint arXiv:2308.04152, 2023.
  * [8] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al.  Rt-1: Robotics transformer for real-world control at scale.  arXiv preprint arXiv:2212.06817, 2022.
  * [9] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al.  Rt-2: Vision-language-action models transfer web knowledge to robotic control.  arXiv preprint arXiv:2307.15818, 2023.
  * [10] Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al.  Vision-language foundation models as effective robot imitators.  arXiv preprint arXiv:2311.01378, 2023.
  * [11] Rutav Shah, Roberto Martín-Martín, and Yuke Zhu.  Mutex: Learning unified policies from multimodal task specifications.  arXiv preprint arXiv:2309.14320, 2023.
  * [12] Jiachen Li, Qiaozi Gao, Michael Johnston, Xiaofeng Gao, Xuehai He, Suhaila Shakiah, Hangjie Shi, Reza Ghanadan, and William Yang Wang.  Mastering robot manipulation with multimodal prompts through pretraining and multi-task fine-tuning, 2023.
  * [13] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  Palm-e: An embodied multimodal language model.  arXiv preprint arXiv:2303.03378, 2023.
  * [14] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan.  Vima: robot manipulation with multimodal prompts.  In Proceedings of the 40th International Conference on Machine Learning, pages 14975–15022, 2023.
  * [15] Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, and Katsushi Ikeuchi.  Gpt-4v (ision) for robotics: Multimodal task planning from human demonstration.  arXiv preprint arXiv:2311.12015, 2023.
  * [16] Yingdong Hu, Fanqi Lin, Tong Zhang, Li Yi, and Yang Gao.  Look before you leap: Unveiling the power of GPT-4V in robotic vision-language planning.  CoRR, abs/2311.17842, 2023.
  * [17] Yi Chen, Yuying Ge, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu.  Egoplan-bench: Benchmarking egocentric embodied planning with multimodal large language models.  CoRR, abs/2312.06722, 2023.
  * [18] Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo.  Embodiedgpt: Vision-language pre-training via embodied chain of thought.  Advances in Neural Information Processing Systems, 36, 2024.
  * [19] Jingkai Sun, Qiang Zhang, Yiqun Duan, Xiaoyang Jiang, Chong Cheng, and Renjing Xu.  Prompt, plan, perform: Llm-based humanoid control via quantized imitation learning, 2023.
  * [20] Jiaqi Wang, Zihao Wu, Yiwei Li, Hanqi Jiang, Peng Shu, Enze Shi, Huawen Hu, Chong Ma, Yiheng Liu, Xuhui Wang, et al.  Large language models for robotics: Opportunities, challenges, and perspectives.  arXiv preprint arXiv:2401.04334, 2024.
  * [21] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji.  MME: A comprehensive evaluation benchmark for multimodal large language models.  CoRR, abs/2306.13394, 2023.
  * [22] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang.  The dawn of lmms: Preliminary explorations with gpt-4v(ision).  CoRR, abs/2309.17421, 2023.
  * [23] Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo.  Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models.  CoRR, abs/2306.09265, 2023.
  * [24] Chaoyou Fu, Renrui Zhang, Zihan Wang, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, Yunhang Shen, Mengdan Zhang, Peixian Chen, Sirui Zhao, Shaohui Lin, Deqiang Jiang, Di Yin, Peng Gao, Ke Li, Hongsheng Li, and Xing Sun.  A challenger to gpt-4v? early explorations of gemini in visual expertise.  CoRR, abs/2312.12436, 2023.
  * [25] Yunkang Cao, Xiaohao Xu, Chen Sun, Xiaonan Huang, and Weiming Shen.  Towards generic anomaly detection and understanding: Large-scale visual-linguistic model (GPT-4V) takes the lead.  CoRR, abs/2311.02782, 2023.
  * [26] Wenhao Wu, Huanjin Yao, Mengxi Zhang, Yuxin Song, Wanli Ouyang, and Jingdong Wang.  Gpt4vis: What can GPT-4 do for zero-shot visual recognition?  CoRR, abs/2311.15732, 2023.
  * [27] Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Linran Xu, Dengke Shang, Zheng Zhu, Shaoyan Sun, Yeqi Bai, Xinyu Cai, Min Dou, Shuanglu Hu, Botian Shi, and Yu Qiao.  On the road with gpt-4v(ision): Early explorations of visual-language model on autonomous driving.  CoRR, abs/2311.05332, 2023.
  * [28] Jiageng Mao, Yuxi Qian, Hang Zhao, and Yue Wang.  Gpt-driver: Learning to drive with gpt.  arXiv preprint arXiv:2310.01415, 2023.
  * [29] Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mcvay, Oleksandr Maksymets, Sergio Arnaud, et al.  Openeqa: Embodied question answering in the era of foundation models.  In 2nd Workshop on Mobile Manipulation and Embodied Intelligence at ICRA 2024, 2024.
  * [30] Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martín-Martín, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, et al.  Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments.  In Conference on Robot Learning, pages 477–490. PMLR, 2022.
  * [31] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba.  Virtualhome: Simulating household activities via programs.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8494–8502, 2018.
  * [32] Chaoqun He, Renjie Luo, Shengding Hu, Yuanqian Zhao, Jie Zhou, Hanghao Wu, Jiajie Zhang, Xu Han, Zhiyuan Liu, and Maosong Sun.  Ultraeval: A lightweight platform for flexible and comprehensive evaluation for llms, 2024.
  * [33] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [34] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  arXiv preprint arXiv:2303.05499, 2023.
  * [35] An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu.  Spatialrgpt: Grounded spatial reasoning in vision language model.  arXiv preprint arXiv:2406.01584, 2024.
  * [36] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia.  Spatialvlm: Endowing vision-language models with spatial reasoning capabilities.  arXiv preprint arXiv:2401.12168, 2024.

##  Appendix A Appendix

##  Appendix B Appendix Overview

The appendix includes the following content:

  1. 1.

Detailed information about the proposed MFE-ETP benchmark. (Sec. C)

  2. 2.

Additional experimental results. (Sec. D)

  3. 3.

Detailed information of MFMs (Sec. E).

  4. 4.

Accessibility and Statement(Sec. F)

##  Appendix C Details of MFE-ETP Benchmark

###  C.1 BEHAVIOR-100 and VirtualHome

BEHAVIOR-100 [30] simulates 100 everyday household tasks for embodied AI, with
a distribution similar to the full space of simulatable tasks in the American
Time Use Survey (ATUS). It aims to create realistic and complex scenarios for
AI testing. It introduces a predicate logic-based language (BDDL) for defining
tasks and possesses simulator-agnostic features for versatile applications.
Due to the reality, diversity, and complexity of home tasks, state-of-the-art
AI still struggles with the benchmark’s challenges. More information can be
found in BEHAVIOR-100[30].

VirtualHome [31] utilizes "programs" to model complex household activities,
that can be implemented manually through a game-like interface. Specifically,
VirtualHome [31] uses the Unity3D platform to enable human agents to perform
tasks in a simulated environment. This helps create video datasets of
household activities, each of which is a series of actions and interactions
executed by agents. More information can be found in VirtualHome [31].

To avoid redundancy of similar tasks while maintaining task diversity, we
illustrate the mapping relationships between our abstracted 20 typical
household task types and specific task instances in BEHAVIOR-100 [30],
VirtualHome[31], and our benchmark In Tab. 2 and Tab. 3.

Table 2: Task mapping part1.

Task Type |  BEHAVIOR-100 |  VirtualHome
---|---|---
Cooking |  chopping vegetables |  cook some food
|  preparing salad |  prepare afternoon tea
|  thawing frozen food |
Beverage Making |  making tea |  make coffee
Meal Service |  serving a meal |  serve breakfast
|  serving hors d oeuvres |  serve cookies
|  |  set up table
Storing |  bottling fruit |  store meat to freezer
|  preserving food |  preserve food
|  storing food |
Surface Cleaning |  cleaning barbecue grill |  clean sink
|  cleaning table after clearing |  wipe off shoes
|  cleaning up after a meal |  sweep and wipe table off with rag
|  cleaning windows |  clean toilet
|  cleaning bathtub |  clean large jacuzzi tub
|  cleaning toilet |
Room Cleaning |  cleaning bathrooms |  clean kitchen
|  cleaning bedroom |  clean bathroom
|  cleaning up the kitchen only |

Table 3: Task mapping part2.

Task Type | BEHAVIOR-100 | VirtualHome
---|---|---
Dishwashing | loading the dishwasher | wash dishes by hand
| wash dishes with dishwasher
| take dishes out of dishwasher
Laundry |  | wash clothes
| put away clean clothes
Recycling | putting leftovers away | throw leftovers
Organizing | clearing the table after dinner | organize livingroom
collect misplaced items | organize bedroom
re-shelving library books | organize washstand
| re-shelving books
| bring dirty plate to sink
Packing | packing bags or suitcase | pack lunch
packing boxes for household move or trip |
packing car for trip |
boxing books up for storage |
packing adult s bags |
packing child s bag |
packing food for work |
packing lunches |
packing picnics |
Sorting | sorting books | sort clothing
sorting mail | sort mail
sorting groceries |
Necessities Maintenance |  | change toilet paper roll
| change toothbrush
| replace towel
Gardening |  | water houseplants
Floor Care | cleaning carpets | vacuum
cleaning floors | clean floor
mopping floors |
vacuuming floors |
Furniture Care | cleaning closet | arrange chairs
cleaning cupboards | straighten bookshelf
| change curtains
| lay tablecloth
| polish table
Appliance Care | cleaning freezer | clean fridge
cleaning microwave oven | add paper to printer
cleaning oven |
cleaning stove |
cleaning up refrigerator |
defrosting freezer |
Decoration |  | hang pictures
| put up decoration
| straighten pictures on wall
Event Prep | assembling gift baskets | light candle
filling a Christmas stocking |
filling an Easter basket |
Miscellaneous Tasks |  | turn on light
| turn off light
| change light

###  C.2 MFMs Prompt Texts

As indicated in other research, the success of embodied tasks with MFMs is
heavily influenced by the design of prompts. Therefore, we have elaborately
designed a series of standardized prompt templates to ensure that the
evaluation process uses the same question format and language style, thereby
reducing the impact of differences in prompt quality. Fig.8 and Fig.9 present
examples of prompts on embodied Q&A and planning Q&A, respectively.

###  C.3 Full Object Properties and Action List

Tab.4 lists the definitions of each considered functional property and its
representative objects. In addition, we illustrate in Tab.5 the definition of
all actions that can satisfy embodied reasoning for the typical household
tasks we consider.

Table 4: List of object properties.

Property | Annotation | Example
---|---|---
breakable |  | Mark if the object is brittle, that is, it can be broken into smaller
---
pieces by a human dropping it on the floor.
wine bottle, room light
cleaningTool | Is a [object] designed to clean things? | scrub brush
cookable | Can a [object] be cooked? | biscuit, pizza
grabbable |  | If an object has this attribute, it is usually lightweight
---
and can be potentially grabbed and picked up by the robot.
apple, bottle, rag, plate
openable | Mark if the object is designed to be opened. | mixer, keg
sliceable | Can a [object] be sliced easily by a human with a knife? | sweet corn, sandwich
slicingTool | Can a [object] slice an apple? | blade, razor
toggleable |  | The object can be switched between a finite number of
---
discrete states and is designed to do so.
hot tub, light bulb
waterSource | where you can get water | sink

Table 5: List of actions for embodied reasoning.

Action | Annotation
---|---
navigate_to(arg1) |  | Navigate to the arg1, which can be a object.
---
Preconditions: arg1 is not within reachable distance for the robot.
Postconditions: arg1 is within reachable distance for the robot.
grasp(arg1): |  | Grasp arg1.
---
Preconditions: arg1 is within reachable distance and no object is currently
held.
Postconditions: arg1 is being held, and the robot’s position is unchanged
compared to the robot’s position before the action is performed.
place_onTop(arg1, arg2): |  | Place arg1 on top of arg2.
---
Preconditions: arg1 is currently being held, and arg2 is reachable .
Postconditions: arg1 is put on top of arg2. The robot’s position is unchanged
compared to the robot’s position before the action is performed and arg2 is
reachable.
place_under(arg1, arg2): |  | Place arg1 under arg2.
---
Preconditions: arg1 is currently being held, and arg2 is reachable.
Postconditions:arg1 is put under arg2. The robot’s position is unchanged
compared to the robot’s position before the action is performed and arg2 is
reachable.
place_inside(arg1, arg2): |  | Place arg1 inside of arg2.
---
Preconditions: arg1 is currently being held, and arg2 is reachable.
Postconditions:arg1 is put inside of arg2. The robot’s position is unchanged
compared to the robot’s position before the action is performed and arg2 is
reachable.
place_onLeft(arg1, arg2): |  | Place arg1 on left of arg2.
---
Preconditions: arg1 is currently being held, and arg2 is reachable .
Postconditions: arg1 is put on left of arg2. The robot’s position is unchanged
compared to the robot’s position before the action is performed and arg2 is
reachable.
place_onRight(arg1, arg2): |  | Place arg1 on right of arg2.
---
Preconditions: arg1 is currently being held, and arg2 is reachable.
Postconditions: arg1 is put on right of arg2. The robot’s position is
unchanged
compared to the robot’s position before the action is performed and arg2 is
reachable.
open(arg1): |  | Open arg1. Preconditions: Arg1 is closed, and arg1 is reachable.
---
Postconditions: Arg1 is open.
close(arg1): |  | Close arg1. Preconditions: Arg1 is open, and arg1 is reachable.
---
Postconditions: Arg1 is closed.
slice(arg1): |  | Slice arg1. Preconditions: arg1 is not sliced, and arg1 is reachable.
---
Postconditions: arg1 is sliced
wipe(arg1, arg2): |  | Wipe across the surface of arg2 with arg1. Preconditions: Arg1 is
---
currently being held, and arg2 is reachable. Postconditions:
Arg1 continues to be held, arg2 holds state unchanged.
wait(arg1): |  | Wait for arg1 seconds. Preconditions: None. Postconditions: arg1
---
second(s) has(have) passed.
toggle(arg1): |  | Press the button of arg1 to turn it on or off, Preconditions: Arg1 is open
---
or closed, and arg1 is reachable. Postconditions: Arg1 is closed or open.
water(arg1, arg2): |  | Water arg2 with arg1.
---
Preconditions: arg1 is currently being held, and arg2 is reachable.
Postconditions: arg1 continues to be held, arg2 is watered.

Figure 8: Examples of prompts on embodied Q&A. Taking the ’Object Type Q&A’ as
an example, the final input prompt text consists of task descriptions (red
box) and specific task instructions (blue box).

Figure 9: Examples of prompts on planning Q&A. In the final input prompt text,
the task name (red box) and the general description (blue box) of the task
goal will be replaced with specific task instances.

##  Appendix D Additional Experimental Results

Figure 10: Qualitative Results of GPT-4V in Object Understanding.

Figure 11: Qualitative Results of GPT-4V in Spatio-temporal Perception.

Figure 12: Qualitative Results of GPT-4V in Task Understanding.

Figure 13: Failed cases of planning Q&A. We highlight the wrong steps in blue.

###  D.1 Results of Qualitative Analysis

Based on our proposed framework, we further qualitatively analyze the
evaluation results of the state-of-the-art GPT-4V[4].

Object Understanding Through manual analysis of specific failed cases of
GPT-4V, we attribute the main causes of incorrect answers to limitations
inherent in GPT-4V, poor perspective (even under the best perspective we can
obtain, the objects are still difficult to recognize), and low resolution.
Fig. (10) shows a few representative failed cases.

Spatio-temporal Perception Previous works have mentioned that GPT-4V lacks
spatio-temporal understanding ability [27, 36], which is consistent with our
evaluation results. By analyzing specific failed cases of spatio-temporal
perception Q&A tasks, we found that GPT-4V often fails to perceive distance
and direction of 2D images, especially the left and right directions. (as in
Fig. 11) Also, we found that GPT-4V performs poorly in terms of temporal
perception as well, which is reflected in its inability to accurately sort
images according to the task completion process.

Task Understanding Fig. 12 shows a few representative failed cases. Combined
with Tab. 1, the main limitations of GPT-4V in task understanding are mainly
in the inability to accurately select task-related objects and select
reasonable action execution sequences, which are essential knowledge for task
planning.

Embodied Reasoning We analyzed the corresponding results of planning Q&A and
found some common issues: firstly, task related object recognition faces
challenges, including insufficient recognition, incorrect recognition, and
chaotic spatial relationships when there are a large number of objects.
Secondly, the MFMs’ spatiotemporal perception ability is not strong,
manifested in insufficient understanding of the current position, as well as
the poor ability to perceive whether an object is within the operating range,
which often leads to providing unnecessary or missing navigation actions. In
terms of action selection, the system may wrongly or repeatedly call actions,
or perform actions beyond predefined ones. In addition, GPT-4V may give some
difficult to execute conditional branch statements, or assume a condition when
identifying key information unclear. These issues comprehensively affect the
efficiency and accuracy of the system, and require specific optimization
measures to improve. Finally, GPT-4V cannot provide accurate planning for
tasks involving too many objects. Fig. 13 shows a few representative failed
cases.

###  D.2 Linear Regression Analysis of The Impact of Each Aspect on Task
Planning

In this section, we will use linear regression coefficients as proxies for the
impact of various factors on embodied reasoning. Specifically, for each
robotic task, we use the binary result of embodied reasoning as the dependent
variable and mark successful cases as 1 and failed cases as 0. Take the basic
capabilities corresponding to all aspects as a multivariate independent
variable, and use their accuracy as the value. The obtained coefficients are
shown in the Tab. 6. It can be concluded that spatial perception is the most
important for embodied reasoning, with a coefficient of 0.4, followed by 0.14
for object recognition, 0.09 for action sequence understanding, and 0.07 for
operation, and those of other aspects are relatively closer to 0 or minus,
indicating that their impact on embodied reasoning is not significant. Note
that some aspects, such as the coefficient of time perception, are negative,
which does not necessarily indicate that a good ability in that aspect will
have a negative impact on embodied reasoning. It is more likely that this is
because the planning success rate is low, and many times when planning fails,
GPT-4V can also get relative high scores on these aspects.

Table 6: Linear regression coefficients for each aspects.

Aspect | Type | Property | Temporal | Spatial | Relevant | Operation | Sequence | Goal
---|---|---|---|---|---|---|---|---
Coefficient | 0.14 | 0.01 | -0.1 | 0.4 | -0.04 | 0.07 | 0.09 | 0.06

###  D.3 Validity Analysis of LLM-based Metrics

For the larger scale evaluation benchmark we propose, it is a challenge to
efficiently evaluate embodied Q&As. In essence, we would like MFMs to give
reasonable results rather than results that are exactly consistent with the
ground truth. While human evaluation remains the gold standard, it is also
expensive and time-consuming. Thus, we use an automatic LLM-Based evaluation
metric in this work as described in Sec 3.3. To show the effectiveness of LLM-
Based evaluation metric, we manually score the output of state-of-the-art
GPT-4V and compare them with the evaluation results of GPT 3.5, and the
experimental results are shown in Tab. 7. ICC1 (Intraclass Correlation
Coefficient 1) is primarily used to evaluate the absolute consistency among
different evaluators in a single measurement scenario. This type of ICC
focuses on the consistency of each evaluator’s individual rating of the same
object or subject. We calculated the ICC1 value to be 0.95, indicating high
consistency between the human scores and the GPT-3.5 scores. Its extremely low
p-value (0.000014) statistically confirms that this consistency is
significant. Thus, the results of the LLM-based evaluation metric are almost
aligned with the manual evaluation results, and the LLM evaluation metric is
preferable for benchmarking.

Table 7: Evaluation metric comparison. Human means using scores given by five human annotators, GPT-3.5 metric utilizes GPT-3.5 for automated evaluation. Metric | Type | Property | Temporal | Spatial | Relevant | Operation | Sequence | Goal
---|---|---|---|---|---|---|---|---
Human | 32.2 | 57.5 | 21.7 | 64.8 | 47.0 | 78.8 | 37.5 | 84.0
GPT-3.5 | 34.9 | 58.1 | 30.0 | 52.3 | 52.9 | 71.2 | 37.0 | 83.0

##  Appendix E Detailed Information of MFMs

Tab. 8 shows the versions of each MFM. In particular, we access GPT-4V through
OpenAI API, and for other open-source models, we utilize a single NVIDIA
GeForce RTX 4090 GPU.

Table 8: Versions of each MFM. Model | Version
---|---
BLIP-2[1] | Flan-T5-XL
MiniGPT-4[2] | Vicuna-7B
InstructBLIP[3] | Flan-T5-XL
GPT-4V[4] |  gpt-4-vision-preview
LLaVA-1.5[5] | LLaMA-7B
MiniCPM[6] | MiniCPM-Llama3-V-2_5

##  Appendix F Accessibility and Statement

#### URL to dataset

Both the code and data are slated for release at our repositorys:
https://github.com/TJURLLAB-EAI/MFE-ETP.

#### Author Statement

We accept complete accountability for any infringement of rights that may
arise during the utilization or dissemination of the data presented in this
work. We commit to undertaking necessary measures, including the modification
or deletion of any data involved in such violations, to address these concerns
promptly.

We commit to maintaining the data and the codes. Updates to any future
versions of the dataset and code will be made available at this location. The
data provided is designed for academic purposes.

#### License

MFE-ETP is released under the MIT License.


# Author’s Sentiment Prediction

Mohaddeseh Bastan, Mahnaz Koupaee, Youngseo Son, Richard Sicoli,
Niranjan Balasubramanian
Stony Brook University, Stony Brook, New York
{mbastan, mkoupaee, yson, rpsicoli<EMAIL_ADDRESS>

###### Abstract

We introduce PerSenT, a dataset of crowd-sourced annotations of the sentiment
expressed by the authors towards the main entities in news articles. The
dataset also includes paragraph-level sentiment annotations to provide more
fine-grained supervision for the task. Our benchmarks of multiple strong
baselines show that this is a difficult classification task. The results also
suggest that simply fine-tuning document-level representations from BERT isn’t
adequate for this task. Making paragraph-level decisions and aggregating them
over the entire document is also ineffective. We present empirical and
qualitative analyses that illustrate the specific challenges posed by this
dataset. We release 111Code and dataset released at
https://stonybrooknlp.github.io/PerSenT/ this dataset with 5.3k documents and
38k paragraphs covering 3.2k unique entities as a challenge in entity
sentiment analysis.

##  1 Introduction

11footnotetext:  This work is licensed under a Creative Commons Attribution
4.0 International License. License details:
http://creativecommons.org/licenses/by/4.0/.

How can we automatically infer the sentiment of an author towards an entity
based only on the text of their news article? This task can be seen as a part
of complete document understanding, with potential uses in detecting
journalistic bias, and in collecting articles that express certain viewpoints
towards entities. There are no readily useful datasets or effective solutions
for author sentiment inference in the news domain. Sentiment analysis
solutions have covered a wide range of domains including movie reviews [Pang
et al., 2008, Singh et al., 2013, Socher et al., 2011], product reviews [Dave
et al., 2003, Turney, 2002, Fang and Zhan, 2015], and social media [Abbasi et
al., 2008, Pak and Paroubek, 2010]. In the news domain, the closest is
inferring the sentiment expressed by one entity (or a group) towards another
entity (or a group) mentioned in a news article [Choi et al., 2016]. However,
this does not necessarily cover the sentiment expressed by the author towards
a specific entity.

To address this gap, we introduce PerSenT, a crowdsourced dataset of sentiment
annotations on news articles about people. For each article, annotators judge
what the author’s sentiment is towards the main (target) entity of the
article. The annotations also include similar judgments on paragraphs within
the article. Our experiments with multiple strong classification models show
that this is a difficult task that introduces multiple unmet challenges.

The task we propose is difficult mainly because most information contained in
a news article is likely irrelevant for the purposes of inferring the author’s
sentiment. This is a key difference compared to previously studied sentiment
problems in domains such as product or movie reviews, where most information
is likely relevant for sentiment inference. On the one hand, a single global
document-level representation requires careful aggregation of information
related to the target entity that is relevant for sentiment inference. On the
other, it is also not easy to make local decisions (say at a paragraph level)
and aggregate them — in many cases paragraph-level decisions can be noisy due
to discourse gaps, and furthermore not all paragraph-level decisions should
contribute equally to the final decision. Indeed neither approach works
satisfactorily. For instance, fine-tuning pre-trained BERT [Devlin et al.,
2018] base model only yields an F1 of 0.48 on one test set and 0.42 on the
more difficult test set for the document-level task. Paragraph-level
prediction using BERT is even more difficult, with 0.43 and 0.40 F1 values for
the two test sets respectively (see section 3.4 for more details). Motivated
by the nature of the task — requiring focus towards the target entity and
aggregating information across the discourse — we also benchmark models that
focus on entity-oriented representations (Recurrent Entity Network),
discourse-based representations. However, we find that their performance is
worse than the pre-trained BERT classifier.

We conduct further analysis to show the nature of the challenges introduced by
this news domain dataset. One of the main challenges in this task is as shown
in Figure 1, not all paragraphs necessarily convey the author’s sentiment
towards the target entity. Thus, making a single decision from a
representation of the entire document can be ineffective. Furthermore, our
analysis shows that prediction in documents with many unique entities is more
difficult. We also find that it is easier to predict document-level sentiment
when there are many paragraphs that convey the same sentiment. Last, we also
provide a qualitative error analysis that indicates the main categories of
inference errors. In summary, this work introduces a new sentiment inference
task in the news domain that introduces new challenges in document-level
sentiment inference along with a dataset that will support further research in
this area.

Figure 1: A positive (truncated) document towards the target entity Lisa
Pratt. A bold green/red word is lexically positive/negative towards the
entity. A green/red unbold word is positive/negative towards other entities in
the document. The solid green line means the author’s view towards the entity
is semantically positive on the span. The target entity and all her mentions
are in italics.

##  2 PerSenT Dataset

Given a news article about a person (we refer to this as the main or target
entity), the author sentiment prediction task is to infer the authors’ view
towards the main entity. This task requires models to carefully track
information that is specific to the entity and aggregate sentiment related
information across multiple mentions of the entity in the document.

Since there are no datasets that directly support this task of author
sentiment prediction, we create a new dataset PerSenT. It consists of news
articles that have been previously used for other types of sentiment analysis
and related tasks. The criteria for choosing a document is that the news
article should mainly talk about a person entity and that it should not be
very long or short. We used three resources to select news articles:

  * •

MPQA [Deng and Wiebe, 2015, Wiebe et al., 2005]: This dataset contains news
articles manually annotated for opinions, beliefs, emotions, sentiments,
speculations, etc. It also has target annotations which are entities and event
anchored to the heads of noun or verb phrases. All decisions on this dataset
are made on sentence-level and over short spans.

  * •

KBP Challenge [Ellis et al., 2014]: This resource contains TAC 2014 KBP
English sentiment slot filling challenge dataset. This is a document-level
sentiment filling dataset. In this task, given an entity and a sentiment
(positive/negative) from the document, the goal is to find entities toward
which the original entity holds the given sentimental view. We selected
documents from this resource which have been used in the following similar
work in sentiment analysis task [Choi et al., 2016].

  * •

Media Rank [Ye and Skiena, 2019]: This dataset ranks about 50k news sources
along different aspects. It is also used for classifying political ideology of
news articles [Kulkarni et al., 2018].

We used the following pre-processing steps on the collected news articles.
First we find all the person entities in each article, using Stanford NER
(Name Entity Resolution) tagger [Finkel et al., 2005] and all mentions of them
using co-reference resolution [Clark and Manning, 2016, Co, 2017]. We removed
articles which are not likely to have a main entity of focus. We used a simple
heuristic of removing articles in which the most frequent person entity is
mentioned only three times or less (even when counting co-referent mentions).
For the articles that remain we deemed the most frequent entity to be the main
entity of the article. We also filtered out extremely long and extremely short
articles to keep the articles which have at least 3 paragraphs and at most 16
paragraphs.

Documents are randomly separated into train, dev, and two test sets. We ensure
that each entity appears in only one of the sets. Our goal here is to avoid
easy to learn biases over entities. To avoid the most frequent entities from
dominating the training or the test sets, we remove articles that covered the
most frequent entities and use them as a separate test set (referred to as
frequent test set) in addition to the randomly drawn standard test set.

###  2.1 Crowdsourcing Sentiment Annotations

We obtained document and paragraph level annotations with the help of Amazon
Mechanical Turk workers. The workers first verified if the target entity we
provide is indeed the main entity in the document. Then, they rated each
paragraph in a document that contained a direct mention or a reference to the
target entity. Last, they rated the sentiment towards the entity based on the
entire document. In both cases, the workers made assessments about the authors
view based on what they said about the target entity. For both paragraph and
document level sentiment, the workers chose from five rating categories:
Negative, Slightly Negative, Neutral, Slightly Positive, or Positive. We then
combine the fine-grained annotations to obtain three coarse-grained classes
Negative, Neutral, or Positive.

###  2.2  Quality Control

To ensure quality control, we designed various assessments. First, at random,
we introduced one document for which the sentiment was already rated by an
expert. We removed annotations from users whose ratings differed from the
expert’s rating on this control subset. Second, the workers were required to
annotate all paragraphs, to ensure that they read all paragraphs before making
decisions about the document. Third, we evaluated different annotations for
each document. If all three were different from each other, we reassigned it
to two other annotators. Finally, 10% of annotations were randomly evaluated
by experts and the low-quality annotators are disqualified and their HITs are
reassigned to other annotators.

We measured the inter-rater reliability of annotators using weighted Fleiss’s
Kappa [Marasini et al., 2016], in which the penalty for each dissimilar
classification is based on the distance between two classes. For example, if
two annotators classify a document as a positive, the agreement weight is 1,
but if one classifies as a positive, and the other classifies as slightly
positive the agreement weight is 0.9. The weights between different classes
are shown in Table 1 where negative, slightly negative, neutral, slightly
positive, and positive classes are shown with -2, -1, 0, 1, and 2. The
weighted agreement number for the whole dataset is 0.79 for paragraph-level
annotations and 0.81 for document-level annotations which shows the
annotations are highly reliable. Also in 90% of the instances, at least two
out of three annotators agree with each other. For evaluating whether the main
entity is extracted correctly or not in each article, if at least two out of
three annotators declare that the main entity is not selected correctly, then
the document is not included in the dataset.

| -2 | -1 | 0 | 1 | 2
---|---|---|---|---|---
-2 | 1 | $cos(\pi/8)$ | $cos(\pi/4)$ | $cos(3\pi/8)$ | 0
-1 | $cos(\pi/8)$ | 1 | $cos(\pi/8)$ | $cos(\pi/4)$ | $cos(3\pi/8)$
0 | $cos(\pi/4)$ | $cos(\pi/8)$ | 1 | $cos(\pi/8)$ | $cos(\pi/4)$
1 | $cos(3\pi/8)$ | $cos(\pi/4)$ | $cos(\pi/8)$ | 1 | $cos(\pi/8)$
2 | 0 | $cos(3\pi/8)$ | $cos(\pi/4)$ | $cos(\pi/8)$ | 1
Table 1: Inter class weights used for computing inter annotated agreement.

###  2.3  Dataset Characteristics

Few popular entities tend to dominate news collection. In our collection,
there were four entities which were the main entity in nearly 800 articles. To
avoid these entities from dominating the train or test splits, we moved them
to a separate test collection. We split the remaining into a training,
development (dev), and test sets at random. Thus our collection includes one
standard test set consisting of articles drawn at random (Test Standard),
while the other is a test set which contains multiple articles about a small
number of popular entities (Test Frequent). The dataset statistics are shown
in Table 2. The distribution among three classes, negative, neutral, and
positive is unbalanced: 51% of documents are positive, 37% are neutral, and
only 11% are negative. The distribution in paragraph annotations is also
unbalanced: 43% of paragraphs are positive, 46% are neutral, and only 11% are
negative. Since the documents are a true random sample from news articles, we
do not modify these distributions.

(a) Distribution of paragraph length in dev set

(b) Distribution of document length in dev set

Figure 2: Distribution of paragraph and document length in dev set  Figure 3:
Distribution of paragraph and document label agreement

Figure 2(a) shows the distribution of paragraph length (number of words in the
paragraph) in the development (dev) set. Most paragraphs are between 20 to 60
words long and very few paragraphs are longer than 128 words long, which use
as the maximum length in our experiments. Figure 2(b) shows the distribution
of document lengths (number of words in the document) in terms of the number
of words in the development set. As shown in this figure, most of the
documents are between 100 to 600 words with only a few larger ones. Based on
this distribution, we set the maximum length of documents to 512 in our
experiments and trim longer documents 222In more than 90% of documents the 512
limit gets us at least 75% of the document content.. Overall, the task
requires handling long input contexts.

Figure 3 shows the percentage of paragraphs with paragraph-level and document-
level label agreement. As we see more than 70% of the documents (409 out of
578 documents in dev set) have at least 50% of label agreement between
paragraph labels and document label.

Dataset | Documents | Paragraphs | Unique
---|---|---|---
Negative | Neutral | Positive | Total | Negative | Neutral | Positive | Total | Entities
Train | 351 | 1246 | 1758 | 3355 | 2398 | 10933 | 10622 | 23953 | 2376
Dev | 58 | 216 | 304 | 578 | 388 | 1997 | 1784 | 4169 | 417
Test St. | 73 | 213 | 293 | 579 | 536 | 1934 | 1791 | 4261 | 426
Test Fr. | 139 | 320 | 368 | 827 | 964 | 2857 | 2148 | 5969 | 4
Total | 621 | 1995 | 2723 | 5339 | 4286 | 17721 | 16345 | 38352 | 3223
Table 2: We split the dataset into train, dev, and two test sets. Test
Standard (Test St.) is a random collection of news articles, whereas Test
Frequent (Test Fr.) is a collection that contains multiple articles on a
smaller subset of frequently discussed entities.

##  3 Evaluation

We benchmark the performance of strong classification baselines to understand
and highlight the difficulty of this task.

###  3.1 Baselines

Deep Averaging Network (DAN) We use the deep unordered text classification
method [Iyyer et al., 2015] as one of the baselines. The method can be seen as
a deep neural bag of the words model in which each layer magnifies meaningful
differences in the word embedding average, which provides a simple yet strong
document-level classification baseline.

Modified Recurrent Entity Network The original Recurrent Entity Model [Henaff
et al., 2016] is equipped with a dynamic, fixed-size memory which can maintain
and update a representation of the world. We re-purpose this model for
inferring entity sentiment. Firstly, we use two memory slots, one to maintain
the information related to the target entity and another to maintain all other
information of the world. We initialize the key of the target entity with the
corresponding word embeddings denoting the entity. Secondly, we use a bi-
directional LSTM to extract sentence-level information, rather than directly
updating the memory with word-vectors. This baseline is aimed at testing the
benefits of using entity-focused representations.
Hierarchical Discourse-based LSTM Hierarchical sequence models can create
multi-level representations over discourse structures [Son et al., 2018]. We
use discourse structures for our sentiment analysis model inspired by prior
work that found them useful for related text categorization tasks [Ji and
Smith, 2017]. We extract elementary discourse units (EDUs) using [Ji and
Eisenstein, 2014], then feed them to Hierarchical BiLSTM models as described
in [Son et al., 2018]. This model will encode discourse information for
building document-level representations.
Large Pre-trained Transformer: BERT + Fine-tuning We use the Bidirectional
Encoder Representation from Transformers (BERT) [Devlin et al., 2018] which
produced state of the art results for many NLP tasks. We use the BERT based
model (lower-case) and one non-linear layer on top of that for classification.
For the document-level task, we set the maximum sequence of the model to
512333The experiments include documents whose length are greater than 512 see
subsection 3.5 for more detailed information and for paragraph-level
experiments we set the sequence length to 128.

###  3.2 Prediction over Document-Level Representations

The first block in Table 5 shows the performance of the different models for
the document-level task, when making predictions over a single representation
computed over the entire document. Fine-tuning BERT on this target task
performs the best followed by the simpler deep averaging network (DAN) . The
more complex Modified REN and Discourse HLSTM models barely outperform a
weighted random baseline, showing that complex models cannot learn from
scratch effectively on this dataset. More importantly, even the pre-trained
BERT model only achieves modest performance overall illustrating the
difficulty of this task and the challenge in learning effective document-level
representations. The results also show the difficulty of classification for
the documents with frequent entities.

###  3.3 Aggregating Paragraph-Level Decision

Next we describe the performance when making paragraph-level decisions and
aggregate them to yield a document-level decision. Table 5, second block,
shows that when BERT classifier is trained to make paragraph-level decisions
that are then directly aggregated (using majority or average), it performs
worse than making a single decision over the entire document. We hypothesize
that this is mainly because the paragraph level decisions can be challenging
because of the limited context (BERT achieves macro F1 of 0.40 in paragraph-
level sentiment analysis).

###  3.4 Fine-Tuning for Focused Representations

We hypothesize that learning which pieces of information relate to the target
entity and which pieces are relevant for sentiment inference can help produce
more effective representations. In the extreme case, if the models know
exactly which paragraphs to focus on, then they can be more effective at the
document-level task. We conducted an oracle experiment, where the paragraphs
were annotated with their true paragraph-level sentiment label, and then fed
to the document-level BERT. The first row in the third block of Table 5 (BERT
Gold Label) shows the results of this oracle for the test set. This suggests
that learning from paragraph-level decisions can help document-level
prediction if we can learn to make good paragraph-level decisions.

BERT representations are learned on general language modeling tasks, which may
not capture this inductive bias. We devised one fine-tuning task and one
multi-task to focus BERT representations:

(i) BERT + ELM Sequential: Here the BERT model is first fine-tuned on the
paragraph-level prediction task using paragraph supervision, and then further
fine-tuned for the document-level task. The results are in BERT Sequential
row. This fine tuning results in the best macro-F1 between the paragraph-level
supervision models, showing the benefits of focusing the representations on
different levels.

(ii) BERT + ELM MultiTask: We devise an entity-based language modeling task
(masked entity language model or Masked ELM), following BERT’s original Masked
LM task [Devlin et al., 2018]. For each document in the PerSenT dataset all
entities and their mentions are masked one at a time. Then we use pretrained
BERT and retrain it with a classification task to predict whether the masked
entity is the main entity of the document or not. For instance, if we have 10
entities and their mentions in a document, we create 10 different versions of
that document in Masked ELM dataset. If the masked entity is the main entity,
that document is labeled True, otherwise it is labeled as False. Statistics of
this Masked ELM task are shown in Table 3. The results of fine-tuning BERT on
this Masked ELM task on development set is shown in Table 4. This task forces
BERT to learn contextual information that relate to the target entity token.
Since this task is closer to the BERT’s original LM task, fine-tuning top
1,2,3, and 4 layers are explored and the best is reported. We use this task in
two ways. In the sequential mode, we use this task as an intermediate training
task, before fine-tuning on the target document-level task. In the multitask
mode, we use this ELM task to derive an additional loss. The model is
optimized on a combination of both target loss and this additional loss.
Neither mode is better than the direct fine-tuning in terms of macro. While
the initial results here are not positive, integrating multiple losses when
fine-tuning models can be tricky and requires more systematic experimentation.

Set | True Labels | False Labels | Total
---|---|---|---
Train | 29916 | 16028 | 45944
Dev | 4626 | 2561 | 7187
Table 3: Masked Entity Language Model Dataset Statistics | precision | recall | f1-score | support
---|---|---|---|---
False | 0.90 | 0.72 | 0.80 | 2561
True | 0.86 | 0.96 | 0.91 | 4626
macro avg | 0.88 | 0.84 | 0.85 | 7187
Table 4: Masked ELM on dev set results | Model | Test Standard | Test Frequent
---|---|---|---
|  | Neg | Neut | Pos | MacroF1 | Neg | Neut | Pos | MacroF1
Document |  Supervision | Weighted Random | 0.10 | 0.30 | 0.60 | 0.33 | 0.11 | 0.32 | 0.56 | 0.33
REN | 0.06 | 0.34 | 0.62 | 0.34 | 0.04 | 0.36 | 0.56 | 0.32
Modified REN | 0.13 | 0.38 | 0.56 | 0.36 | 0.10 | 0.45 | 0.46 | 0.34
Discourse HLSTM | 0.22 | 0.43 | 0.59 | 0.41 | 0.18 | 0.45 | 0.5 | 0.38
DAN | 0.18 | 0.50 | 0.65 | 0.44 | 0.11 | 0.46 | 0.51 | 0.36
BERT Classifier | 0.28 | 0.54 | 0.62 | 0.48 | 0.25 | 0.54 | 0.46 | 0.42
Para |  Aggr | BERT | Majority | 0.18 | 0.52 | 0.56 | 0.42 | 0.29 | 0.51 | 0.44 | 0.41
Average | 0.16 | 0.51 | 0.56 | 0.41 | 0.28 | 0.52 | 0.46 | 0.42
Foc |  Rep | BERT Gold Label | 0.82 | 0.78 | 0.87 | 0.82 | 0.79 | 0.74 | 0.82 | 0.78
BERT+ELM Sequential | 0.12 | 0.52 | 0.66 | 0.43 | 0.16 | 0.48 | 0.51 | 0.38
BERT+ELM MultiTask | 0.17 | 0.48 | 0.60 | 0.42 | 0.24 | 0.47 | 0.49 | 0.40
Table 5:  Macro F1-score for different models on each class. There are three
main classes: Negative (Neg), Neutral (Neu), and Positive (Pos). For each
class F1 is reported and for each model macro F1 is reported as it is very
important for the model to predict the minority class (Negative) as well as
the other two classes. The model which has higher Macro-F1 on Dev set is
selected and the performance on two test sets are evaluated. ELM is the short
form of ELM which is discussed in 3.4

###  3.5 Analysis

Document Length Figure 4(a) shows how model performance varies with document
length. Model performance is highest for documents length between 180-300
words, the length that is most prevalent in the training data. For shorter and
longer documents the performance is significantly lower, hinting at the
difficulty in generalizing along the length dimension. For documents longer
than 512, which is the input size limit for BERT, the performance does not
drop too much. This is likely because we have trimmed longer documents in the
training data to 512 tokens. Using models without this length limitation
(e.g., XLNet [Yang et al., 2019] and Longformer [Beltagy et al., 2020]) should
help address this issue directly.
Para vs. Document label agreement Figure 4(c) shows the importance of
agreement between document level labels and paragraph level labels. The higher
the agreement between document and paragraph labels, the higher the macro F1
score. It can be a good test of generalization to see if models can assess
document-level sentiment accurately even in the presence of conflicting local
paragraph-level sentiments.

(a) F1-score vs the number of words in a document for document-level BERT

(b) F1-score for different number of unique entities in a document

(c) F1-score for different percentage of agreements between document level
label and paragraph labels

(d) Comparison of Masked ELM accuracy with sentiment classification task

Figure 4: Analyzing F1 score in different criteria

Number of entities We hypothesized that if there are more entities in a
document then the models will fail because they will have a tough time
aggregating information relevant to the target entity. Figure 4(b) shows the
macro F1 score vs the number of unique entities in a document. It turns out
that only when the number of unique entities is substantially high (more than
10) there is a significant drop in performance. For smaller number of entities
the trend is in the opposite direction.
Target Entity Failure We expect the classification to fail when the model is
unable to reliably predict information about the target entity. The Masked ELM
task is a proxy for this test. Figure 4(d) shows the relation between the
performance of BERT on the sentiment classification and the Masked ELM tasks.
When BERT is able to reliably locate the target entity, i.e., when it has
higher accuracy in Masked ELM task, it is usually also more accurate in
classifying the sentiment towards that entity. This further motivates better
exploration tasks that ensure focusing representations towards the target
entity.

Error Analysis We conducted a manual error analysis to identify the type of
mistakes. We analyzed a random subset of 10% of the documents from development
set which are misclassified by BERT. The results are shown in Figure 5. We see
misleading words, neutral paragraphs, and long documents are most prevalent in
documents where BERT fails. Other minor categories include cases where much of
non-author sentiment is expressed, presence of quotations and multiple
entities. The other category combines errors in finding the main entity, error
in annotation, and other unknown reasons.

##  4 Related Work

###  4.1  Aspect-Level Sentiment Analysis

The goal is to determine the sentiment toward specific aspect of a target such
as product, movie, restaurant, etc [Pang et al., 2008, Schouten and Frasincar,
2016]. While there are broad similarities, the main objective here is to infer
the author’s view towards an entity and not towards different ”aspects” of the
entity. In [Chen et al., 2017], the authors proposed a multiple attention
mechanism to capture sentiment features and then combined the result with a
recurrent neural network. By using the weighted memory network, they captured
the sentiment features automatically and used its capacity to capture
sentiments toward different opinion targets of a sentence. They evaluate their
model on four different sentence level target specific sentiment analysis and
outperforms previous models. ?) proposed a novel model by augmenting the long
short term memory (LSTM) network with a hierarchical attention mechanism
consisting of a target level attention and a sentence-level attention.
Commonsense knowledge is also incorporated into the training for sentiment
classification as an external knowledge. The major difference of aspect level
sentiment analysis and our task is that, in aspect-level the text is talking
about different aspects of one element and the goal is to find the sentiment
towards each aspect. The aspect-level reviews are for short texts and the
opinion towards each aspect is persistent in the text. In entity-based
document-level sentiment analysis the goal is to find the whole author’s view
towards one entity, not different aspect of one entity.

Figure 5: Different types of errors for BERT classifier on dev set.

###  4.2 Document-Level Sentiment Analysis

Many works have studied document-level sentiment analysis [Pang et al., 2002,
Moraes et al., 2013, Lin et al., 2006, Tang et al., 2015]. ?) showed the
effectiveness of sentence level prediction on document level classification.
This paper focuses on models to jointly classify sentiment on multiple levels
of granularity, i.e. on sentence level and document level. The authors try to
show how using the information on single sentences can help the sentiment
prediction task on the whole document. Our work is inspired by recent efforts
that focus on entity-centric sentiment model, which learnt sentiments between
pairs of entities [Choi et al., 2016]. While closely related this task cannot
be re-purposed to infer author sentiment directly.

###  4.3 News Domain Sentiment Analysis

Sentiment analysis in news articles has been studied from various perspectives
for different types of applications [Yu et al., 2013, Ahmad et al., 2007,
Godbole et al., 2007, Balahur et al., 2013]. We propose a new dataset and task
that requires different levels of linguistic features (i.e., discourse units,
sentences, and entities) for accurate sentiment inferences.

###  4.4 Stance Detection

Stance detection is the task of detecting whether the author is in favor of
the target or against it and it is important to determine not only the
author’s stance expressed in her/his debate posts but also the reasons why
s/he supports or opposes the issue under debate [Hasan and Ng, 2014, Zarrella
and Marsh, 2016]. Most of the researches in stance detection field are using
debate posts in forums or political congress debates [Walker et al., 2012,
Hasan and Ng, 2013, Augenstein et al., 2016]. The stance detection texts are
usually not longer than a forum post [Hasan and Ng, 2013] or twitter text
[Mohammad et al., 2016], and the classification task is usually to predict
whether the author is in favor or against the target. In stance detection the
models are looking for favorability towards a given target of interest which
may not be mentioned or targeted in the text [Mohammad et al., 2016,
Augenstein et al., 2016]. But in sentiment analysis the author is explicitly
mentioning the target and it is formulated as finding whether the text is
positive, negative, or neutral towards the main entity. As such, our task is
categorized as sentiment analysis task where the targets are persons and
mentioned directly in the text. However, the authors may not explicitly
express their opinion.

##  5 Conclusions & Discussion

In this work, we introduced author sentiment inference in the news domain, a
new sentiment task and release a challenge dataset. Most information contained
in a news article might be irrelevant for the purposes of inferring the
authors sentiment towards the target entity. This makes the task quite
challenging for both models that operate over global representation and for
models that aggregate local decisions. We hope the task and the dataset will
be a useful resource for the community to build more effective text
classification models.

##  6 Acknowledgements

This work was supported in part by the National Science Foundation under Grant
IIS-1815358. We also thank Google for supporting this research through Google
Cloud Credits program.

## References

  * [Abbasi et al., 2008]  Ahmed Abbasi, Hsinchun Chen, and Arab Salem.  2008\.  Sentiment analysis in multiple languages: Feature selection for opinion classification in web forums.  ACM Transactions on Information Systems (TOIS), 26(3):12.
  * [Ahmad et al., 2007]  Khurshid Ahmad, David Cheng, and Yousif Almas.  2007\.  Multi-lingual sentiment analysis of financial news streams.  In 1st International Workshop on Grid Technology for Financial Modeling and Simulation, volume 26, page 001. SISSA Medialab.
  * [Augenstein et al., 2016]  Isabelle Augenstein, Tim Rocktäschel, Andreas Vlachos, and Kalina Bontcheva.  2016\.  Stance detection with bidirectional conditional encoding.  arXiv preprint arXiv:1606.05464.
  * [Balahur et al., 2013]  Alexandra Balahur, Ralf Steinberger, Mijail Kabadjov, Vanni Zavarella, Erik Van Der Goot, Matina Halkia, Bruno Pouliquen, and Jenya Belyaeva.  2013\.  Sentiment analysis in the news.  arXiv preprint arXiv:1309.6202.
  * [Beltagy et al., 2020]  Iz Beltagy, Matthew E. Peters, and Arman Cohan.  2020\.  Longformer: The long-document transformer.  arXiv:2004.05150.
  * [Chen et al., 2017]  Peng Chen, Zhongqian Sun, Lidong Bing, and Wei Yang.  2017\.  Recurrent attention network on memory for aspect sentiment analysis.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, September. Association for Computational Linguistics.
  * [Choi et al., 2016]  Eunsol Choi, Hannah Rashkin, Luke Zettlemoyer, and Yejin Choi.  2016\.  Document-level sentiment inference with social, faction, and discourse context.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 333–343.
  * [Clark and Manning, 2016]  Kevin Clark and Christopher D Manning.  2016\.  Improving coreference resolution by learning entity-level distributed representations.  arXiv preprint arXiv:1606.01323.
  * [Co, 2017]  HuggingFace Co.  2017\.  neuralcoref.
  * [Dave et al., 2003]  Kushal Dave, Steve Lawrence, and David M Pennock.  2003\.  Mining the peanut gallery: Opinion extraction and semantic classification of product reviews.  In Proceedings of the 12th international conference on World Wide Web, pages 519–528. ACM.
  * [Deng and Wiebe, 2015]  Lingjia Deng and Janyce Wiebe.  2015\.  Mpqa 3.0: An entity/event-level sentiment corpus.  In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1323–1328.
  * [Devlin et al., 2018]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * [Ellis et al., 2014]  Joe Ellis, Jeremy Getman, and Stephanie M Strassel.  2014\.  Overview of linguistic resources for the tac kbp 2014 evaluations: Planning, execution, and results.  In Proceedings of TAC KBP 2014 Workshop, National Institute of Standards and Technology, pages 17–18.
  * [Fang and Zhan, 2015]  Xing Fang and Justin Zhan.  2015\.  Sentiment analysis using product review data.  Journal of Big Data, 2(1):5, Jun.
  * [Finkel et al., 2005]  Jenny Rose Finkel, Trond Grenager, and Christopher Manning.  2005\.  Incorporating non-local information into information extraction systems by gibbs sampling.  In Proceedings of the 43rd annual meeting on association for computational linguistics, pages 363–370. Association for Computational Linguistics.
  * [Godbole et al., 2007]  Namrata Godbole, Manja Srinivasaiah, and Steven Skiena.  2007\.  Large-scale sentiment analysis for news and blogs.  Icwsm, 7(21):219–222.
  * [Hasan and Ng, 2013]  Kazi Saidul Hasan and Vincent Ng.  2013\.  Stance classification of ideological debates: Data, models, features, and constraints.  In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1348–1356.
  * [Hasan and Ng, 2014]  Kazi Saidul Hasan and Vincent Ng.  2014\.  Why are you taking this stance? identifying and classifying reasons in ideological debates.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 751–762.
  * [Henaff et al., 2016]  Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun.  2016\.  Tracking the world state with recurrent entity networks.  arXiv preprint arXiv:1612.03969.
  * [Iyyer et al., 2015]  Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III.  2015\.  Deep unordered composition rivals syntactic methods for text classification.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 1681–1691.
  * [Ji and Eisenstein, 2014]  Yangfeng Ji and Jacob Eisenstein.  2014\.  Representation learning for text-level discourse parsing.  In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 13–24.
  * [Ji and Smith, 2017]  Yangfeng Ji and Noah Smith.  2017\.  Neural discourse structure for text categorization.  arXiv preprint arXiv:1702.01829.
  * [Kulkarni et al., 2018]  Vivek Kulkarni, Junting Ye, Steve Skiena, and William Yang Wang.  2018\.  Multi-view models for political ideology detection of news articles.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3518–3527.
  * [Lin et al., 2006]  Wei-Hao Lin, Theresa Wilson, Janyce Wiebe, and Alexander Hauptmann.  2006\.  Which side are you on?: identifying perspectives at the document and sentence levels.  In Proceedings of the tenth conference on computational natural language learning, pages 109–116. Association for Computational Linguistics.
  * [Ma et al., 2018]  Yukun Ma, Haiyun Peng, and Erik Cambria.  2018\.  Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive lstm.  In Thirty-Second AAAI Conference on Artificial Intelligence.
  * [Marasini et al., 2016]  Donata Marasini, Piero Quatto, and Enrico Ripamonti.  2016\.  Assessing the inter-rater agreement for ordinal data through weighted indexes.  Statistical methods in medical research, 25(6):2611–2633.
  * [McDonald et al., 2007]  Ryan McDonald, Kerry Hannan, Tyler Neylon, Mike Wells, and Jeff Reynar.  2007\.  Structured models for fine-to-coarse sentiment analysis.  In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 432–439.
  * [Mohammad et al., 2016]  Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry.  2016\.  Semeval-2016 task 6: Detecting stance in tweets.  In Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pages 31–41.
  * [Moraes et al., 2013]  Rodrigo Moraes, JoãO Francisco Valiati, and Wilson P GaviãO Neto.  2013\.  Document-level sentiment classification: An empirical comparison between svm and ann.  Expert Systems with Applications, 40(2):621–633.
  * [Pak and Paroubek, 2010]  Alexander Pak and Patrick Paroubek.  2010\.  Twitter as a corpus for sentiment analysis and opinion mining.  In LREc, volume 10, pages 1320–1326.
  * [Pang et al., 2002]  Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan.  2002\.  Thumbs up?: sentiment classification using machine learning techniques.  In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pages 79–86. Association for Computational Linguistics.
  * [Pang et al., 2008]  Bo Pang, Lillian Lee, et al.  2008\.  Opinion mining and sentiment analysis.  Foundations and Trends® in Information Retrieval, 2(1–2):1–135.
  * [Schouten and Frasincar, 2016]  Kim Schouten and Flavius Frasincar.  2016\.  Survey on aspect-level sentiment analysis.  IEEE Transactions on Knowledge & Data Engineering, (1):1–1.
  * [Singh et al., 2013]  Vivek Kumar Singh, Rajesh Piryani, Ashraf Uddin, and Pranav Waila.  2013\.  Sentiment analysis of movie reviews: A new feature-based heuristic for aspect-level sentiment classification.  In Automation, computing, communication, control and compressed sensing (iMac4s), 2013 international multi-conference on, pages 712–717. IEEE.
  * [Socher et al., 2011]  Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning.  2011\.  Semi-supervised recursive autoencoders for predicting sentiment distributions.  In Proceedings of the conference on empirical methods in natural language processing, pages 151–161. Association for Computational Linguistics.
  * [Son et al., 2018]  Youngseo Son, Nipun Bayas, and H Andrew Schwartz.  2018\.  Causal explanation analysis on social media.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3350–3359.
  * [Tang et al., 2015]  Duyu Tang, Bing Qin, and Ting Liu.  2015\.  Document modeling with gated recurrent neural network for sentiment classification.  In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 1422–1432.
  * [Turney, 2002]  Peter D Turney.  2002\.  Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews.  In Proceedings of the 40th annual meeting on association for computational linguistics, pages 417–424. Association for Computational Linguistics.
  * [Walker et al., 2012]  Marilyn A Walker, Pranav Anand, Robert Abbott, and Ricky Grant.  2012\.  Stance classification using dialogic properties of persuasion.  In Proceedings of the 2012 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 592–596. Association for Computational Linguistics.
  * [Wiebe et al., 2005]  Janyce Wiebe, Theresa Wilson, and Claire Cardie.  2005\.  Annotating expressions of opinions and emotions in language.  Language resources and evaluation, 39(2-3):165–210.
  * [Yang et al., 2019]  Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le.  2019\.  Xlnet: Generalized autoregressive pretraining for language understanding.  CoRR, abs/1906.08237.
  * [Ye and Skiena, 2019]  Junting Ye and Steven Skiena.  2019\.  Mediarank: Computational ranking of online news sources.  arXiv preprint arXiv:1903.07581.
  * [Yu et al., 2013]  Yang Yu, Wenjing Duan, and Qing Cao.  2013\.  The impact of social and conventional media on firm equity value: A sentiment analysis approach.  Decision Support Systems, 55(4):919–926.
  * [Zarrella and Marsh, 2016]  Guido Zarrella and Amy Marsh.  2016\.  Mitre at semeval-2016 task 6: Transfer learning for stance detection.  arXiv preprint arXiv:1606.03784.