11institutetext: Japan Advanced Institute of Science and Technology, Ishikawa
9231292, JAPAN
11email<EMAIL_ADDRESS>
# ILoveEye: Eyeliner Makeup Guidance System with Eye Shape Features

Hange Wang  Haoran Xie  Kazunori Miyata

###### Abstract

Drawing eyeliner is not an easy task for whom lacks experience in eye makeup.
Everyone has a unique pair of eyes, so they need to draw eyeliner in a style
that suits their eyes. We proposed ILoveEye, an interactive system that
supports eye-makeup novices to draw natural and suitable eyeliner. The
proposed system analyzes the shape of the user’s eyes and classifies the eye
types from camera frame. The system can recommend the eyeliner style to the
user based on the designed recommendation rules. Then, the system can generate
the original patterns corresponding to the eyeliner style, and the user can
draw the eyeliner while observing the real-time makeup guidance. The user
evaluation experiments are conducted to verify that the proposed ILoveEye
system can help some users to draw reasonable eyeliner based on the specific
eye shapes and improve their eye makeup skills111This is a pre-peer review,
pre-print version of this article. The final authenticated version is
available online at proceedings of HCII 2022(International Conference on
Human-Computer Interaction).

###### Keywords:

Eyeliner Makeup Eye Features Analysis Makeup System Interactive Guidance.

##  1 Introduction

Makeup is a daily skill like other creative activities such as painting and
sculpting, which requires the users to master skillful techniques of makeup
and possess a high-level understanding of makeup based on facial features. Eye
makeup plays an important role in the daily facial makeup. However, it is
still a challenging issue for common users with few makeup experiences to
achieve satisfactory outcome. Usually, the users often seek makeup tutorials
from video sites or social medias, such as Youtube and TikTok. These video
tutorials commonly do not match personal facial features, which have
significant relationship with the strategies of eye makeup. In this work, we
focus on the guidance system for eyeliner makeup which is the basic facial
makeup in our daily lives.

About makeup guidance systems, it is a common approach to provide the
corresponding makeup suggestions based on the user’s facial features and show
the anticipated makeup effect on the face in real time. We found that it is
beneficial to provide visual recommendation makeup styles and visual makeup
tutorials for whom has few knowledge and experience of makeup. This work aims
to provide intuitive and effective eyeliner makeup guidance in real time.

In this paper, we propose, ILoveEye, an interactive rule-based makeup system
based on the analysis of user’s eye shape features for less-experienced users
as shown in Figure 1. To obtain the feature points of eyes, we utilize state-
of-the-art deep learning based facial recognition models in this work. To
classify the features of eye shapes, we conduct eye shape classification and
extract the features of the eye contour. After analyzed the typical eye shapes
including almond, round, downturned, and close-set eyes, we propose a rule-
based classification model to obtain the labels of the eye shape features with
three feature values of the eyes: eye aspect ratio, outer corner orientation,
and eye distance. For the makeup guidance, we investigate rules of typical
eyeliner styles according to the relationship between eye shape and eyeliner
styles. In terms of the eye feature points obtained from the proposed system,
we adopt the classification rules to match current feature points of user’s
eye. The proposed system can reproduce the eyeliner styles and display in
real-time as guidance to support the user to complete the eyeliner makeup.

Figure 1: The proposed makeup guidance framework of ILoveEye.

We designed and conducted an evaluation experiment to verify the feasibility
of the proposed ILoveEye system. In our user study, we invited participants to
join the experiments, who were less-experienced in facial makeup with few
skills of drawing eyeliner. To clarify the relationship between system
evaluation and user’s skill background, we designed a questionnaire which were
conducted after experienced the proposed system. From the questionnaire
results, we found that the proposed system could effectively classify the eye
shape and support the user to draw eyeliner interactively. In addition, we
analyzed the Pearson correlation coefficients between the user’s makeup
background and system evaluation. The analysis results indicated a significant
and positive correlation between the frequency of wearing makeup and the skill
improvement after using the system. We conclude that the users who normally
wear makeup are acquired with experience and knowledge of makeup, thus they
could draw better eyeliner after learning the exact contours of their eye
shape features using the proposed system.

##  2 Related Work

###  2.1 Eye Features

The analysis of eye features plays an important role in eye makeup system.
Bhat et al. proposed a method for detecting eye contours by using active shape
model [4]. An efficient method for detecting eye contours was proposed in
real-time with the dataset of the eye contours [7]. For shape classification
of eyes, Alzahrani et al. summarized the characteristics of eye shapes and
designed eye shape classification rules [3]. In this work, we originally
design the classification rules by collecting relevant information and confirm
the effectiveness of our classification rules by conducting evaluation study.

###  2.2 Guidance System

Along the development of augmented reality and image processing technologies,
the guidance systems for supporting our daily activities have been explored
extensively. The deep learning based human mesh modeling was used to compare
the user’s postures and target ones for core training [20]. To help take a
good selfie, a voice guidance system was proposed with crowdsourcing
evaluation of photo aesthetic [6]. The spatial augmented reality has been used
for calligraphy practice [10], golf training [11] and dance support [9]. To
meet the user’s intentions in design activities, the user sketches were
utilized for domino arrangement [16] and lunch box decoration [19]. In this
work, we especially focus on the guidance system for eye makeup using
augmented reality approach.

###  2.3 Makeup Guidance

With the development of facial recognition technologies, it becomes feasible
to provide suitable makeup guidance based on facial features. Chiocchia et al.
developed a mobile application to help users select cosmetics and showed users
makeup tutorials [5]. Face features can be obtained for makeup image synthesis
with different makeup references [8]. The face dataset was constructed to
analyze the facial attractiveness for recommend system [12]. A rule-based
system was proposed to automatically classify the faces and suggest the makeup
approach [1]. iMake proposed a novel eye-makeup design system to extract the
color and shape features [15]. A mirror system utilizing augmented reality
(AR) was proposed to capture and analyze the facial features in real-time
[17]. An interactive mobile application was proposed for makeup tutorials [2].
A makeup instruction with AR was proposed to provide special makeup tools
[18]. In this work, we aim to provide the real-time guidance to support users
for drawing better eyeliner and improving makeup skills.

##  3 System Overview

Figure 1 shows the overall workflow of the proposed ILoveEye system which
consists of three main parts with deep learning based face detection. First,
the proposed system detects eyes using the normal web camera and obtains the
geometric information of the eye contours. Second, we design a rule-based
model using geometric features to classify eye shapes. Finally, the proposed
system generates the eyeliner makeup guidance based on the previously analyzed
classification results displayed on the screen in real-time. The proposed
ILoveEye system enables users to observe the guidance patterns on the screen
while drawing suitable eyeliners.

###  3.1 Eye Detection

About eye shape detection, we adopted the open-source deep learning based face
detector, MediaPipe [13], which contains a pre-trained face recognition model
with high efficiency and accuracy. The detection output is a face mesh map as
shown in Figure 2 with 486 landmarks. From this face mesh, 16 feature points
are selected to describe the eye contour. Through our practical testing, we
found that this detector can effectively identify the shape of the eye contour
with high accuracy to meet our research target.

Figure 2: Eye contour landmarks in human face mesh.

To implement the eye detection module, we first invoke the connected camera
(60 fps in our prototype). To get the shape of the eyes accurately, the
proposed system asks the user to save the mesh map of the current frame. The
proposed system selects the landmarks of the eye parts from the saved mesh
map. We found that the camera angle may affect the detection results, so we
asked the user to sit at a proper distance and allow the user to adjust the
camera angle until the satisfied eye contour is achieved.

###  3.2 System Workflow

After analyzed the eye features, the proposed system then determines the
eyeliner styles in terms of eye aspect ratio, outer corner orientation, and
eye distance as shown in Figure 3. If the eye size is labeled with “small” or
“average”, the system recommends Style-Basic. If the aspect ratio is labeled
with “big”, the proposed system recommends Style-Basic with the lower eyeliner
thickness. The system can analyze the orientation of the eye outer corner to
recommend Style-Winged, Style-Drop and Style-Extend. The system can recommend
lower eyeliner styles: Style-Inner and Style-Outer, depending on eye
distances. Finally, the proposed system combines all recommended eyeliner
styles to generate a polygon as a mask, which is visualized in orange color
for makeup guidance (Figure 3).

Figure 3: Workflow of the recommendation function.

###  3.3 Eye Features Analysis

We reviewed different typical eye types, such as round eyes: larger and more
circular; close-set eyes: less space between eyes; down-turned eyes: taper
downward at the outer corner [3]. We found that the eye shapes are related to
the size of the eye, the angle of the eye outer corner, and the distance
between the two eyes. Therefore, we set these three features as the
classification conditions for determining eye types.

Figure 4: Eye feature parameters: (a) width and height of eye; (b) outer
corner orientation; (c) eye distance.

####  3.3.1 Aspect Ratio

In the obtained eye contour landmarks (the right eye as an example), we define
the leftmost and rightmost points $p0$ and $p8$ as the positions of the head
and tail positions. We calculate the lengths of the perpendicular lines from
$p1-p7$, and $p9-p15$ respectively, then find the maximum values of the upper
part and lower part, and sum them as eye height (Figure 4(a)). The length of
$p0-p8$ line is considered as eye width. The ratio of eye width and height is
calculated as aspect ratio $a$. We obtained a relatively reasonable range for
eye styles: $a\in[2.75,3.00]$ for average eye, small long eyes for $a>3.00$,
big round eyes for $a<2.75$.

####  3.3.2 Outer Corner Orientation

We found that the angle of the eye’s outer corner can be used as a reliable
reference, for determining up-turned or down-turned eyes. As shown in Figure
4(b), $p4$ and $p12$ denote the highest and lowest points. Line $p0-p8$
divides the outer corner angle into two angles. The upper angle $\alpha$ is
defined as included angle of lines $p4-p8$ and $p0-p8$, and lower angle
$\beta$ with $p12-p8$. If $\alpha>\beta$, the outer corner is determined to be
down toward and the eye will be labeled as down-turned eye. On the contrary,
the eye will be labeled as upturned eye.

####  3.3.3 Eye Distance

We measure the distance between point most left and right points as eye-
distance $D_{e}$. The average distance of two eyes $D_{avg}$ (Figure 4(c)) is
calculated. If $D_{e}>1.05\times D_{avg}$, the eyes are considered as open-set
type and labeled as “open”; If $D_{e}<0.95\times D_{avg}$, the eyes are
considered as close-set type and labeled as “close”. Otherwise, the eyes are
labeled as “average”.

##  4 Makeup Guidance

Eye makeup uses eyeliner to make eyes look bigger and modify the eye shape
[14]. Generally, there are two typical eyeliner types: outer and inner
eyeliners. However, the inner eyeliner is applied in eyelashes roots which is
difficult to detect by the proposed detection method. Therefore, we focus on
the outer eyeliner in this work. We defined the typical styles of eyeliners
related to eye shape features as shown in Figure 5.

Figure 5: Eyeliner styles defined in the proposed ILoveEye system.

###  4.1 Eyeliner Styles

####  4.1.1 Style-Basic

Style-Basic is a thick line along the upper eyelid that starts from the inner
corner of the eye to the outer corner (Figure 6(a)). In this work, a thick
line is generated along with the shape of the upper eyelid. We calculate the
mid-point for each line segment of the upper eyelid. Then, we set the
perpendicular line of each line segments from mid-points to $p0’,p1’…p7’$, and
the length of each perpendicular line is defined as eyeliner thickness $h$. By
connecting the points of the upper eyelid, the closed polygon is the basic
upper eyeliner style as Style-Basic.

####  4.1.2 Style-Winged

For the upturned-winged eyeliner pattern, we define an outermost point $E$. As
shown in Figure 6(b), we measure typical samples of winged eyeliner. The angle
of the eyeliner tail is about 15 degrees for natural look, so we define the
average angle between $E-p8$ and $p0-p8$ as 15 degrees. The length of the wing
is set to be about 12% of the eye-width. The wing length is also suitable for
other wing eyeliner styles. The coordinates of $E$ point is calculated by
trigonometric functions. By connecting the Style-Basic to the point $E$ to
form a closed polygon, we get the upturned wing eyeliner as Style-Winged.

####  4.1.3 Style-Drop

As shown in Figure 6(c), we get a droopy eye-line style by defining a point
$E$. $p8-E$ is the extension of $p4-p8$, and the length of $p8-E$ is 12% of
eye-width in Style-Winged. The closed polygon formed by connecting Style-Basic
to point $E$ is regarded as Style-Drop.

####  4.1.4 Style-Extend

As shown in Figure 6(d), Style-Extend is the eyeliner style that the end of an
eyeliner extends out horizontally. We first define the outermost point of the
eye-liner as point $E$. Points $p0$, $p8$, and $E$ are on the same horizontal
line, and the length of $E-p8$ is 12% of eye-width. Finally, we connect Style-
Basic with point $E$ to get Style-Extend.

####  4.1.5 Basic Lower Eyeliner

The lower eyeliner is always thinner than the upper eyeliner and applied on
corner sides. For lower eyeliner in outer corner side (Figure 6(e)), we
connect the wing of upper eyeliner, and select $p8$, $p9$, $p10$, $p11$ to
create a lower eyeliner style on the inner corner side. We decrease the value
of $h$ from large to small in equal proportion to make the shape appearance
from big to small. We connect the upper eyeliner and select $p13$, $p14$,
$p15$, $p0$ to create a lower eyeliner style on the inner corner side. We
found that the value of h as h/3 of the outer side can make this pattern look
thinner than the outer side. We defined these two lower eyeliner styles as
“Style-Inner” and “Style-Outer”.

Figure 6: Generation of different eyeliner styles. Figure 7: Merging types of
eyeliner styles.

###  4.2 Merging Types

For the lower eyeliner styles, the system selects the recommended style and
merges it with upper eyeliner styles. In the case of merging Style-Outer with
Style-Winged, the system connects point $E$ to lower $p8’$, and gets the
closed polygon as the result for guidance as shown in Figure 7(a).

In the case of merging Style-Inner and Style-Basic (Figure 7(b)), the system
first calculates a fitting curve by $p0-p8$ and $p0’-p7’$, then finds the
cross-point $E$ with line $p0-p8$. We obtain a closed polygon by connecting
the point $E$, Style-Basic and Style-Inner as shown in Figure 7(b).

###  4.3 User Interface

Figure 8 shows the user interface of the proposed ILoveEye system. The
proposed system creates new windows of the left and right eyes separately and
enlarges them to make the user observe the guidance and their eyes clearly.
For the recognized eye contour, the system can display it that the user can
find how well their eyes are recognized.

The user can select one window to save the feature information of the current
frame through keyboard input. Because the feature information of both eyes is
saved at the same time, the recognized eye shape may be in error with the real
shape due to the shooting angle and lens distortion of the camera. Therefore,
we asked the user save the eye features at the correct angle while looking
straight to the camera.

Figure 8: Experiment Environment and user interface.

##  5 System Evaluation

The prototype system was implemented on a desktop computer with Windows 10,
Intel i7-9700 CPU, GeForce RTX 2060 SUPER GPU. The proposed system was
developed in Python with OpenCV ver4.5.2, MediaPipe ver0.8.6, and we set up
Logitech HD Pro camera C920 on the table to capture user’s face.

###  5.1 Experiment Procedure

We conduct a comparison study that participants are asked to draw eye makeup
with and without the proposed system. After the eye-makeup is applied, we
provided makeup remover for the participants to remove their makeup and start
to draw makeup using the proposed system. We first introduced the system
workflow and asked users to draw an eyeliner by referring to the makeup
guidance in orange color. All participants were asked to look at the camera,
the monitor can display the participant’s face and independent images of the
two eyes. The participants can adjust the angle of the camera until the
satisfied eye recognition results were achieved. The analyzed eye features
results were prompted to the participant in text. We provided participants
with two eyeliners (one gel eyeliner and one liquid eyeliner) to draw
eyeliner. When the participant completed the eye makeup, we counted the time
cost and archived makeup images.

###  5.2 Evaluation Method

In the user study, we adopted both subjective evaluation and objective
evaluation to verify the proposed ILoveEye system. Regarding to the subjective
evaluation, we designed a questionnaire to confirm the user experience and the
system usefulness. After collected the basic information of the participants
(gender and age), we first investigated the users’ knowledge about eye makeup.
The question items of the questionnaire are listed as below.

  * •

Q1: Do you wear makeup?

  * •

Q2: Do you watch makeup tutorials (videos, magazines, blogs)?

  * •

Q3: How do you evaluate the level of your makeup skill?

  * •

Q4: How do you evaluate the level of your eye-makeup skill?

  * •

Q5: Do you draw eye-makeup by considering your eye shape?

  * •

Q6: Do you consider this system describe your eye shape correctly?

  * •

Q7: Do you consider this system support you to draw eyeliner?

  * •

Q8: Do you consider this system help you to draw better eyeliner than by
yourself

  * •

Q9: Do you think this system improve your drawing eyeliner skill?

All these questions were evaluated with 5-point Likert scale where 1 denotes
strongly disagree and 5 for strongly agree. Finally, we collected the free
feedback from the participants for any advice or improvement. We invited 8
female graduate students, around 25-year-old. All participants were asked to
join without eye makeup, and the few experience of eye makeup was required.

##  6 Results

Figure 9: Results of questionnaire in our user study.

###  6.1 Subjective Evaluation

The results of the questionnaire are shown in Figure 9. We found that most
participants did not wear makeup very often (Q1: average = 2.63) and were not
confident in their makeup skills (Q3: average = 2.50; Q4: average = 2.25). The
average scores of questions from Q6 to Q9 were Q6: 3.88; Q7: 3.88; Q8: 3.63;
Q9: 4.13. We found that the overall score of our system was above 3.0, which
means our system can help users to draw better eye makeup and improve their
makeup skills. We noticed that one participant rated lower scores on Q6, Q7,
and Q8, and confirmed that it was difficult to satisfy experienced users well
with our makeup guidance, who already had a well-tested routine for drawing
eye makeup.

We analyzed the correlation between participants’ background of makeup
knowledge and evaluation of the system by the Pearson correlation coefficients
for Q1-Q5 and Q6-Q9 as shown in Figure 10. We found a significant and positive
correlation between Q1 and Q9 (p = 0.71). This means that the participants who
wear makeup frequently can feel the skill improvement with the proposed
system. We observed that participants who normally wear makeup are acquired
with some experience and knowledge of makeup. When they learn the exact
contours of their eye features, it can help them to draw eyeliner afterward.
In addition, one participant reported that a better experience for improving
makeup skills than watching tutorial videos was obtained because the system
could describe and show the eye features.

Figure 10: Pearson coefficients for Q1-Q5 and Q6-Q9. Figure 11: Comparison of
completion time in our user study.

###  6.2 Objective Evaluation

In the objective evaluation, we compared the makeup completion time and makeup
effect with and without the proposed system as shown in Figure 11. We found
that after using our system, all users’ completion time were increased. The
users spent more time to find the location of the guidance pattern and
consider how to reproduce the guidance pattern while using the system. In
addition, we analyzed the correlation between completion time, and the
questions in the questionnaire. We found a significant negative correlation
between Q2 and completion time without system (p = -0.723), which means that
the less frequently people watched makeup tutorials, the more time they spent
on completing their eyeliner.

From the eye makeup results in both conditions as shown in Figure 12, we found
that the eye makeup which refers to the makeup guidance was generally thicker
and more clearly than those without using the system. It is verified that the
proposed makeup guidance is easy to understand and could be reproduced by the
users. The detailed comparison results are shown in Figure 13. Observing the
comparison results of winged parts, we found that the eyeliners drawn by the
participants with the proposed system look thicker and much clearer. After
reviewed the evaluation of system interactivity, it is verified that the
makeup guidance provided by the proposed system was easy to understand and
could be reproduced by users. Basically, these thicker and much clear
eyeliners make users’ eyes became visually bigger.

Figure 12: Comparison of makeup effect by participants. Figure 13:  The
difference of eyeliner’s wing (Left without-system; Right: with-ILoveEye).

##  7 Discussion

From the interviews with participants and free comments in questionnaires, we
found that the proposed system can effectively help users with less makeup
experience to draw a better eyeliner that suits the feature of eye shape.
Besides, from the collected participants feedback, we found that the proposed
system could help users understand their eye features and improve their eye-
makeup skills. However, most recommended eyeliner styles to the participants
were Style-Wing, Style-Extend and Style-Outer, which means that they have
similar eye shapes. All participants in this study were Chinese and Japanese,
it would be better to verify the proposed system for more nationalities. For
the eye recognition, it was difficult to achieve accurate recognition results
due to camera angle, which was adjust the camera angle manually. Although the
proposed system can capture the eye contour accurately, the detection of
eyelids was very limited which means that the system may have difficulty for
users when eyelids heavily affect the visual effect of the eyes.

In our user study, the users had to keep a distance from the monitor so that
the nearsighted users may have difficulty to see the exact location following
the guidance. The frame rates were sometimes low for the smoothness of
observing the guidance. One participant reported that it would be helpful if
the system could provide more recommended solutions for users to choose. For
example, a scene choice function can be added for different usages such as the
party, business and shopping. It would be promising to provide a face preview
with the applied makeup styles.

##  8 Conclusion

In this work, we proposed ILoveEye, an eyeliner guidance system based on the
analysis of eye shape. This system captures the user’s eyes by a camera and
recognizes the eye contours for analyzing the eye shape. The proposed system
can provide a recommended eyeliner guidance that users can draw eyeliners by
referring to guidance. We utilized a deep learning based model to recognize
the eye contour. We proposed the rule-based model to classify the eye shape
features. We designed a user study to verify the effectiveness of the proposed
system. From the user study, our system can effectively identify the user’s
eyes and recognize the contours of the eyes, and the user can follow the
guidance to complete the eye makeup. The system provided an effective guidance
function to help users draw better eye makeup. We found that users were
satisfied with recommended guidance and the user with more makeup experience
could obtain the improvement for their makeup skills.

To improve the user experience of our proposed system and help users draw
eyeliner, there are some possible future works. In this study, the facial
recognition model could not effectively identify the eyelids when the system
recognized the user’s eye contour, which may decrease the classification
accuracy of the user’s actual eye shape. To solve this issue, we plan to build
a novel dataset containing images of eyes with labeled eyelid shapes and train
a supervised learning model to optimize the effect of eye shape recognition.
Furthermore, some participants reported that our proposed system provided few
eyeliner styles, which may not be suitable for rare eye shapes, it is worth to
designing more eyeliner styles as makeup guidance in the future to meet the
user diversity. We think that the proposed ILoveEye system can be extended to
improve other makeup skills such as eyebrows makeup.

## Acknowledgement

The authors thank the anonymous reviewers for their valuable comments. This
project has been partially funded by JAIST Research Grant and JSPS KAKENHI
grant JP20K19845, Japan.

## References

  * [1] Alashkar, T., Jiang, S., Fu, Y.: Rule-based facial makeup recommendation system. In: 2017 12th IEEE International Conference on Automatic Face Gesture Recognition (FG 2017). pp. 325–330 (2017). https://doi.org/10.1109/FG.2017.47
  * [2] Almeida, D.R.O.d., Guedes, P.A., Silva, M.M.O.d., Buarque Vieira e Silva, A.L., Lima, J.P.S.d.M., Teichrieb, V.: Interactive makeup tutorial using face tracking and augmented reality on mobile devices. In: 2015 XVII Symposium on Virtual and Augmented Reality. pp. 220–226 (2015). https://doi.org/10.1109/SVR.2015.39
  * [3] Alzahrani, T., Al-Nuaimy, W., Al-Bander, B.: Integrated multi-model face shape and eye attributes identification for hair style and eyelashes recommendation. Computation 9(5) (2021). https://doi.org/10.3390/computation9050054, https://www.mdpi.com/2079-3197/9/5/54
  * [4] Bhat, S., Savvides, M.: Evaluating active shape models for eye-shape classification. In: 2008 IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 5228–5231 (2008). https://doi.org/10.1109/ICASSP.2008.4518838
  * [5] Chiocchia, I., Rau, P.L.P.: Facial feature recognition system development for enhancing customer experience in cosmetics. In: Rau, P.L.P. (ed.) Cross-Cultural Design. Experience and Product Design Across Cultures. pp. 27–40. Springer International Publishing, Cham (2021)
  * [6] Fang, N., Xie, H., Igarashi, T.: Selfie guidance system in good head postures. In: ACM International Conference on Intelligent User Interfaces Workshops (2018)
  * [7] Fuhl, W., Santini, T., Kasneci, E.: Fast and robust eyelid outline and aperture detection in real-world scenarios. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 1089–1097 (2017). https://doi.org/10.1109/WACV.2017.126
  * [8] Guo, D., Sim, T.: Digital face makeup by example. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. pp. 73–79 (2009). https://doi.org/10.1109/CVPR.2009.5206833
  * [9] He, Y., Zheng, X., Yagami, A., Peng, Y., Yoshida, S., Xie, H., Kanai, H., Miyata, K.: Interactive dance support system using spatial augmented reality. In: 2021 Nicograph International (NicoInt). pp. 27–33 (2021). https://doi.org/10.1109/NICOINT52941.2021.00013
  * [10] He, Z., Xie, H., Miyata, K.: Interactive projection system for calligraphy practice. In: 2020 Nicograph International (NicoInt). pp. 55–61 (2020). https://doi.org/10.1109/NicoInt50878.2020.00018, https://doi.org/10.1109/NicoInt50878.2020.00018
  * [11] Ikeda, A., Tanaka, Y., Hwang, D.H., Kon, H., Koike, H.: Golf training system using sonification and virtual shadow. In: ACM SIGGRAPH 2019 Emerging Technologies. SIGGRAPH ’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3305367.3327993, https://doi.org/10.1145/3305367.3327993
  * [12] Liu, L., Xu, H., Xing, J., Liu, S., Zhou, X., Yan, S.: ”wow! you are so beautiful today!”. In: Proceedings of the 21st ACM International Conference on Multimedia. p. 3–12. MM ’13, Association for Computing Machinery, New York, NY, USA (2013). https://doi.org/10.1145/2502081.2502126, https://doi.org/10.1145/2502081.2502126
  * [13] Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C., Yong, M.G., Lee, J., Chang, W., Hua, W., Georg, M., Grundmann, M.: Mediapipe: A framework for building perception pipelines. CoRR abs/1906.08172 (2019), http://arxiv.org/abs/1906.08172
  * [14] Matsushita, S., Morikawa, K., Yamanami, H.: Measurement of eye size illusion caused by eyeliner, mascara, and eye shadow. Journal of cosmetic science 66 3, 161–74 (2015)
  * [15] Nishimura, A., Siio, I.: Imake: Eye makeup design generator. In: Proceedings of the 11th Conference on Advances in Computer Entertainment Technology. ACE ’14, Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/2663806.2663823, https://doi.org/10.1145/2663806.2663823
  * [16] Peng, Y., Mishima, Y., Igarashi, Y., Miyauchi, R., Okawa, M., Xie, H., Miyata, K.: Sketch2domino: Interactive chain reaction design and guidance. In: 2020 Nicograph International (NicoInt). pp. 32–38 (2020). https://doi.org/10.1109/NicoInt50878.2020.00013, https://doi.org/10.1109/NicoInt50878.2020.00013
  * [17] Rahman, A.S.M.M., Tran, T.T., Hossain, S.A., Saddik, A.E.: Augmented rendering of makeup features in a smart interactive mirror system for decision support in cosmetic products selection. In: 2010 IEEE/ACM 14th International Symposium on Distributed Simulation and Real Time Applications. pp. 203–206 (2010). https://doi.org/10.1109/DS-RT.2010.30
  * [18] Treepong, B., Mitake, H., Hasegawa, S.: Makeup creativity enhancement with an augmented reality face makeup system. Comput. Entertain. 16(4) (nov 2018). https://doi.org/10.1145/3277452, https://doi.org/10.1145/3277452
  * [19] Xie, H., Peng, Y., Wang, H., Miyata, K.: Sketchmehow: Interactive projection guided task instruction with user sketches. In: HCI International 2021 - Late Breaking Papers: Cognition, Inclusion, Learning, and Culture. pp. 513–527. Springer International Publishing, Cham (2021)
  * [20] Xie, H., Watatani, A., Miyata, K.: Visual feedback for core training with 3d human shape and pose. In: 2019 Nicograph International (NicoInt). pp. 49–56 (2019). https://doi.org/10.1109/NICOInt.2019.00017


$U=\begin{bmatrix}U^{(1)}\\\ \vdots\\\ U^{(m)}\end{bmatrix}$ with
$U^{(i)}~{}\in~{}\mathcal{O}(d+1,(d+1)m)$. Let $\\{h_{j}\\}_{j\in[m]}$ with
$h_{j}(x)=f_{\mathrm{nc}}(U^{(j)}x;\alpha,d)$ and
$h=\frac{1}{m}\sum_{j=1}^{m}h_{j}$. Let $x\in\mathbb{R}^{(d+1)m}$ and
$y^{(j)}=U^{(j)}x\in\mathbb{R}^{d+1}$. Let
$\mathcal{I}=\\{j\in[m],y^{(i)}_{d}=y^{(i)}_{d+1}=0\\}$. Then it holds

$\|\nabla
h(x)\|^{2}\geq\frac{\alpha^{\frac{3}{2}}|\mathcal{I}|}{16m^{2}}\enspace.$

###### Proof.

We have

$\displaystyle\|\nabla h(x)\|^{2}$
$\displaystyle=\left\|\frac{1}{m}\sum_{j=1}^{d}\nabla h_{j}(x)\right\|^{2}$
$\displaystyle=\left\|\frac{1}{m}\sum_{j=1}^{m}(U^{(j)})^{\top}\nabla
f_{\mathrm{nc}}(U^{(j)}x;\alpha,d)\right\|^{2}$
$\displaystyle=\frac{1}{m^{2}}\left\|\sum_{j=1}^{m}(U^{(j)})^{\top}\nabla
f_{\mathrm{nc}}(U^{(j)}x;\alpha,d)\right\|^{2}$
$\displaystyle\qquad+\frac{2}{m^{2}}\sum_{\begin{subarray}{c}j,l=1\\\ j\neq
l\end{subarray}}^{m}\nabla
f_{\mathrm{nc}}(U^{(j)}x;\alpha,d)^{\top}U^{(l)}(U^{(j)})^{\top}\nabla
f_{\mathrm{nc}}(U^{(j)}x;\alpha,d)$
$\displaystyle=\frac{1}{m^{2}}\sum_{j=1}^{m}\left\|(U^{(j)})^{\top}\nabla
f_{\mathrm{nc}}(U^{(j)}x;\alpha,d)\right\|^{2}$

where the last equality comes from the fact that for $j\neq l$,
$U^{(l)}(U^{(j)})^{\top}=0$ since
$U~{}\in~{}\mathcal{O}~{}((d~{}+~{}1)m,(d+1)m)$. Now, using the third part of
Proposition B.6, we get

$\displaystyle\|\nabla h(x)\|^{2}$
$\displaystyle\geq\frac{1}{m^{2}}\sum_{j\in\mathcal{I}}\left\|(U^{(j)})^{\top}\nabla
f_{\mathrm{nc}}(U^{(j)}x;\alpha,d)\right\|^{2}$
$\displaystyle\geq\frac{1}{m^{2}}\sum_{j\in\mathcal{I}}\left\|\nabla
f_{\mathrm{nc}}(U^{(j)}x;\alpha,d)\right\|^{2}$
$\displaystyle\geq\frac{\alpha^{\frac{3}{2}}|\mathcal{I}|}{16m^{2}}\enspace.$

∎

###  B.2 Main proof

Now we are ready to prove 2.

###### Proof.

We consider $U\in\mathcal{O}((T+1)m,(T+1)m)$ and we denote

$U=\begin{bmatrix}U^{(1)}\\\ \vdots\\\ U^{(m)}\end{bmatrix}$

with
$U^{(j)}=(u^{(j)}_{1},\dots,u^{(j)}_{T+1})^{\top}~{}\in~{}\mathcal{O}(T+1,(T+1)m)$.

For $j\in[m]$, we choose
$\overline{F}_{j}:\mathbb{R}^{(T+1)m+(T+1)m}\to\mathbb{R}$ defined by

$\overline{F}_{j}(z,x)=f_{\mathrm{nc}}(U^{(j)}z;\alpha,T)$

and we set $\overline{F}=\frac{1}{m}\sum_{j=1}^{m}\overline{F}_{j}$. We also
define for $i\in[n]$ $\overline{G}_{i}(z,x)=\frac{1}{2}\|z-x\|^{2}$,
$\overline{G}=\frac{1}{n}\sum_{i=1}^{n}\overline{G}_{i}$,
$\overline{z}^{*}(x)=\operatorname*{arg\,min}_{z}\overline{G}(z,x)$ and
$\overline{h}(x)=\overline{F}(\overline{z}^{*}(x),x)=f_{\mathrm{nc}}(U^{(j)}x;\alpha,T)$.
By Proposition B.6, $\overline{F}_{j}$ is $4+\frac{\alpha c}{m}$ smooth, and
$\overline{G}_{i}$ is $1$-smooth and $1$-strongly convex.

We have

$\overline{h}(0)-\inf_{x}\overline{h}(x)\leq\sqrt{\alpha}+10\alpha T\enspace.$

We finally consider $F_{j}(z,x)=\lambda_{F}\overline{F}_{j}(z/\beta,x/\beta)$,
$G_{i}(z,x)=\lambda_{G}\overline{G}_{i}(z/\beta,x/\beta)$. As a consequence,
we have $z^{*}(x)=\operatorname*{arg\,min}G=\overline{z}^{*}(x)$ and
$h(x)=F(z^{*}(x),x)$. We also consider a _fixed_ indices sequence
$(i_{t},j_{t})$. We set

$\displaystyle\alpha=\min\left\\{1,\frac{m}{c}\right\\},\quad\lambda_{F}=\frac{160m\epsilon}{L^{F}_{1}\alpha^{3/2}},\quad\beta=\sqrt{5\lambda_{F}/L^{F}_{1}},\quad\lambda_{G}=\beta^{2}\mu_{G},\quad
T=\frac{\Delta L^{F}_{1}}{1760m\epsilon}\sqrt{\alpha}\enspace.$

We can check that each $F_{j}$ is $L^{F}_{1}$-smooth, and each $G_{i}$ is
$\mu_{G}$-strongly convex. Assuming $\epsilon~{}\leq~{}\Delta
L^{F}_{1}\alpha/(1760m)$ ensures that $h(0)-\inf_{x}h(x)\leq\Delta$ (we can
check that $h(0)=\lambda_{F}\overline{h}(0)$ and $\inf
h=\lambda_{F}\inf\overline{h}$).

Let us assume without loss of generality that the algorithm at initialization
we have $z^{0}=v^{0}=x^{0}=0$ and consider $(z^{t},v^{t},x^{t})$ the output of
an algorithm with the known sequence $(i_{t},j_{t})$.

Given our inner function and the fact that $\nabla_{2}F(z,x)=0$ for any
$(z,x)\in\mathbb{R}^{(m+1)d+(m+1)d}$, we have

$\displaystyle z^{t+1}$
$\displaystyle\in\mathrm{Span}(z^{0}-x^{0},\dots,z^{t}-x^{t})$ (52)
$\displaystyle v^{t+1}$
$\displaystyle\in\mathrm{Span}(v^{0}+\nabla_{1}F_{j_{0}}(z^{0},x^{0}),\dots,v^{t}+\nabla_{1}F_{j_{t}}(z^{t},x^{t}))$
(53) $\displaystyle x^{t+1}$
$\displaystyle\in\mathrm{Span}(v^{0},\dots,v^{t})\enspace.$ (54)

Since $v^{0}=0$, we have by Equation 53
$v^{1}\in\mathrm{Span}(\nabla_{1}F_{j_{0}}(z^{0},x^{0}))$ and by induction

$v^{t+1}~{}\in~{}\mathrm{Span}(\nabla_{1}F_{j_{0}}(z^{0},x^{0}),\dots,\nabla_{1}F_{j_{t}}(z^{t},x^{t}))\enspace.$

Therefore, using Equation 54, we have

$x^{t+1}\in\mathrm{Span}(\nabla_{1}F_{j_{0}}(z^{0},x^{0}),\dots,\nabla_{1}F_{j_{t}}(z^{t},x^{t}))\enspace.$

Since $z^{0}=0$, by Equation 52, $z^{1}\in\mathrm{Span}(x^{0})$ and by
induction

$z^{t+1}\in\mathrm{Span}(x^{0},\dots,x^{t})\enspace.$

As a consequence,

$\displaystyle
x^{t}\in\mathrm{Span}(\nabla_{1}F_{j_{0}}(\mathrm{Span}(x^{0}),x^{0}),\dots,\nabla_{1}F_{j_{t}}(\mathrm{Span}((x^{s})_{s\leq
t}),x^{t}))\enspace.$

Let us denote $y^{(j,t)}=U^{(j)}x^{t}$. Since $x^{0}=0$, $y^{(j_{0},0)}=0$ and
by the second part of Proposition B.2,
$x^{1}\in\mathrm{Span}(u^{(j_{0})}_{1})$.

Now we assume that for all $s\leq t$ we have

$x^{s}~{}\in~{}\mathrm{Span}(u^{(j_{0})}_{1},\dots,u^{(j_{0})}_{s},\dots,u^{(j_{s-1})}_{1},\dots,u^{(j_{s-1})}_{s})\enspace.$

There exist scalars
$\alpha_{1},\dots,\alpha_{r},\beta_{1,1},\beta_{2,1},\beta_{2,2},\dots,\beta_{t,1},\dots,\beta_{t,t}$
such that

$\displaystyle
x^{t+1}=\sum_{r=1}^{t}\alpha_{r}\nabla_{1}F_{j_{r}}\left(\sum_{s=1}^{r}\beta_{r,s}x^{s},x^{r}\right)\enspace.$

Let $X^{r}=\sum_{s=1}^{r}\beta_{r,s}x^{s}$. For $r\in\\{1,\dots,t\\}$, we have
by induction hypothesis

$X^{r}\in\mathrm{Span}(u^{(j_{0})}_{1},\dots,u^{(j_{0})}_{r},\dots,u^{(j_{r-1})}_{1},\dots,u^{(j_{r-1})}_{r})\enspace.$

By orthogonality, we have

$\mathrm{Span}(u^{(j_{0})}_{1},\dots,u^{(j_{0})}_{r},\dots,u^{(j_{r-1})}_{1},\dots,u^{(j_{r-1})}_{r})\perp\mathrm{Span}(u^{(j_{r})}_{r+1},\dots,u^{(j_{r})}_{T+1})\enspace.$

As a consequence

$U^{(j_{r})}X^{r}=(\langle u^{(j_{r})}_{1},X^{r}\rangle,\dots,\langle
u^{(j_{r})}_{r},X^{r}\rangle,0,\dots,0)\enspace.$

We can use Proposition B.2 to say

$\displaystyle\nabla_{1}F_{j_{r}}\left(X^{r},x^{r}\right)$
$\displaystyle\in\mathrm{Span}(u^{(j_{r})}_{1},\dots,u^{(j_{r})}_{r+1})\subset\mathrm{Span}(u^{(j_{0})}_{1},\dots,u^{(j_{0})}_{r},u^{(j_{0})}_{r+1},\dots,u^{(j_{r})}_{1},\dots,u^{(j_{r})}_{r+1})\enspace.$

And we get finally

$x^{t+1}\in\mathrm{Span}(u^{(j_{0})}_{1},\dots,u^{(j_{0})}_{t},u^{(j_{0})}_{t+1},\dots,u^{(j_{t})}_{1},\dots,u^{(j_{t})}_{t+1})\enspace.$

By induction, for any $t$, we have

$x^{t}\in\mathrm{Span}(\underbrace{u^{(j_{0})}_{1},\dots,u^{(j_{0})}_{t},\dots,u^{(j_{t})}_{1},\dots,u^{(j_{t})}_{t}}_{\text{at
most $mt$ vectors}})$

and so

$x^{t}\perp\mathrm{Span}((u^{(j)}_{1},\dots,u^{(j)}_{T+1})_{j\in[m]\setminus\\{j_{0},\dots,j_{t}\\}},(u^{(j)}_{t+1},\dots,u^{(j)}_{T+1})_{j\in\\{j_{0},\dots,j_{t}\\}})\enspace.$

As a consequence, for $t\leq\frac{m}{2}T$, let
$\mathcal{I}=\\{j,y^{(j,t)}_{T}=y^{(j,t)}_{T+1}=0\\}$ with
$y^{(j,t)}=U^{(j)}x^{t}$. Since $t\leq\frac{m}{2}T$, we have
$|\mathcal{I}|\leq\frac{m}{2}$ and by B.7, we have

$\|\nabla h(x^{t})\|\geq\epsilon\enspace.$

If we define $T((x^{t})_{t},h)=\inf\\{t\in\mathbb{N},\|\nabla
h(x^{t})\|^{2}\leq\epsilon\\}$, we just showed that for the fixed sequence
$(i_{t},j_{t})$, we have

$T((x^{t})_{t},h)\geq\frac{m}{2}T=\Omega(\sqrt{m}\epsilon^{-1})\enspace.$

The right-hand side being independent from the sequence $(i_{t},j_{t})$, for
$t\leq\frac{m}{2}T$, we have

$\mathbb{E}[\|\nabla h(x^{t})\|^{2}]>\epsilon$

where the expectation is taken over the random choice of
$i_{0},\dots,i_{t-1},j_{0},\dots,j_{t-1}$. ∎

##  Appendix C Details on the experiments

We performed the experiments with the Python package Benchopt [Moreau et al.,
2022]. For each experiment, we use minibatches instead of single samples to
estimate oracles because it is more efficient in practice. We use a batch size
of 64 for the stochastic inner and outer oracles.

###  C.1 Influence of the parameter $q$

We solve a regularization selection problem for an $\ell^{2}$-regularized
logistic regression problem. Here, we assume that we have a regularization
parameter per feature. We have $n_{\mathrm{train}}=49,990$ training samples
$(d^{\mathrm{train}}_{i},y_{i}^{\mathrm{train}})_{i\in[n_{\mathrm{train}}]}$
and $n_{\mathrm{val}}=91,701$ validation samples
$(d^{\mathrm{val}}_{i},y_{i}^{\mathrm{val}})_{i\in[n_{\mathrm{train}}]}$
coming from the
IJCNN17771https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html
dataset. Mathematically, it boils down to solve Problem (1) with $F$ and $G$
given by

$\displaystyle F(\theta,\lambda)$
$\displaystyle=\frac{1}{n_{\mathrm{val}}}\sum_{j=1}^{n_{\mathrm{val}}}\varphi(y_{j}^{\mathrm{val}}\langle
d_{j}^{\mathrm{val}},\theta\rangle)$ $\displaystyle G(\theta,\lambda)$
$\displaystyle=\frac{1}{n_{\mathrm{train}}}\sum_{i=1}^{n_{\mathrm{train}}}\varphi(y_{i}^{\mathrm{train}}\langle
d_{i}^{\mathrm{train}},\theta\rangle)+\frac{1}{2}\sum_{k=1}^{p}\mathrm{e}^{\lambda_{k}}\theta_{k}^{2}$

where $\varphi$ is the logistic loss defined by
$\varphi(u)=\log(1+\mathrm{e}^{-u})$. The inner and outer step sizes are set
to 0.05.

###  C.2 Comparison with competitors on the datacleaning task

The datacleaning problem aims to train a multiclass classifier while having
some training samples with noisy labels. On the one hand we have training
labelled samples
$(d_{i}^{\mathrm{train}},y_{i}^{\mathrm{train}})_{i\in[n_{\mathrm{train}}]}$
with potentially corrupted labels with probability $p_{c}$ (in the experiments
$p_{c}=0.5$). On the other hand, we have a validation set
$(d_{j}^{\mathrm{val}},y_{j}^{\mathrm{val}})_{j\in[n_{\mathrm{val}}]}$ where
all the samples are clean. The datacleaning problem consists in learning a
classifier on all these samples by giving less weight to corrupted labels. It
can be cast as a bilevel optimization problem like (1) where the function $F$
and $G$ are given by

$\displaystyle F(\theta,\lambda)$
$\displaystyle=\frac{1}{n_{\mathrm{val}}}\sum_{j=1}^{n_{\mathrm{val}}}\ell(\theta
d_{j}^{\mathrm{val}},y_{j}^{\mathrm{val}})$ $\displaystyle G(\theta,\lambda)$
$\displaystyle=\frac{1}{n_{\mathrm{train}}}\sum_{i=1}^{n_{\mathrm{train}}}\sigma(\lambda_{i})\ell(\theta
d_{i}^{\mathrm{train}},y_{i}^{\mathrm{train}})+C_{r}\|\theta\|^{2}$

where $\theta\in\mathbb{R}^{C\times p}$,
$\lambda\in\mathbb{R}^{n_{\mathrm{train}}}$, $\ell$ is the cross entropy loss
and $\sigma$ is the sigmoid function defined by
$\sigma(\lambda)=\frac{1}{1+\mathrm{e}^{-\lambda}}\in(0\,,\,1]$.

We run this experiment on the MNIST dataset. We used $20,000$ training
samples, $5,000$ validation samples and $10,000$ test samples. The parameter
$C_{r}$ is set to $0.2$ after a manual search to get the best performance. For
the tuning of the step sizes of each method, we set
$(\rho_{t},\gamma_{t})=(\alpha t^{-a},\beta t^{-b})$ where $(a,b)$ are the
rate provided by the analysis of each method, $\alpha$ is chosen among 4
values between $10^{-}3$ and $10^{0}$ spaced on a logarithmic scale. The
scaling parameter $\beta$ is set to $\frac{\beta}{r}$ where $r$ is chosen
among 6 values between $10^{-5}$ and $10^{0}$ spaced on a logarithmic scale.
The size of the inner loop $q$ is parametrized as $q=a\frac{n+m}{b}$ where
$b=64$ is the batch size and $a$ is chosen in
$\\{2^{-1},2^{3},2^{6},2^{9}\\}$.


# Building and Evaluating Open-Domain Dialogue Corpora
with Clarifying Questions

Mohammad Aliannejadi1, Julia Kiseleva2, Aleksandr Chuklin3,
Jeffrey Dalton4, Mikhail Burtsev5
1University of Amsterdam, 2Microsoft Research, 3Google Research
4University of Glasgow, 5MIPT & AIRI
<EMAIL_ADDRESS><EMAIL_ADDRESS>
chuklin@google.com,jeff.dalton<EMAIL_ADDRESS>
###### Abstract

Enabling open-domain dialogue systems to ask clarifying questions when
appropriate is an important direction for improving the quality of the system
response. Namely, for cases when a user request is not specific enough for a
conversation system to provide an answer right away, it is desirable to ask a
clarifying question to increase the chances of retrieving a satisfying answer.
To address the problem of _‘asking clarifying questions in open-domain
dialogues’_ : (1) we collect and release a new dataset focused on open-domain
single- and multi-turn conversations, (2) we benchmark several state-of-the-
art neural baselines, and (3) we propose a pipeline consisting of offline and
online steps for evaluating the quality of clarifying questions in various
dialogues. These contributions are suitable as a foundation for further
research.

##  1 Introduction

The ultimate goal of a conversational system is to assist users by returning
an appropriate answer in response to their requests Kiseleva et al. (2016a);
Li et al. (2021). Recent progress on neural approaches to natural language
processing Devlin et al. (2019); Liu et al. (2019); Clark et al. (2020), and
the availability of large amounts of conversational data have triggered a
renaissance in end-to-end neural open-domain chatbots Adiwardana et al.
(2020); Roller et al. (2021); Zhang et al. (2020); Burtsev et al. (2017);
Dalton et al. (2020). There has been great progress on suggesting measures to
evaluate what makes a conversation satisfying for users using various human
evaluation techniques Li et al. (2019a); See et al. (2019b). Those efforts
showed that suggested large pre-trained models do not always perform
seamlessly See et al. (2019a), and there are still several challenges needed
to be solved for open-domain conversational systems Huang et al. (2020).

Figure 1: Examples of clarification questions embedded into the open domain
conversations: (a) represents a clear user request, so clarification was
unnecessary; (b) and (c) demonstrate a situation when the request is ambiguous
and a system needs to act.

Nass and Moon (2000) conclude that people have similar expectations from
talking to bots and humans. This similarity is a possible explanation for why
sometimes user requests might be ambiguous and incomplete, as shown in Fig. 1
(b) and (c). This ambiguity is especially challenging to handle in a dialogue
setting, where a system is limited by returning only one answer in response to
each request, unlike in web search setup where diversification of results is
possible and acceptable Vallet and Castells (2012). Previous research has
shown that users are much more forgiving about system mistakes if they can act
on them with minimal efforts spent Kocielnik et al. (2019); Kiseleva et al.
(2016b). Therefore it is more appropriate to ask a clarifying question in user
request ambiguity rather than generating incorrect answers. There are separate
attempts to explore the following related tasks: (1) identifying a moment when
the question should be asked in the course of conversation Hancock et al.
(2019); and (2) retrieving a clarification question Rao and III (2018); Wang
et al. (2018). In this paper, we aim to combine these related aspects and
study the following problem of _generating clarifying questions for open-
domain conversations_ : the system must identify wherever the question is
ambiguous, and, if so then instead of trying to answer it directly, it should
ask a good clarifying question (Fig. 1). One possible stumbling block
preventing the community from studying the problem of open-domain clarifying
question generation to enhance user experience while interacting with a
conversational bot Huang et al. (2020) is the lack of suitable datasets, which
we address in this work. To summarise, the main contributions of this work
are:

  1. C1

releasing a dataset dedicated to the problem of asking a clarifying question
in open-domain dialogue systems. The dataset includes single- ($\sim$15K) and
multi-turn ($\sim$1.5M) conversations and covers $\sim$300 various topics and
it is suited to study: (1)  _when_ a clarifying question should be asked given
the current context of the conversation; and (2)  _which_ question should be
asked;

  2. C2

benchmarking several state-of-the-art (SoTA) neural models; and

  3. C3

building an evaluation pipeline that provides fast iteration and involves two
stages: (1) offline (automatic evaluation); and (2) online (human-in-a-loop to
converse with the system).111The pipeline was designed as part of the ConvAI3
Aliannejadi et al. (2020) data challenge (https://convai.io)

We release the collected dataset, offline evaluation pipeline, and the code
for running explored neural SoTA models. These models can be employed as
baselines for the task.222Available at https://github.com/aliannejadi/ClariQ

##  2 Related work

Our work is broadly relevant to two strands of research: learning to ask
clarifying questions in open-domain conversational settings (Section 2.1) and
evaluating dialogue systems (Section 2.2).

###  2.1 Learning to ask clarifying questions

Information retrieval community has paid close attention to the problem of
ambiguity in user search queries. Previously this problem was addressed
through the diversification of search result pages Radlinski and Dumais
(2006); Kong and Allan (2016, 2014), including via usage of personal and
contextual data Jiang et al. (2015); Kato and Tanaka (2016). Recently, Rosset
et al. (2020); Aliannejadi et al. (2019); Zamani et al. (2020a) suggest
techniques to address ambiguity by generating clarifying questions.

Where the general settings are: (1) a user is issuing an _ambiguous_ keyword
query; (2) a search engine’s goal is to suggest conversational clarifying
questions to help to find the required information Krasakis et al. (2020);
Lotze et al. (2021); Sekulic et al. (2021); Aliannejadi et al. (2021). These
works also resulted in a number of datasets, e.g. _Qulac_ Aliannejadi et al.
(2019) and _MIMICS_ Zamani et al. (2020b), which consists of queries, issued
by real users, and behavioral signals such as clicks. Braslavski et al. (2017)
focus on characteristics, forms, and general patterns of clarifying questions.

Suggesting a clarifying questions is closely related to question answering
(Q&A) Kwiatkowski et al. (2019); Soleimani et al. (2021) and question
generation (QG) domains Gao et al. (2019); Chai and Wan (2020). Trienes and
Balog (2019) made an attempt to understand unclear questions, Li et al. (2017)
suggesting an RL-based method for deciding _when_ to ask for user feedback in
Q&A setup.

Recently, proactive bot behavior has started to attract researchers’ attention
in dialogue settings yet remains rather untouched Huang et al. (2020). Rao and
III (2018) designed a model to rank a candidate set of clarification questions
by their usefulness to the given post at Stack Exchange, which targeted the
problem _which_ question to ask. The resulted dataset was released, but it
covers specific narrow topics. In contrast, Hancock et al. (2019) focused on
_when_ to ask a question in order to self-retrain a bot, which has been
resulted in releasing a dataset. Wang et al. (2018) studied QG techniques in
application to open-domain conversations.

###  2.2 Evaluating Dialogue Systems

Dialogue systems are generally separated into two types: task-oriented and
open-domain. The task-oriented ones usually have clear criteria for
evaluation, e.g. turn correction ratio, inappropriate utterance ratio, proxies
for accuracy, and success rate (Takanobu et al., 2019; Li et al., 2016; Su et
al., 2018; Li et al., 2020). Despite significant efforts to introduce
automatic metrics to evaluate open-domain conversations Reiter (2018);
Novikova et al. (2017); Lowe et al. (2017), it remains area for exploration Li
et al. (2019a, b, 2021). To the best of our knowledge the current standard
approach for evaluating open-domain dialogues requires employing human
assessments via crowdsourcing platforms Zhang et al. (2018); Li et al. (2019a)
or engaging volunteers to participate in research competitions Burtsev et al.
(2018); Dinan et al. (2020); Burtsev and Logacheva (2020); Aliannejadi et al.
(2020).

Therefore, we can conclude that understanding and generating open-domain
clarification questions is a major component in conversational information-
seeking systems, which is still under exploration. Hence, our efforts on
collecting datasets and investigating the performance of the neural SoTAs are
timely and useful for future research in this area.

##  3 Problem Setting

Figure 2: A pipeline for asking clarifying questions for open-domain
conversations using example Fig 1 (C).

Our main goal is to collect a dataset to enable studying clarifying questions
generation whenever appropriate, as depicted in examples in Fig. 1. Fig. 2
demonstrates a pipeline that makes it possible to process user requests in the
open domain as follows: ‘User Request Understanding’ (URU) decides which
module to call either ‘Clarifying Question Generation’ (CQG) or ‘Answer
Generation’ (AG). In this work, we focus on the first two. We aim to collect
the following data:

  * •

User Request ($U$): an initial user request in the conversational form, e.g.,
_‘What is Fickle Creek Farm?’_ with a label reflecting whether clarification
is needed;

  * •

Set of clarification questions ($\\{Q\\}$): a set of possible reasonable
clarifying questions that address multiple aspects/facets of $U$, e.g.,
$Q_{1}:$ _‘Do you want to know the location of fickle creek farm?’_ , $Q_{2}:$
_‘Would you like to know the history of fickle creek farm?’_ 333Candidate
clarifying questions should also address out-of-collection facets.;

  * •

User Answers ($A$): each question is supplied with a user answer, e.g., the
answer to $Q_{1}$ is $A_{1}:$ _‘No, I want to find out where can I purchase
fickle creek farm products’_ , the answer to $Q_{2}$ is $A_{2}:$ _‘I just need
general information about fickle creek’_

The collected dataset of pairs $(U,\\{Q,A\\})$ can be easily transformed to a
set of single-turn conversations consisting of the coherent and consistent
triples $(U,Q,A)$ as shown in the example in Fig. 2. We ask items in a triple,
$U$, $Q$ and $A$, satisfy the following _requirements_ :

  1. R1

user requests $(U)$ must cover various conversational topics to represent
open-domain dialogues;

  2. R2

the final collection of $U$ should contain both types: ambiguous and
unambiguous;

  3. R3

each inquiry $U$ to the system should be in the conversational form;

  4. R4

the need for clarification should be predetermined as a label for each $U$ in
the collection;

  5. R5

each clarifying question $(Q)$ should be reasonable, coherent with $U$ and
address multiple facets of every ambiguous request $U$; and

  6. R6

each user answer $A$ should be consistent with the clarifying question from
the system.

After collecting of single-turn conversations, they are used to train various
conversational agents. To collect multi-turn conversations, the two best-
performing agents are utilized to converse with crowdsourced workers, who
evaluate a system quality and reply to suggested clarifying questions.
Finally, the two agents are evaluated using Acute-eval framework Li et al.
(2019a), which is best available practice for online evaluation of open-domain
dialogue systems. Overall, our pipeline for _data collection_ and _evaluation_
is summarized in Fig. 3.

Figure 3: The pipeline highlights steps for two main goals: to collect required datasets and to perform the reproducible evaluation. Table 1: An example of facets for the incomplete query. Topic | Facet
---|---
Neil Young | Find albums by Neil Young to buy
Find biographical information about Neil Young
Find lyrics or sheet music for Neil Young’s songs
Find a list of Neil Young tour dates.

##  4 Data Collection

Following the suggested pipeline in Fig. 3 this section describes our results
regarding the collected datasets to initiate more follow-up studies of the
problem of _‘asking clarifying questions in open-domain dialogues_ ‘, namely,
$P_{1}$: single-turn dialogues (Sec. 4.1) and $P_{3}$ multi-turn ones (Sec.
4.2).

###  4.1 P1: Crowdsourcing Single-Turn Dialogues

##### Collecting Conversational User Requests

Our collection of single-turn conversations is built on top of the TREC Web
track 2009-2014444https://trec.nist.gov/data/webmain.html data, which was
originally designed to evaluate search result diversification. The TREC
collection contains 300 search topics. The presence of varied search topics,
which express different user information needs, helps us imitate open domain
user-system interactions, which are required in R1. Each topic in the
collection is specific, ambiguous, or faceted Clarke et al. (2009). In this
work, we use the term ‘facet’ to refer to the subtopics of both faceted and
ambiguous topics. For clarity, the example of mapping from the search topic to
set of facets is provided in Tab. 1. Faceted and ambiguous topics make an
ideal case to study the effect of clarifying questions as they can be
interpreted in various ways, which is required by R2. User information needs
are expressed in the form of short search topic description555sometimes also
can be referred as ‘keyword query’ (Tab. 1) because the TREC Web track
collection was designed for web search needs. Therefore, to satisfy
requirement R3 and to make requests lexical diverse, we ask expert annotators
to convert those short keyword queries to the proper conversational request.
The examples of such conversion are presented in Tab. 2. To satisfy the
requirement R4, annotators are asked to provide a score for each request to
reflect if clarification is needed ranging from 1 to 4, where ‘1’ stands for
very low or no need for clarification and ‘4’ indicates a highly ambiguous
request (examples are provided in Tab. 2). Two annotators assess clarification
need of a query. In case of disagreement, we assign an additional annotator to
make the final assessment. We achieve a high inter-annotator agreement on this
task (Cohen’s $\kappa=0.78$).

Table 2: Examples of keyword queries converted to conversational requests with assigned clarification score (C. score). Search Topic | Conversational Request | C. Score
---|---|---
average charitable donation | What is average charitable donation? | 1
gmat prep classes | How to prepare for the GMAT? | 2
von willebrand disease | What is von Willebrand Disease? | 3
land surveyor | I’m interested to know about land surveyor | 3
alexian brothers hospital | Give me information about Alexian Brothers hospitals | 3
worm | I’m looking for information on worm | 4

##### Collecting Clarifying Questions $\\{Q\\}$

To collect $\\{Q\\}$ for every $U$ that satisfies R5: (1) we utilized Qulac
dataset666https://github.com/aliannejadi/qulac by converting topics into
conversational requests; (2) we significantly extended it by crowdsourcing
more data through Human Intelligence Task (HIT) on Amazon Mechanical
Turk777http://www.mturk.com, which design follows a general strategy proposed
in Aliannejadi et al. (2019). Namely, we asked the workers to imagine
themselves acting as a conversational agent888such as Microsoft Cortana,
Alexa, or Google Assistant. where an imaginary user had asked them about a
topic. Then, we described the concept of facet to them, supporting it with
multiple examples. Finally, we ask Turkers to do the following:

  * •

discover the facets of each $U$ using a preferred search engine and scan the
results in the first three pages; and

  * •

generate six questions related to $U$, aiming to address the facets they had
figured out.

We assigned two workers per HIT, resulting in $12$ questions per $U$ in the
first round. To preserve the questions’ language diversity, we limited each
worker to a maximum of two HITs. HITs were available to workers residing in
the U.S. who had an approval rate of over 97%.

##### Controlling Quality of Clarifying Questions

To estimate the quality of the collected questions, we aim to address two main
concerns: (1)  _how good are the collected clarifying questions?_ ; and (2)
_Is the set of clarifying questions diverse (in other words, addressing
different facets associated with the topic)?_ Given the high complexity of
this task, we appointed two expert annotators. They were instructed to read
all the collected questions on each topic, marking invalid and duplicate
questions. Annotators were asked to match a question to a facet if its answer
would address the facet. Finally, to ensure that all facets were covered by at
least one question, we asked the annotators to generate an additional question
for each facet that needed more specific questions.

##### Collecting Answers

To satisfy R6, we designed another HIT to collect coherent and consistent
answers to the clarifying questions. The task started with detailed
instructions followed by several examples. The workers were given $U$ and a
facet description. Then we instruct them to assume that they had submitted the
initial user request $U$ with their actual information need being the given
facet. Then workers were required to write the answer to the one clarifying
question that was presented to them. If a question required information other
than what workers were provided with, they were instructed to use a ‘No
answer’ tag. Each worker was allowed to complete a maximum of $100$ HITs to
ensure language diversity. Workers were based in the U.S. with an approval
rate of 95% or greater.

##### Controlling quality of Collected Answers

During the course of data collection, we performed regular quality checks on
the collected answers. The checks were done manually on 10% of submissions per
worker. In case we observed any invalid submissions among the sampled answers
of one worker, we then studied all the submissions from that workers. Invalid
submissions were then removed from the collection, and the worker was banned.
Finally, we assigned all invalid answers to other workers to complete.
Moreover, we employed basic behavioral check techniques in the design of the
HIT. For example, we disabled copy/paste features of text inputs and tracked
workers’ keystrokes. This enabled us to detect and reject low-quality
submissions.

As an outcome, we have a high-quality collection of single-turn conversations
in the form of required triples: $(U,Q,A)$, which is marked as $P_{1}$ in our
pipeline in Fig. 3. Tab. 3 provides a statistics on collected dataset of
single-turn conversations.

###  4.2 P3: Crowdsourcing Multi-Turn Dialogues

The collected dataset of single-turn is sufficient to train and evaluate
several conversational agents. More technical details on training and
evaluation are provided in Sec. 5. For now, we assume that as a result of
$P_{2}$: $\mathit{DA}$ is one of the best-performing trained dialogue agents.
We assume that the trained $\mathit{DA}$ can have a conversation with users.
Namely, it should either ask a clarification question or give a factual answer
to the user’s request at each dialog step. Therefore, the trained
$\mathit{DA}$ is capable of:

  * •

providing clarification question whenever appropriate in the course of the
conversation;

  * •

interpreting user’s answer to the clarifying question.

Figure 4: An example of the task provided at the HIT for multi-turn
conversations, which asks to submit the answer given dialogue history/context.

To collect multi-turn conversations, we utilize best-performing dialogue
agents that can accommodate an arbitrary number of turns, having two goals in
mind:

  1. G1

evaluating the quality of the agents in multi-turn settings where they
converse with real humans; and

  2. G2

collecting a new dataset of multi-turn conversations with respect to
clarification questions.

To reach G1, the idea is to run the agents multiple times with different turns
and evaluate them accordingly. For that purpose, we design a HIT similar to
the ones described in Sec. 4.1 with the difference in the context of a
conversation. We instructed crowd workers to understand the user’s actual
information need and imagine they are looking for the same information. Then,
follow a conversation on that as presented in Fig. 4. The workers were
instructed to answer the last question in the conversation while considering
the conversation’s context, which consists of previous questions and answers.
The context of conversation could include 1-2 rounds of question-answer
interactions. To check the quality of clarifying questions returned by the
trained dialogues, we instructed workers to indicate if the question was not
understandable or was not in a proper language. As a result, such questions
were removed from the collection. We use the same quality check procedure for
the collected answers as described in the previous section. Tab. 3 provides a
statistics on collected multi-turn conversations, to achieve G2.

##### Synthetic Multi-Turn Conversations

We also generate synthetic multi-turn conversations for training purposes. To
do so, for each topic, we create a set of all possible combinations of
questions (2 or 3 questions) together with their corresponding answers.

Table 3: Statistics over collected data.

# search topics | 298
---|---
# faceted & ambiguous topics | 250
# single topics | 48
# facets | 1070
# questions | 3,304
Average terms per question | 9.74 $\pm$ 2.62
Average terms per answer | 8.48 $\pm$ 4.40
# synthetic conversations (all) | 1,596,757
# synthetic conversations (2 turns) | 203,050
# synthetic conversations (3 turns) | 1,393,707
# human-machine dialogues (1 turn) | 15,226
# human-machine dialogues (3 turns) | 499

Table 4: Performance for predictors returning classification need score based
on _dev_ and _test_ sets. The best-performing model is marked in bold.

Model |  | Precision | Recall | F1-Measure | MSE
---|---|---|---|---|---
RoBERTa-based | dev | 0.6039 | 0.5600 | 0.5551 | 0.6200
test | 0.5981 | 0.6557 | 0.6070 | 0.5409
BART | dev | 0.7008 | 0.7000 | 0.6976 | 0.5200
test | 0.4813 | 0.4754 | 0.4756 | 0.7705
BERT-based | dev | 0.5218 | 0.4800 | 0.5000 | 0.8200
test | 0.3931 | 0.4918 | 0.4253 | 0.6557

##  5 Models and Evaluation

Following the suggested pipeline in Fig. 3, we explain our contributions
regarding the _evaluation_ of _‘asking clarifying questions in open-domain
dialogues’_ problem, namely:

  * •

$P_{2}$: how the dialogue agents are trained and automatically evaluated based
on single-turn conversations (Sec. 5.1);

  * •

$P_{4}$: how evaluation of multi-turn conversations is performed from both
perspectives: offline automatic manner and having human-in-the-loop using
Acute-eval Li et al. (2019a) (Sec. 5.2).

We design our experiments to collect answers to the following research
questions:

  1. RQ1

When to ask clarifying questions during open-domain dialogues? (Sec. 5.1.1)

  2. RQ2

Which clarifying question to ask for a given context of a conversation? (a.
the single-turn conversations case is described in Sec. 5.1.2; b. multi-turn
one – Sec. 5.2)

###  5.1 P2: Evaluating Single-Turn Agents

The collected dataset, described in Sec. 4.1, is split into training (70%),
validation (dev) (10%), and test (20%) sets. We split the data based on the
search topic and maintained the same split for all single-turn and multi-turn
experiments. During the evaluation procedure, the following is used: (1) a set
of conversational user requests, and (2) a set of questions (i.e., question
bank), which contains all collected questions on all the topics.

####  5.1.1 Predicting Clarification Need

##### Task

The task is, given a user request, return a score from $1$ (no need for
clarifying questions) to $4$ (cannot provide any answers without user
clarification) indicating the necessity of asking clarifying questions (as
depicted in module ‘Understanding User Request’ in Fig. 2).

##### Automatic Evaluation

To evaluate the performance of the suggested classifier, we use Precision,
Recall, F1-Measure, and Mean Squared Error (MSE). Tab. 4 presents the
collected results of various classification methods, which includes Roberta-
based classifier Liu et al. (2019), BART Chipman et al. (2010), and BERT-based
classifier Devlin et al. (2019). Based on the supplied results, we can answer
RQ1: the task is rather difficult and potentially can benefit from more
exploration despite the reasonable performance of the proposed baselines.

####  5.1.2 Returning Clarifying Question

##### Task

The task is, given a user request which needs clarification, return the most
suitable clarifying question from the supplied question bank (as shown in
module CQG in Fig. 2).

##### Automatic Evaluation

We introduce two main strategies for evaluation: (1) document relevance and
(2) question relevance.

##### _Document Relevance_

To estimate the relevance of the retrieved documents we use the following
standard metrics: Mean Reciprocal Rank (MRR) Voorhees (1999); Radev et al.
(2002), Precision (P)@[1,3,5,10,20], Normalized Discounted Cumulative Gain
(nDCG)@[1,3,5,20] Wang et al. (2013). These metrics are computed as follows: a
selected clarifying question, together with its corresponding answer, is added
to the original user request. The updated query is then used to retrieve (or
re-rank) documents from the collection. The quality of the question is then
evaluated by measuring how much the question and its answer affect document
retrieval performance when added to the initial request. We evaluate document
relevance based on the relevance assessments provided by the TREC Web Track.

Table 5: A set of document relevance related metrics reported on _dev_ and
_test_ sets. NDCG@3 (in bold) reported on the test set is used as the main
metric to rank the quality of the models.

Model |  | MRR | P@1 | NDCG@3 | NDCG@5
---|---|---|---|---|---
B1: Worst Q. | dev | 0.0841 | 0.0125 | 0.0252 | 0.0313
test | 0.0541 | 0.0000 | 0.0097 | 0.0154
B2: No Q. | dev | 0.3000 | 0.2063 | 0.1475 | 0.1530
test | 0.3223 | 0.2268 | 0.1134 | 0.1059
B3: Best Q. | dev | 0.4882 | 0.4187 | 0.3337 | 0.3064
test | 0.4881 | 0.4275 | 0.2107 | 0.1759
M1: Roberta | dev | 0.3640 | 0.2813 | 0.2002 | 0.1954
test | 0.3190 | 0.2342 | 0.1265 | 0.1130
M2: ELECTRA | dev | 0.3761 | 0.3000 | 0.2113 | 0.1955
test | 0.3140 | 0.2379 | 0.1229 | 0.1097
M3: BERT | dev | 0.3596 | 0.2750 | 0.1879 | 0.1882
test | 0.3044 | 0.2119 | 0.1131 | 0.1021
M4: BM25 | dev | 0.3096 | 0.2313 | 0.1608 | 0.1530
test | 0.3134 | 0.2193 | 0.1151 | 0.1061
M5:  BERT +BM25 | dev | 0.3180 | 0.2437 | 0.1625 | 0.1550
test | 0.3216 | 0.2453 | 0.1196 | 0.1097
M6:  Roberta +BM25 | dev | 0.3606 | 0.2813 | 0.1942 | 0.1891
test | 0.3045 | 0.2156 | 0.1108 | 0.1025

##### _Question Relevance_

Models are also evaluated in how well they can rank relevant questions higher
than other questions in the question bank. For this task, which we call
‘question relevance,’ the models are evaluated in terms of Recall@[10,20,30].
Since the precision of models is evaluated in the document relevance task,
here we focus only on recall.

The suggested evaluation metrics are collected for a number of baselines (B)
and fine-tuned state-of-the-art NLP models (M):

  * •

B1: Worst Question, when the dialogue system returns the least relevant
question;

  * •

B2: No Question, when the system never returns any clarifying question;

  * •

B3: Best Question, which show oracle performance as it always returns the most
relevant question from the bank.

  * •

M1: fine-tuned version of the pre-trained Roberta Liu et al. (2019);

  * •

M2: fine-tuned sequence classification model based on pre-trained ELECTRA
Clark et al. (2020); Ou and Lin (2020);

  * •

M3: fine-tuned version of pre-trained BERT Devlin et al. (2019)

  * •

M4: BM25 Robertson et al. (1995); and

  * •

‘+BM25’ for M5 and M6: means that BM25 on top on neural baseline for the final
re-ranking of the questions from the bank.

Based on results of automatic evaluations of all the methods suggested above
are reported in Tab. 5 and Tab 6 for single-turn conversations, to answer
RQ2.a, we can conclude the performance of the best-performing fine-tuned
neural SoTAs is reasonable and we can use them for multi-turn conversation.

Table 6: A set of question relevance related metrics reported on _dev_ and
_test_ sets. Recall@30 (in bold) reported on the test set is used as the main
metric to rank the quality of the models.

Model |  | R@5 | R@10 | R@20 | R@30
---|---|---|---|---|---
M1: Roberta | dev | 0.3649 | 0.6694 | 0.8265 | 0.8587
test | 0.3395 | 0.6251 | 0.8176 | 0.8568
M2: ELECTRA | dev | 0.3604 | 0.6749 | 0.8478 | 0.8761
test | 0.3404 | 0.6329 | 0.8335 | 0.8744
M3: BERT | dev | 0.3492 | 0.6196 | 0.7337 | 0.7632
test | 0.3438 | 0.6228 | 0.7987 | 0.8409
M4: BM25 | dev | 0.3245 | 0.5638 | 0.6675 | 0.6913
test | 0.3170 | 0.5705 | 0.7292 | 0.7682
M5:  BERT +BM25 | dev | 0.3454 | 0.6166 | 0.7354 | 0.7621
test | 0.3272 | 0.6061 | 0.8013 | 0.8433
M6:  Roberta +BM25 | dev | 0.3637 | 0.6409 | 0.7484 | 0.7793
test | 0.3361 | 0.6219 | 0.7960 | 0.8360

###  5.2 P4: Evaluating Multi-Turn Conversations

##### Task

The task is, given an ongoing conversation with multiple turns, select or
generate the next question that would clarify the user’s intent best. The main
goal is to learn from previous user feedback and ask a question that would
lead to the highest information gain.

##### Automatic Evaluation

Similar to the single-turn task, we evaluate the effectiveness of the baseline
models based on document relevance. Therefore, we utilize the whole
conversation context, clarifying questions, and human responses to retrieve
documents from the collection and assess the quality of a question based on
its impact on ranking performance. Note that we do not evaluate multi-turn
models in terms of question relevance, since the question relevance is
intended to evaluate recall of questions related to the search topic. Due to
complexity and costs of the evaluation, we pick two best-performing models
from Sec. 5.1.2 for this task. To do so, we use the synthetic training data to
fine-tune ELECTRA and Roberta similarly to our single-turn setup, but in the
multi-turn case the whole history context is considered as a user request. The
module for deciding whenever the request needs clarification is preserved. We
see in Tab. 7 that ELECTRA outperforms Roberta in terms of all evaluation
metrics by a margin. One promising future research line might be exploring
what properties of these two models lead to that difference in their
effectiveness for this task.

Table 7: A set of document relevance related metrics reported on test set for
multi-turn conversations. NDCG@3 (in bold) is used as the main metric to rank
the quality of the models.

Model | MRR | P@1 | NDCG@3 | NDCG@5
---|---|---|---|---
ELECTRA | 0.1798 | 0.1161 | 0.0553 | 0.0536
Roberta | 0.1669 | 0.1067 | 0.0522 | 0.0494

Table 8: Pairwise comparison of ELECTRA and Roberta on the multi-turn
conversations. The values report the percentage of judgements in which ELECTRA
wins Roberta in terms of HU, EG, IN, KL, and CL.101010Abbreviations are
explained in the text.

Comparison | HU | EG | IN | KL | CL
---|---|---|---|---|---
ELECTRA vs. Roberta | 0.57 | 0.59 | 0.56 | 0.57 | 0.56

##### Human Evaluation

To ensure that our automatic evaluation reflects the dialogues’ quality, we
conduct a pairwise human evaluation on two of the baselines. We use and extend
the Acute-eval human annotation framework Li et al. (2019a) to evaluate 120
randomly sampled dialogue pairs. For consistency, we use the four questions
that the authors suggest measuring Humanness (HU), Engangingness (EG),
Interestingness (IN), Knowledgeable (KL), and add a fifth one, specific to our
task, on Clarification (CL). We modify the crowdsourcing task to inform the
annotators about the conversation’s main goal (i.e., information seeking).
Furthermore, it is crucial to ensure that the annotators consider the model’s
ability to understand the user’s feedback and incorporate the additional
knowledge when asking its next question. Therefore, we added another question
to examine this aspect of the conversation. As shown in Fig. 5, after showing
two full conversations to the annotators, they evaluated the model’s ability
of clarification by answering the following question: _‘Which one asks better
(or more reasonable) clarifying questions?’_. Tab. 8 reports the results of
our human evaluation of 120 dialogue pairs in terms of percentage of cases
that ELECTRA beats Roberta based on the human annotation. We see that ELECTRA
is judged to be the best model in most cases for all five aspects. It is
interesting to see that the human annotation is in line with the proposed
automatic annotation, suggesting that our approach approximates the true
quality of the models. Moreover, we see that our new evaluation dimension,
Clarification, achieves a similar result to other dimensions, which suggests
that it should be included in the evaluation framework for the open-domain
dialogues.

Figure 5: Sample on-boarding dialogue for clarification annotation using
Acute-eval framework.

Based on reported results for automatic evaluation (Tab. 7), which is aligned
with human-in-a-loop one (Tab. 8), we can conclude that suggested methods can
be solid baselines for the follow-up research as an answer RQ2.b.

##  6 Conclusions

Asking clarifying questions when a user request is ambiguous is essential for
developing human-like open-domain dialogue systems. In this work, we introduce
a large-scale dataset that covers almost 300 different topics and is suitable
as a foundation for further research. The collected dataset includes both
single- and multi-turn conversations. We benchmark several state-of-the-art
neural models that were fine-tuned for asking clarifying questions. Based on
how these models performed, we conclude that they are solid baselines for
future research that more fully explores the problem space. In this paper, we
also suggest an offline automatic evaluation pipeline, which agrees with
human-in-loop evaluation.

We publicly release the collected datasets, the code for training baselines,
and our evaluation procedure in order to push forward the state-of-the-art.

## Acknowledgement

This work was supported in part by the NWO Innovational Research Incentives
Scheme Vidi (016.Vidi.189.039) and an Engineering and Physical Sciences
Research Council grant EP/V025708/1. We thank Microsoft Research, Google, and
Amazon Science for their support to collect the data and organize the ConvAI3
data challenge. We also thank all the participants of the data challenge for
taking part in the competition and releasing their models. Any opinions,
findings and conclusions or recommendations expressed in this material are
those of the authors and do not necessarily reflect those of the sponsors.

## References

  * Adiwardana et al. (2020) Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020\.  Towards a human-like open-domain chatbot.  _arXiv preprint arXiv:2001.09977_.
  * Aliannejadi et al. (2021) Mohammad Aliannejadi, Leif Azzopardi, Hamed Zamani, Paul Thomas, and Nick Craswell. 2021.  Analysing mixed initiatives and search strategies during conversational search.  In _International Conference on Information and Knowledge Management (CIKM)_.
  * Aliannejadi et al. (2020) Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail S. Burtsev. 2020.  ConvAI3: Generating clarifying questions for open-domain dialogue systems (ClariQ).  _arXiv preprint arXiv:2009.11352_.
  * Aliannejadi et al. (2019) Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft. 2019.  Asking clarifying questions in open-domain information-seeking conversations.  In _ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)_ , pages 475–484.
  * Braslavski et al. (2017) Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka. 2017.  What do you mean exactly? analyzing clarification questions in CQA.  In _Conference on Conference Human Information Interaction and Retrieval (CHIIR)_ , pages 345–348.
  * Burtsev et al. (2017) Mikhail Burtsev, Aleksandr Chuklin, Julia Kiseleva, and Alexey Borisov. 2017.  Search-oriented conversational AI (SCAI).  In _ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR)_ , pages 333–334.
  * Burtsev and Logacheva (2020) Mikhail Burtsev and Varvara Logacheva. 2020.  Conversational intelligence challenge: Accelerating research with crowd science and open source.  _AI Magazine_ , 41(3):18–27.
  * Burtsev et al. (2018) Mikhail Burtsev, Varvara Logacheva, Valentin Malykh, Iulian Vlad Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W Black, Alexander Rudnicky, and Yoshua Bengio. 2018.  The first conversational intelligence challenge.  In _The NIPS’17 Competition: Building Intelligent Systems_ , pages 25–46.
  * Chai and Wan (2020) Zi Chai and Xiaojun Wan. 2020.  Learning to ask more: Semi-autoregressive sequential question generation under dual-graph interaction.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 225–237.
  * Chipman et al. (2010) Hugh A Chipman, Edward I George, Robert E McCulloch, et al. 2010.  Bart: Bayesian additive regression trees.  _The Annals of Applied Statistics_ , 4(1):266–298.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations (ICLR)_.
  * Clarke et al. (2009) Charles L. A. Clarke, Nick Craswell, and Ian Soboroff. 2009.  Overview of the TREC 2009 web track.  In _Text REtrieval Conference (TREC)_.
  * Dalton et al. (2020) Jeff Dalton, Aleksandr Chuklin, Julia Kiseleva, and Mikhail Burtsev. 2020.  Proceedings of the 5th international workshop on search-oriented conversational AI (SCAI).  In _International Workshop on Search-Oriented Conversational AI (SCAI)_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , pages 4171–4186.
  * Dinan et al. (2020) Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2020.  The second conversational intelligence challenge (ConvAI2).  In _The NeurIPS’18 Competition_ , pages 187–208.
  * Gao et al. (2019) Yifan Gao, Piji Li, Irwin King, and Michael R. Lyu. 2019.  Interconnected question generation with coreference alignment and conversation flow modeling.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 4853–4862.
  * Hancock et al. (2019) Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazaré, and Jason Weston. 2019\.  Learning from dialogue after deployment: Feed yourself, chatbot!  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 3667–3684.
  * Huang et al. (2020) Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020.  Challenges in building intelligent open-domain dialog systems.  _ACM Transactions on Information Systems (TOIS)_ , 38(3):1–32.
  * Jiang et al. (2015) Di Jiang, Kenneth Wai-Ting Leung, Lingxiao Yang, and Wilfred Ng. 2015.  Query suggestion with diversification and personalization.  _Knowledge-Based Systems_ , 89:553–568.
  * Kato and Tanaka (2016) Makoto P Kato and Katsumi Tanaka. 2016.  To suggest, or not to suggest for queries with diverse intents: Optimizing search result presentation.  In _ACM International Conference on Web Search and Data Mining (WSDM)_ , pages 133–142.
  * Kiseleva et al. (2016a) Julia Kiseleva, Kyle Williams, Ahmed Hassan Awadallah, Aidan C Crook, Imed Zitouni, and Tasos Anastasakos. 2016a.  Predicting user satisfaction with intelligent assistants.  In _ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)_ , pages 45–54.
  * Kiseleva et al. (2016b) Julia Kiseleva, Kyle Williams, Jiepu Jiang, Ahmed Hassan Awadallah, Aidan C Crook, Imed Zitouni, and Tasos Anastasakos. 2016b.  Understanding user satisfaction with intelligent assistants.  In _Conference on Human Information Interaction and Retrieval (CHIIR)_ , pages 121–130.
  * Kocielnik et al. (2019) Rafal Kocielnik, Saleema Amershi, and Paul N Bennett. 2019.  Will you accept an imperfect AI? exploring designs for adjusting end-user expectations of AI systems.  In _CHI Conference on Human Factors in Computing Systems (CHI)_ , pages 1–14.
  * Kong and Allan (2014) Weize Kong and James Allan. 2014.  Extending faceted search to the general web.  In _ACM International Conference on Conference on Information and Knowledge Management (CIKM)_ , pages 839–848.
  * Kong and Allan (2016) Weize Kong and James Allan. 2016.  Precision-oriented query facet extraction.  In _ACM International on Conference on Information and Knowledge Management (CIKM)_ , pages 1433–1442.
  * Krasakis et al. (2020) Antonios Minas Krasakis, Mohammad Aliannejadi, Nikos Voskarides, and Evangelos Kanoulas. 2020.  Analysing the effect of clarifying questions on document ranking in conversational search.  In _ACM SIGIR International Conference on the Theory of Information Retrieval (ICTIR)_ , pages 129–132.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Li et al. (2017) Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc’Aurelio Ranzato, and Jason Weston. 2017.  Dialogue learning with human-in-the-loop.  In _International Conference on Learning Representations (ICLR)_.
  * Li et al. (2019a) Margaret Li, Jason Weston, and Stephen Roller. 2019a.  Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons.  _arXiv preprint arXiv:1909.03087_.
  * Li et al. (2016) Xiujun Li, Zachary C Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen. 2016.  A user simulator for task-completion dialogues.  _arXiv preprint arXiv:1612.05688_.
  * Li et al. (2019b) Ziming Li, Julia Kiseleva, and Maarten de Rijke. 2019b.  Dialogue generation: From imitation learning to inverse reinforcement learning.  In _AAAI Conference on Artificial Intelligence (AAAI)_ , pages 6722–6729.
  * Li et al. (2020) Ziming Li, Sungjin Lee, Baolin Peng, Jinchao Li, Julia Kiseleva, Maarten de Rijke, Shahin Shayandeh, and Jianfeng Gao. 2020.  Guided dialogue policy learning without adversarial learning in the loop.  In _Findings of the Association for Computational Linguistics (EMNLP)_ , pages 2308–2317.
  * Li et al. (2021) Ziming Li, Dookun Park, Julia Kiseleva, Young-Bum Kim, and Sungjin Lee. 2021.  DEUS: A data-driven approach to estimate user satisfaction in multi-turn dialogues.  _arXiv preprint arXiv:2103.01287_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Lotze et al. (2021) Tom Lotze, Stefan Klut, Mohammad Aliannejadi, and Evangelos Kanoulas. 2021.  Ranking clarifying questions based on predicted user engagement.  _arXiv preprint arXiv:2103.06192_.
  * Lowe et al. (2017) Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017.  Towards an automatic turing test: Learning to evaluate dialogue responses.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 1116–1126.
  * Nass and Moon (2000) Clifford Nass and Youngme Moon. 2000.  Machines and mindlessness: Social responses to computers.  _Journal of social issues_ , 56(1):81–103.
  * Novikova et al. (2017) Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017.  Why we need new evaluation metrics for NLG.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2241–2252.
  * Ou and Lin (2020) Wenjie Ou and Yue Lin. 2020.  A clarifying question selection system from ntes_along in ConvAI3 challenge.  _arXiv preprint arXiv:2010.14202_.
  * Radev et al. (2002) Dragomir R. Radev, Hong Qi, Harris Wu, and Weiguo Fan. 2002.  Evaluating web-based question answering systems.  In _International Conference on Language Resources and Evaluation (LREC)_ , pages 1153–1156.
  * Radlinski and Dumais (2006) Filip Radlinski and Susan Dumais. 2006.  Improving personalized web search using result diversification.  In _ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)_ , pages 691–692.
  * Rao and III (2018) Sudha Rao and Hal Daumé III. 2018.  Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 2737–2746.
  * Reiter (2018) Ehud Reiter. 2018.  A structured review of the validity of bleu.  _Computational Linguistics_ , 44(3):393–401.
  * Robertson et al. (1995) Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995.  Okapi at TREC-3.  _Nist Special Publication Sp_ , 109:109.
  * Roller et al. (2021) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021\.  Recipes for building an open-domain chatbot.  In _Conference of the European Chapter of the Association for Computational Linguistics (EACL)_ , pages 300–325.
  * Rosset et al. (2020) Corby Rosset, Chenyan Xiong, Xia Song, Daniel Campos, Nick Craswell, Saurabh Tiwary, and Paul Bennett. 2020.  Leading conversational search by suggesting useful questions.  In _The Web Conference_ , pages 1160–1170.
  * See et al. (2019a) Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, and Christopher D. Manning. 2019a.  Do massively pretrained language models make better storytellers?  In _Conference on Computational Natural Language Learning (CoNLL)_ , pages 843–861.
  * See et al. (2019b) Abigail See, Stephen Roller, Douwe Kiela, and Jason Weston. 2019b.  What makes a good conversation? how controllable attributes affect human judgments.  In _Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , pages 1702–1723.
  * Sekulic et al. (2021) Ivan Sekulic, Mohammad Aliannejadi, and Fabio Crestani. 2021.  User engagement prediction for clarification in search.  In _European Conference on Information Retrieval (ECIR)_ , pages 619–633.
  * Soleimani et al. (2021) Amir Soleimani, Christof Monz, and Marcel Worring. 2021.  NLQuAD: A non-factoid long question answering data set.  In _Conference of the European Chapter of the Association for Computational Linguistics (EACL)_ , pages 1245–1255.
  * Su et al. (2018) Shang-Yu Su, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Yun-Nung Chen. 2018\.  Discriminative deep dyna-q: Robust planning for dialogue policy learning.  In _Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3813–3823.
  * Takanobu et al. (2019) Ryuichi Takanobu, Hanlin Zhu, and Minlie Huang. 2019.  Guided dialog policy learning: Reward estimation for multi-domain task-oriented dialog.  In _Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 100–110.
  * Trienes and Balog (2019) Jan Trienes and Krisztian Balog. 2019.  Identifying unclear questions in community question answering websites.  In _European Conference on Information Retrieval (ECIR)_ , pages 276–289.
  * Vallet and Castells (2012) David Vallet and Pablo Castells. 2012.  Personalized diversification of search results.  In _ACM SIGIR conference on Research and Development in Information Retrieval (SIGIR)_ , pages 841–850.
  * Voorhees (1999) EM Voorhees. 1999.  Proceedings of the 8th text retrieval conference.  _TREC-8 Question Answering Track Report_ , pages 77–82.
  * Wang et al. (2018) Yansen Wang, Chenyi Liu, Minlie Huang, and Liqiang Nie. 2018.  Learning to ask questions in open-domain conversational systems with typed decoders.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 2193–2203.
  * Wang et al. (2013) Yining Wang, Liwei Wang, Yuanzhi Li, Di He, and Tie-Yan Liu. 2013.  A theoretical analysis of NDCG type ranking measures.  In _Conference on Learning Theory (COLT)_ , pages 25–54.
  * Zamani et al. (2020a) Hamed Zamani, Susan Dumais, Nick Craswell, Paul Bennett, and Gord Lueck. 2020a.  Generating clarifying questions for information retrieval.  In _The Web Conference_ , pages 418–428.
  * Zamani et al. (2020b) Hamed Zamani, Gord Lueck, Everest Chen, Rodolfo Quispe, Flint Luu, and Nick Craswell. 2020b.  MIMICS: A large-scale data collection for search clarification.  In _ACM International Conference on Information and Knowledge Management (CIKM)_ , pages 3189–3196.
  * Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018.  Personalizing dialogue agents: I have a dog, do you have pets too?  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 2204–2213.
  * Zhang et al. (2020) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020.  DIALOGPT : Large-scale generative pre-training for conversational response generation.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , pages 270–278.