11institutetext: ShareChat
11email: {hiteshsagtani, jeunen<EMAIL_ADDRESS>
# Learning-to-Rank with Nested Feedback

Hitesh Sagtani  0009-0003-6995-1912 Olivier Jeunen  0000-0001-6256-5814
Aleksei Ustimenko  0009-0006-4942-7779

###### Abstract

Many platforms on the web present ranked lists of content to users, typically
optimized for engagement-, satisfaction- or retention- driven metrics.
Advances in the Learning-to-Rank (LTR) research literature have enabled rapid
growth in this application area. Several popular interfaces now include nested
lists, where users can enter a 2nd-level feed via any given 1st-level item.
Naturally, this has implications for evaluation metrics, objective functions,
and the ranking policies we wish to learn. We propose a theoretically grounded
method to incorporate 2nd-level feedback into any 1st-level ranking model.
Online experiments on a large-scale recommendation system confirm our
theoretical findings.

###### Keywords:

Learning-to-Rank Recommender systems User feedback

##  1 Introduction & Related Work

Rankings are at the heart of how users interact with content on the web. This
holds for applications and use-cases across web search and e-commerce
recommendations to streaming and social media platforms. Indeed, platforms use
ranked list interfaces to serve content to users. Machine learning models
typically power these rankings, learned to optimize some metrics deemed
relevant to the business or its users. Due to this wealth of application
areas, the Learning-to-Rank (LTR) literature has seen vast industry adoption
[23, 33, 13, 39, 14, 18].

Nevertheless, real-world examples of ranking interfaces often deviate from the
traditional setup where a single ranked list is shown. Famous examples here
include the _gridwise_ page layout that is now standardized in video streaming
platforms [11], and the research literature has looked into more general
interfaces as well [32, 40]. These works keep a broad focus, making few
assumptions about the setting to provide effective methods with universal
appeal.

Figure 1: A nested-feed interface. When receiving feedback on item $n.k$ in
the 2nd-level feed (denoted $\star$), the 1st-level item $n$ should be
attributed as well.

Recently, _nested_ ranking lists have started to gain in popularity,
particularly seeing adoption in short-video feeds on social media platforms
such as Reddit, Instagram, and ShareChat [16]. Here, users are presented with
a scrollable 1st-level feed, where they can enter a full-screen 2nd-level feed
via any given 1st-level item. This differs from the grid layout discussed
above, as only a single level is presented to the user at any given time.
Figure 1 visualizes such a layout of _nested_ feeds, which are the core topic
of this paper.

A naïve modeling approach would instantiate both levels with independent
ranking policies that optimize level-specific objectives. This implicitly
assumes that rewards across feeds are independent, which ignores the
hierarchically nested structure. Indeed, as shown in Figure 1, positive
feedback on 2nd-level items should be (partially) attributed to the relevant
1st-level item, as to allow the 1st-level ranking policy to learn from this
_nested_ feedback.

Existing methods in the literature that go beyond single lists either deal
with complex settings where no assumptions are placed on the reward or
examination dependencies in the interface [32], or page-level re-ranking
strategies when multiple independent lists are present [40]. Even though
nested ranking interfaces are prevalent in practical applications, they are
currently under-explored in the research literature. Our work formally
introduces this problem setting. Under the commonly assumed position-based
model [8], we theoretically derive the optimal objective for the 1st-level
ranking model when 2nd-level nested feedback is available.

Related Work. Learning-to-Rank (LTR) is a classical problem in information
retrieval that has received extensive research attention [29, 27]. It has
found widespread adoption in various application areas, including web search
[35], question answering [43], e-commerce recommendations [24], streaming
platforms [23], and recommendation systems [9, 22, 18]. LTR approaches can be
categorized into different types, such as pointwise [28], pairwise [26, 4,
10], and listwise [41, 7, 42, 6] methods. Recent advancements in LTR include
using online learning algorithms with non-linear models [31] and the proposal
of stochastic bandit algorithms like BatchRank [45]. In the context of social
media platforms, LTR algorithms are usually trained on implicit signals
derived from user interactions, such as user clicks [19], dwell time [44], and
various other engagement signals [25], that are then treated as indicators of
relevance. However, these signals are subject to biases, such as positional
bias, where the ranking position influences user clicks. To address this
issue, many studies have explored methods to mitigate bias in ranking using
counterfactual inference frameworks through empirical risk minimization [20,
1]. Qingyao et al. [2] evaluated and compared the performance of such state-
of-the-art unbiased LTR algorithms.

##  2 Modeling Nested Ranking Signals

In the rest of the work, we refer to the 1st-level feed as L1 (Level 1), and
the 2nd-level feed as L2 (Level 2). To calculate the final relevance label for
a feed, we combine the following user signals using linear scalarization:
_likes_ , _shares_ , favourites, and _video clicks_. The latter signal
indicates that a user clicks on an L1 video to enter the L2 feed. We observe
the correlation among these signals to be significantly different for L1 and
L2 feed for our platform, highlighting variations in user behaviours across
feeds, resulting in different trade-offs between signals.

We consider some distribution of users $u\sim\mathcal{U}$ that interact with
our platform. Classical LTR deals with the problem of finding and evaluating a
ranking for a list of items $A=\\{a_{1},\ldots,a_{n}\\}$ with per-item
relevance signals $y_{i}\forall i\in[1,n]$, e.g., a click or crowd-sourced
label. Typically, ranking quality is measured by Discounted Cumulative Gain
(DCG):

$\mathrm{DCG}=\sum_{i=1}^{n}\frac{y_{i}}{\log_{2}(1+i)}.$ (1)

Given some model $f(u,a)$ we can sort items $a\in A$ in decreasing order to
obtain a ranking. LTR is then: finding a model $f(u,a)$ such that
$\mathbb{E}_{u\sim\mathcal{U}}\mathrm{DCG}$ is maximized. $\mathrm{DCG}$
objectives can be effectively optimized by state-of-the-art LTR approaches
like LambdaRank [5], StochasticRank [38] or YetiRank [12].

Denote $R(u,a)$ as the online relevance signal that we observe for the ranked
list. Note that DCG adheres to the Position-Based Model (PBM) [8], when we
assume that the probability of user viewing position $i$ is equal to
$\frac{1}{\log_{2}(1+i)}$. As a result,
$y_{i}=\frac{R(u,a_{i})}{\mathbb{P}(u\text{ viewed position }i)}$ is an
unbiased estimator of $\mathbb{E}\big{(}R(u,a_{i})|u,a,\text{viewed})$. Under
these reasonable assumptions, $\mathrm{DCG}$ can be seen as an offline
estimator of an online metric $Q$ [17, 15]:

$\displaystyle Q=\mathbb{E}_{u\sim\mathcal{U}}\sum_{i=1}^{n}R(u,a_{i})=$ (2)
$\displaystyle\mathbb{E}_{u\sim\mathcal{U}}\sum_{i=1}^{n}\mathbb{E}\big{(}R(u,a_{i})|u,a,\text{viewed})\mathbb{P}(u\text{
viewed position }i)=\mathbb{E}_{u\sim\mathcal{U}}\mathrm{DCG}.$ (3)

In our case, when the user clicks on the item $a_{i}$, our system retrieves
another _nested_ list of items $B_{i}=\\{b_{i1},\ldots,b_{im}\\}$, ranks them,
and presents them to the user. For an item $a\in A$ we denote as $R_{A}(u,a)$
a _relevance_ signal (e.g., an indicator of a positive interaction) that
indicates user preference on the L1 feed, and we analogously define
$R_{B}(u,b)$ for the L2 feed. The goal of the LTR model is to maximize the
following online metric:

$Q=\mathbb{E}_{u\sim\mathcal{U}}\sum_{i=1}^{n}\big{(}R_{A}(u,a_{i})+\sum_{j=1}^{m}R_{B}(u,b_{ij})\big{)}.$
(4)

Suppose we ignore the contribution from $R_{B}$ and assume a classic setup
with only $R_{A}$: items with lower L1 relevance but better L2 feeds are
penalised and thus, a model trained only on L1 signals will have _worse_
online performance for metric $Q$. This leads to a degradation of the
recommendation system overall, while the “quality” of ranking on the L1 feed
appears high.

In our setup, we consider only how to train a ranker model to rank list $A$ so
that $Q$ is maximized while assuming the ranking for $B$ is fixed. From the
very definition of $Q$ it is seen that we should define
$\widetilde{R}(u,a_{i})=R_{A}(u,a_{i})+\sum_{j=1}^{m}R_{B}(u,b_{ij})$ as the
new relevance signal for the item $a_{i}$. That is, the relevance signal for
the L1 feed should account for the relevance signal on the L2 feed if we want
to find a ranker that maximizes overall quality measured by $Q$.

Thus, to train a model we consider historical logs consisting of $(u,A,B,Y)$
where $Y=(y_{ij})_{i=1,j=0}^{n,m}$ denotes observed values for $y_{i0}\sim
R_{A}(u,a_{i})$ and $y_{ij}\sim R_{B}(u,b_{ij}),j>0$. These logs are
transformed into triplets $(u,A,\widetilde{Y})$, where we define
$\widetilde{y}_{i}=\sum_{j=0}^{m}y_{ij}\sim\widetilde{R}(u,a_{i})$. The
dataset $\\{(u,A,\widetilde{Y})\\}$ now resembles a classic LTR dataset and,
therefore, can serve as input to well-established LTR methods.

We also note that because the ranking on the L2 feed (i.e. the ranking of list
$B$) is fixed, that means that we should not measure the contribution from the
L2 feed using $\mathrm{DCG}$. This is because our observed reward in the logs
$R_{B}(u,b_{ij})$ is already positionally biased. To see that, we can formally
write:

$\displaystyle\mathbb{E}_{u\sim\mathcal{U}}$
$\displaystyle\sum_{j=1}^{m}R_{B}(u,b_{ij})=\mathbb{E}_{u\sim\mathcal{U}}\frac{R_{B}(u,b_{ij})}{\mathbb{P}(u\text{
viewed position }j)}\mathbb{P}(u\text{ viewed position }j)$ (5)

This means that we can get $\mathrm{DCG}$ if we consider debiased labels
$\frac{R_{B}(u,b_{ij})}{\mathbb{P}(u\text{ viewed position }j)}$ but since the
ranking of $B$ is fixed, those debiased labels have _the same_ denominator as
the positional bias and, thus, it cancels out. Therefore, we should not
discount observed relevance signals on $B$ and include them just as sums.

Our final feed-level relevance signals, denoted as $R_{A}(u,a_{i})$ for the L1
feed and $R_{B}(u,b_{ij})$ for the L2 feed, are obtained by linearly combining
individual user signals. These signals are fine-tuned internally to optimize
user retention. However, since our objective is to evaluate the incorporation
of user feedback from the L2 feed into the L1 feed, we do not delve into the
specifics of how we tune the weights for each individual feed. In the
subsequent section, we assign the following synthetic labels as the relevance
signals for evaluation: $S_{1}$ as $R_{A}(u,a_{i})$, $S_{2}$ as
$\widetilde{R}(u,a_{i})$ with DCG of $R_{B}(u,b_{ij})$, and $S_{3}$ as
$\widetilde{R}(u,a_{i})$ with sum of $R_{B}(u,b_{ij})$.

##  3 Experimental Validation

To the best of our knowledge, no datasets containing logged user feedback for
nested feed interfaces are publicly available. For this reason, we resort to a
proprietary dataset obtained from ShareChat, a widely popular social media
application with over 180 million monthly active users across 18 regional
languages in India, to substantiate our hypothesis empirically.

We leverage a range of dynamic user and post attributes based on embeddings,
user sign-up date, genre, tags, fatigue score [37] and language, along with
several interaction-based attributes. We find strong correlations between the
relevance label with certain interaction features: the dot product between
user and post embeddings is one such example. We applied negative sampling to
the dataset based on a union of all individual signals, i.e. _likes_ ,
_shares_ , _favs_ and _clicks_. We collected a random sample of 100 million
instances, where less than 5% of the dataset had a positive signal, while the
remaining instances were labeled as ‘0’. To ensure representative data for
training, validation, and testing, we used stratified sampling, dividing the
dataset in a 70:15:15 ratio, respectively.

###  3.1 Offline Training and Experiment Results

Gradient-Boosted Decision Tree (GBDT) methods such as LambdaMart have long
remained the state-of-the-art approach [36] for tabular LTR datasets. Recent
empirical evaluations have shown YetiRank to outperform LambdaMart and other
competing algorithms in most cases [30]. To evaluate the synthetic labels, we
use the YetiRank algorithm implemented in the Catboost [34] library. For
hyperparameter tuning, we perform around 100 iterations of Bayesian
optimization using the hyperopt [3] library for each synthetic label. To
address the class imbalance problem in the dataset during training, we scale
the loss of positive class by the ratio of negative to the positive examples
using scale_pos_weight parameter. We use an overfitting detector on the
validation dataset and stop the training if there is no improvement after 50
iterations on _DCG@10_ and use the best-performing model on the validation set
for each signal.

Table 1: Offline % loss in DCG of predictor label compared to a model trained and ranked only on various “true” relevance signals. DCG |  | Label | True Relevance Signal
---|---|---|---
@k |  | likes | shares | favs | clicks | $S_{1}$ | $S_{2}$ | $S_{3}$
3 |  | $S_{1}$ | 27.8 | 26.8 | 23.7 | 17.3 | 0 | 17.3 | 18.1
| $S_{2}$ | 24.1 | 25.2 | 20.5 | 14.8 | 10.2 | 0 | 15.1
| $S_{3}$ | 20.7 | 24.7 | 20.8 | 13.4 | 12.4 | 8.4 | 0
5 |  | $S_{1}$ | 25.9 | 25.8 | 20.3 | 15.6 | 0 | 16.5 | 17.7
| $S_{2}$ | 22.9 | 24.1 | 17.5 | 14.4 | 10.1 | 0 | 15.2
| $S_{3}$ | 19.2 | 24.3 | 18.2 | 12.6 | 10.5 | 8.1 | 0
10 |  | $S_{1}$ | 24.5 | 24.2 | 19.6 | 15.1 | 0 | 16.1 | 17.3
| $S_{2}$ | 20.8 | 23.6 | 16.7 | 11.6 | 9.3 | 0 | 14.4
| $S_{3}$ | 17.8 | 23.2 | 17.4 | 10.8 | 10.4 | 7.7 | 0

We evaluated the predictors ($S_{1}$, $S_{2}$, $S_{3}$) discussed in the
previous section, considering likes, shares, favs, clicks, $S_{1}$, $S_{2}$
and $S_{3}$ as the true relevance labels and evaluated on the DCG@k metric.
Table 1 shows the % loss in DCG metric values of predictor models compared to
the model trained on the true relevance signal for ranking. For example, when
using _likes_ as true relevance, we observe a 20.7% decrease in _DCG@3_ metric
for model trained on $S_{3}$ synthetic label, compared to the model trained on
_likes_ signal.

From Table 1, we observe that models trained on $S_{2}$ and $S_{3}$ labels
exhibit lower DCG loss compared to $S_{1}$ for all the individual user
signals: likes, shares, favs and clicks, with $S_{3}$ having minimum loss
across most of the signals. This validates our hypothesis that incorporating
L2 feedback leads to improved overall ranking, with a sum-aggregation of L2
feedback to be optimal over discounting, as outlined in Section 2.

We also treat each synthetic label as the _true_ relevance signal to compare
them relatively. We note that the % loss in DCG is much lower when using
$S_{3}$ as predictor for other synthetic labels as the true relevance label
($S_{1}$, $S_{2}$), followed by $S_{2}$ and finally $S_{1}$ exhibiting the
highest loss. Since, $S_{3}$ and $S_{2}$ predictors have both the L1 and L2
feedback information in their synthetic label, they are able to capture user
behavior on both the feeds, compared to the $S_{1}$ predictor which has
feedback for only L1 feed. In addition, the loss is much lower when using the
$S_{3}$ predictor on $S_{2}$ as true relevance compared to the $S_{2}$
predictor with $S_{3}$ as true relevance, the $S_{3}$ label is better able to
capture the information on L2 feed than $S_{2}$.

###  3.2 Online A/B Experimentation

Based on the offline results, we conducted an A/B experiment to evaluate
various synthetic labels. Table 2(b) tabulates the online metrics for ranking
based on $S_{2}$ and $S_{3}$ labels in variant-1 and variant-2 respectively,
with $S_{1}$ being the control.

From Table 2(a), we observe that overall engagements (likes, shares, favs)
have increased for both variants, indicating better posts on the L1 feed that
lead to more engagements on the L2 feed. We also observe increased clicks on
the L1 feed, indicating higher user convergence to the L2 feed compared to the
model trained solely on the $S_{1}$ label. While there is a decrease in dwell
time on the L1 feed, there is an increase for the L2 feed, suggesting that
users spend more time overall in the L2 feed than in the L1 feed. The increase
in clicks on the L1 feed also supports this insight.

To further validate our hypothesis of user convergence to the L2 feed, Table
2(b) shows a decrease in L1 depth (indicating the count of subsequent feed
fetches) and an increase in L2 depth. Increment in L2 transition (#times users
switch from the L1 feed to the L2 feed) and decrease in S2L2 (Second to L2:
time in seconds users take to open any post on L2 feed after session start)
confirms higher and early user convergence to L2 feed. Including the feedback
from the L2 feed improves user experience on the overall platform as indicated
by the platform-level metrics in Table 2(b), with metrics showing increased
user retention, engagements, and interactions with the platform.

We also examined the composition of the suggested posts between the variants
and observed that including the L2 feedback results in more suggestions from
personalized candidate generators like field-aware factorization machines [21]
compared to non-personalized candidate generators (e.g. popularity). However,
we did not observe any statistically significant difference in the genre
distribution of the posts. Finally, we found that variant-2 performs better in
most of the metrics compared to variant-1, which is consistent with the
offline numbers we observed and our hypothesis of using the sum of feedback on
the L2 feed being optimal compared to when discounting is used.

Table 2: Online % gain in metrics when compared to the model trained only on
L1 signal. All results are statistically significant according to a 2-tailed
t-test at $p<0.05$ after Bonferroni correction, except for those marked by+.

(a) Engagement signals for L1 and L2 feed Feed | variant | likes | shares | favs | dwell time | clicks
---|---|---|---|---|---|---
L1 | variant-1 | 5.9 | 4.3 | 5.7 | -1.0+ | 5.1
variant-2 | 6.5 | 5.6 | 5.2 | -1.2+ | 7.1
L2 | variant-1 | 3.6 | 1.1+ | 3.1 | 1.5 | -
variant-2 | 4.2 | 2.7 | 2.5 | 3.3 | -

(b) Platform & transition metrics | L1 to L2 transition metrics | Platform level metrics
---|---|---
variant | S2L2 | L1 depth | L2 depth | L2 transition | Engagements | #Session | Retention
variant-1 | -0.9 | -1.43+ | 2.53 | 8.15 | 0.43 | 0.3+ | 0.12
variant-2 | -2.1 | -4.14 | 5.42 | 11.23 | 0.71 | 0.5 | 0.17

##  4 Discussion & Future Work

In this work, we have proposed a modeling framework for the nested feed
structure that is prevalent in modern applications. We provided a theoretical
explanation for why optimizing for combined signals on both feeds is more
effective than optimizing them independently. We considered different user
feedback signals as the proper relevance signal. We used the DCG metric to
demonstrate that incorporating L2 feedback in the L1 predictor reduces the
loss in DCG compared to using the L1 predictor alone. Furthermore, we observed
consistency between our online metric and the offline metric and observed
improvement in both short-term engagement and user retention at the platform
level. Our current work focused on optimizing the ranking policy for the L1
feed while keeping the L2 feed ranking policy fixed. We envision future
extensions of this work where we can jointly optimize the L1, and L2 feeds
using a single policy trained end-to-end.

## References

  * [1] Ai, Q., Bi, K., Luo, C., Guo, J., Croft, W.B.: Unbiased learning to rank with unbiased propensity estimation. In: The 41st international ACM SIGIR conference on research & development in information retrieval. pp. 385–394 (2018)
  * [2] Ai, Q., Yang, T., Wang, H., Mao, J.: Unbiased learning to rank: online or offline? ACM Transactions on Information Systems (TOIS) 39(2), 1–29 (2021)
  * [3] Bergstra, J., Yamins, D., Cox, D.D., et al.: Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms. In: Proc. of the 12th Python in science conference. vol. 13, p. 20. Citeseer (2013)
  * [4] Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., Hullender, G.: Learning to rank using gradient descent. In: Proc. of the 22nd international conference on Machine learning. pp. 89–96 (2005)
  * [5] Burges, C., Ragno, R., Le, Q.: Learning to rank with nonsmooth cost functions. Advances in neural information processing systems 19 (2006)
  * [6] Burges, C.J.: From ranknet to lambdarank to lambdamart: An overview. Learning 11(23-581), 81 (2010)
  * [7] Cao, Z., Qin, T., Liu, T.Y., Tsai, M.F., Li, H.: Learning to rank: from pairwise approach to listwise approach. In: Proc. of the 24th international conference on Machine learning. pp. 129–136 (2007)
  * [8] Chuklin, A., Markov, I., de Rijke, M.: Click Models for Web Search. Morgan & Claypool (2015). https://doi.org/10.2200/S00654ED1V01Y201507ICR043
  * [9] Duan, Y., Jiang, L., Qin, T., Zhou, M., Shum, H.Y.: An empirical study on learning to rank of tweets. In: Proc. of the 23rd international conference on computational linguistics (Coling 2010). pp. 295–303 (2010)
  * [10] Freund, Y., Iyer, R., Schapire, R.E., Singer, Y.: An efficient boosting algorithm for combining preferences. Journal of machine learning research 4(Nov), 933–969 (2003)
  * [11] Gomez-Uribe, C.A., Hunt, N.: The netflix recommender system: Algorithms, business value, and innovation. ACM Trans. Manage. Inf. Syst. 6(4) (dec 2016). https://doi.org/10.1145/2843948
  * [12] Gulin, A., Kuralenok, I., Pavlov, D.: Winning the transfer learning track of yahoo!’s learning to rank challenge with yetirank. In: Proc. of the Learning to Rank Challenge. pp. 63–76. PMLR (2011)
  * [13] Haldar, M., Ramanathan, P., Sax, T., Abdool, M., Zhang, L., Mansawala, A., Yang, S., Turnbull, B., Liao, J.: Improving deep learning for airbnb search. In: Proc. of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. p. 2822–2830. KDD ’20, ACM (2020). https://doi.org/10.1145/3394486.3403333
  * [14] Jagerman, R., Wang, X., Zhuang, H., Qin, Z., Bendersky, M., Najork, M.: Rax: Composable learning-to-rank using jax. In: Proc. of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. p. 3051–3060. KDD ’22, ACM (2022). https://doi.org/10.1145/3534678.3539065
  * [15] Jeunen, O.: Offline approaches to recommendation with online success. Ph.D. thesis, University of Antwerp (2021)
  * [16] Jeunen, O.: A probabilistic position bias model for short-video recommendation feeds. In: Proc. of the 17th ACM Conference on Recommender Systems. p. 675–681. RecSys ’23, ACM (2023). https://doi.org/10.1145/3604915.3608777
  * [17] Jeunen, O., Potapov, I., Ustimenko, A.: On (normalised) discounted cumulative gain as an offline evaluation metric for top-$n$ recommendation (2023)
  * [18] Jeunen, O., Sagtani, H., Doi, H., Karimov, R., Pokharna, N., Kalim, D., Ustimenko, A., Green, C., Shi, W., Mehrotra, R.: On gradient boosted decision trees and neural rankers: A case-study on short-video recommendations at ShareChat. In: Proc. of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation. FIRE ’23, ACM (2023)
  * [19] Joachims, T., Granka, L., Pan, B., Hembrooke, H., Gay, G.: Accurately interpreting clickthrough data as implicit feedback. In: Acm Sigir Forum. vol. 51, pp. 4–11. Acm New York, NY, USA (2017)
  * [20] Joachims, T., Swaminathan, A., Schnabel, T.: Unbiased learning-to-rank with biased feedback. In: Proc. of the tenth ACM international conference on web search and data mining. pp. 781–789 (2017)
  * [21] Juan, Y., Zhuang, Y., Chin, W.S., Lin, C.J.: Field-aware factorization machines for ctr prediction. In: Proc. of the 10th ACM conference on recommender systems. pp. 43–50 (2016)
  * [22] Karatzoglou, A., Baltrunas, L., Shi, Y.: Learning to rank for recommender systems. In: Proc. of the 7th ACM Conference on Recommender Systems. pp. 493–494 (2013)
  * [23] Karmaker Santu, S.K., Sondhi, P., Zhai, C.: On application of learning to rank for e-commerce search. In: Proc. of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 475–484. SIGIR ’17, ACM (2017). https://doi.org/10.1145/3077136.3080838
  * [24] Karmaker Santu, S.K., Sondhi, P., Zhai, C.: On application of learning to rank for e-commerce search. In: Proc. of the 40th international ACM SIGIR conference on research and development in information retrieval. pp. 475–484 (2017)
  * [25] Lalmas, M., Hong, L.: Tutorial on metrics of user engagement: Applications to news, search and e-commerce. In: Proc. of the Eleventh ACM International Conference on Web Search and Data Mining. pp. 781–782 (2018)
  * [26] Leaman, R., Islamaj Doğan, R., Lu, Z.: Dnorm: disease name normalization with pairwise learning to rank. Bioinformatics 29(22), 2909–2917 (2013)
  * [27] Li, H.: A short introduction to learning to rank. IEICE TRANSACTIONS on Information and Systems 94(10), 1854–1862 (2011)
  * [28] Li, P., Wu, Q., Burges, C.: Mcrank: Learning to rank using multiple classification and gradient boosting. Advances in neural information processing systems 20 (2007)
  * [29] Liu, T.Y., et al.: Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval 3(3), 225–331 (2009)
  * [30] Lyzhin, I., Ustimenko, A., Gulin, A., Prokhorenkova, L.: Which tricks are important for learning to rank? In: International Conference on Machine Learning. pp. 23264–23278. PMLR (2023)
  * [31] Oosterhuis, H., de Rijke, M.: Differentiable unbiased online learning to rank. In: Proc. of the 27th ACM international conference on information and knowledge management. pp. 1293–1302 (2018)
  * [32] Oosterhuis, H., de Rijke, M.: Ranking for relevance and display preferences in complex presentation layouts. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. p. 845–854. SIGIR ’18, ACM (2018). https://doi.org/10.1145/3209978.3209992
  * [33] Pasumarthi, R.K., Bruch, S., Wang, X., Li, C., Bendersky, M., Najork, M., Pfeifer, J., Golbandi, N., Anil, R., Wolf, S.: Tf-ranking: Scalable tensorflow library for learning-to-rank. In: Proc. of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. p. 2970–2978. KDD ’19, ACM (2019). https://doi.org/10.1145/3292500.3330677
  * [34] Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A.V., Gulin, A.: Catboost: unbiased boosting with categorical features. Advances in neural information processing systems 31 (2018)
  * [35] Qin, T., Liu, T.Y., Zhang, X.D., Wang, D.S., Xiong, W.Y., Li, H.: Learning to rank relational objects and its application to web search. In: Proc. of the 17th international conference on World Wide Web. pp. 407–416 (2008)
  * [36] Qin, Z., Yan, L., Zhuang, H., Tay, Y., Pasumarthi, R.K., Wang, X., Bendersky, M., Najork, M.: Are neural rankers still outperformed by gradient boosted decision trees? (2021)
  * [37] Sagtani, H., Jhanwar, M.G., Gupta, A., Mehrotra, R.: Quantifying and leveraging user fatigue for interventions in recommender systems. In: Proc. of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (2023)
  * [38] Ustimenko, A., Prokhorenkova, L.: Stochasticrank: Global optimization of scale-free discrete functions. In: International Conference on Machine Learning. pp. 9669–9679. PMLR (2020)
  * [39] Wang, R., Shivanna, R., Cheng, D., Jain, S., Lin, D., Hong, L., Chi, E.: Dcn v2: Improved deep & cross network and practical lessons for web-scale learning to rank systems. In: Proc. of the Web Conference 2021. p. 1785–1797. WWW ’21, ACM (2021). https://doi.org/10.1145/3442381.3450078
  * [40] Xi, Y., Lin, J., Liu, W., Dai, X., Zhang, W., Zhang, R., Tang, R., Yu, Y.: A bird’s-eye view of reranking: From list level to page level. In: Proc. of the Sixteenth ACM International Conference on Web Search and Data Mining. p. 1075–1083. WSDM ’23, ACM (2023). https://doi.org/10.1145/3539597.3570399
  * [41] Xia, F., Liu, T.Y., Wang, J., Zhang, W., Li, H.: Listwise approach to learning to rank: theory and algorithm. In: Proc. of the 25th international conference on Machine learning. pp. 1192–1199 (2008)
  * [42] Xu, J., Li, H.: Adarank: a boosting algorithm for information retrieval. In: Proc. of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. pp. 391–398 (2007)
  * [43] Yang, L., Ai, Q., Spina, D., Chen, R.C., Pang, L., Croft, W.B., Guo, J., Scholer, F.: Beyond factoid qa: effective methods for non-factoid answer sentence retrieval. In: Advances in Information Retrieval: 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20–23, 2016. Proceedings 38. pp. 115–128. Springer (2016)
  * [44] Yi, X., Hong, L., Zhong, E., Liu, N.N., Rajan, S.: Beyond clicks: dwell time for personalization. In: Proc. of the 8th ACM Conference on Recommender systems. pp. 113–120 (2014)
  * [45] Zoghi, M., Tunys, T., Ghavamzadeh, M., Kveton, B., Szepesvari, C., Wen, Z.: Online learning to rank in stochastic click models. In: International conference on machine learning. pp. 4199–4208. PMLR (2017)


# Fast Anticipatory Motion Planning for Close-Proximity Human-Robot
Interaction

Sam Scheele1, Pierce Howell1, Harish Ravichandar1
1 Structured Techniques for Algorithmic Robotics Lab, Georgia Institute of
Technology, Atlanta GA 30318
{scheele,pierce.howell<EMAIL_ADDRESS>
###### Abstract

Effective close-proximity human-robot interaction (CP-HRI) requires robots to
be able to both efficiently perform tasks as well as adapt to human behavior
and preferences. However, this ability is mediated by many, sometimes
competing, aspects of interaction. We propose a real-time motion-planning
framework for robotic manipulators that can simultaneously optimize a set of
both task- and human-centric cost functions. To this end, we formulate a
Nonlinear Model-Predictive Control (NMPC) problem with kino-dynamic
constraints and efficiently solve it by leveraging recent advances in
nonlinear trajectory optimization. We employ stochastic predictions of the
human partner’s trajectories in order to adapt the robot’s nominal behavior in
anticipation of its human partner. Our framework explicitly models and allows
balancing of different task- and human-centric cost functions. While previous
approaches to trajectory optimization for CP-HRI take anywhere from several
seconds to a full minute to compute a trajectory, our approach is capable of
computing one in 318 ms on average, enabling real-time implementation. We
illustrate the effectiveness of our framework by simultaneously optimizing for
separation distance, end-effector visibility, legibility, smoothness, and
deviation from nominal behavior. We also demonstrate that our approach
performs comparably to prior work in terms of the chosen cost functions, while
significantly improving computational efficiency.

##  I Introduction

###  I-A Adaptive Close-Proximity HRC

Adaptive close-proximity human-robot collaboration provides both psychological
advantages to humans (in the form of increased comfort during interation) and
efficiency advantages in task completion. Though the efficiency gains due to
adaptive motion planning likely vary significantly by task, Lasota and Shah
[8] found that adaptive, anticipatory motion planning improved task completion
speed, reduced robot idle time, and increased separation distance relative to
non-human-aware approaches on an assembly task.

A great deal of work in anticipatory motion planning focuses on non-collision,
such as Majumdar and Tedrake [11] and Brito et al. [2]. Dragan et al. [3]
notably introduced the concept of legibility, and found in [4] that legible
motion leads to greater team fluency and closer collaboration, but their
method for generating legible motion does not model humans as participatory
actors, but rather as passive observers.

###  I-B Nonlinear Model-Predictive Control

If the factors associated with natural collaboration can be modeled as
objective functions, a natural model for human-anticipatory motion planning is
Model-Predictive Control (MPC, or NMPC for a nonlinear system model), in which
we solve an optimization problem to find optimal control inputs based on
predictions of system state for some time horizon. With NMPC, we can provide
an arbitrary cost function and obtain a locally optimal solution.

Use of MPC for HRC is not new - Faroni et al. [5] proposed an MPC-based
approach that modified plans from a high-level planner to adaptively slow
trajectories that get too close to humans and potentially adjust them to
satisfy secondary objectives. Yang et al. [13] used the GPU-accelerated
stochastic MPC solver proposed by Bhardwaj et al. [1] to generate fluid motion
for human-to-robot handovers. However, we are not aware of any other
approaches that use MPC to realize anticipatory and reactive motion planning
while optimizing the path itself for natural collaboration with humans.

Figure 1: High-level flowchart of our method

###  I-C Our Approach

In this work, we formulate human-anticipatory trajectory optimization as a
NMPC problem and solve that problem using ALTRO [6], a solver designed for
nonlinear trajectory optimization. ALTRO uses a variant of the Iterated
Linear-Quadratic Regulator (iLQR) algorithm to warm-start its solutions and
achieve fast solution times, and it terminates when the performance ratio
falls below a threshold. We use a sliding-window approach and implement
optimizations such that the problem can be solved in real-time, enabling the
system to be both anticipatory and reactive.

Relative to our previous work in multi-objective trajectory optimization,
CoMOTO [7], our approach offers the advantage of being more compatible with
modern pose estimation: the proof-of-concept provided in CoMOTO relied on a
single prediction of a full human trajectory, and took tens of seconds to
optimize a solution. In contrast, our work relies only on predictions for
short time horizons, which is consistent with human pose estimation works like
[10] and [12], neither of which predict more than two seconds ahead. Further,
our method is fast enough that it can run in real time, meaning that it can
react to unexpected human motion.

##  II Method

We apply NMPC to a manipulator equipped with joint-space position control,
where the predictions for the MPC are generated by the human pose prediction
network from Luo et al. [9]. The problem is formulated as:

$\displaystyle\min_{u}\quad$ $\displaystyle C(X_{R},X_{H})\quad s.t.$ (1)
$\displaystyle X_{R}^{(0)}$ $\displaystyle=start$ (2) $\displaystyle
X_{R}^{(t+1)}$ $\displaystyle=X_{R}^{(t)}+u^{(t)}\Delta t$ (3) $\displaystyle
X_{R}^{(0)}$ $\displaystyle=goal$ (4) $\displaystyle B_{L}\leq$ $\displaystyle
u^{(t)}\leq B_{U}$ (5)

Where $X_{R}$ and $X_{H}$ are respectively the robot (joint-space) and human
(cartesian-space) trajectories, $C$ is a cost function to be presented below,
$\Delta t$ is a predefined timestep (in our case, 0.2 sec), and $B_{L}$ and
$B_{U}$ are limits on the joint velocity control vector $u$. This approach
assumes that human movement is not reactive to robot movement. This assumption
is likely false, but it may be approximately true over short timespans
(violation of this assumption in the long-term might reasonably be handled by
the reactive nature of our approach, albeit with some consequences for
optimality). Note that while the MPC model assumes the robot is equipped with
a joint-space velocity controller, in reality we use a joint-space position
controller. The use of velocity control in the MPC model allows us access to a
greater range of cost functions without sacrificing model accuracy.

In our prior work, we formulated an overall cost function as a weighted sum of
five objectives. In this work, we reformulate several of these objectives to
allow each timestep in the trajectory to be optimized independently of the
others.

###  II-A Distance Cost

The distance cost is the sum of pairwise distances between all human and robot
joints, scaled by the human prediction covariance matrices:

$C_{dist}(x,u,t)=\sum_{h}\sum_{r}\dfrac{1}{d_{h,r}(t)^{T}(\Sigma_{h}(t))^{-1}d_{h,r}(t)}$

Where $d_{h,r}(t)$ is the difference between the Cartesian positions of human
joint $h$ and robot joint $r$ at time $t$ and $\Sigma_{h}(t)$ is the
prediction covariance matrix of human joint $h$ at time $t$

###  II-B Visibility Cost

The visibility cost remains the same as in CoMOTO:

$C_{vis}(x,u,t)=\dfrac{\angle(O,\mu_{head}(t),p_{eef}(t))}{\sigma_{head}(t)}$

Where $\angle(O,\mu_{head}(t),p_{eef}(t))$ is the angle between the object the
person is assumed to be looking at, the estimated position of their head, and
the robot’s end effector, and $\sigma_{head}(t)$ is the standard deviation in
the estimation of the human head position.

###  II-C Legibility Cost

Legibility measures the ability of a human observer to predict the actual goal
of an agent $G_{R}$ from a set of candidate goals $G$ (for example, an arm
might reach for one of a few objects). Legibility can be naturally expressed
in terms of probabilities and is defined by Dragan et. al. as

$\dfrac{\int P(G_{R}|\xi_{S\rightarrow\xi(t)})f(t)dt}{\int f(t)dt}$

Where the probability term is given as:

$P(G_{R}|\xi_{S\rightarrow\xi(t)})=\dfrac{1}{Z}\dfrac{\exp(-\mathcal{D}[\xi_{S\rightarrow
Q}]-V_{G_{R}}(Q))}{\exp(-V_{G_{R}}(S))}$

Here, $\xi$ is the , $S$ is the trajectory’s start point ($\xi_{0}=S$),
$V_{G_{R}}(Q)$ is a function that gives the minimum value of $C$ for any
trajectory from $Q$ to $G_{R}$, and .

Where:

  * •

$\xi$ is the trajectory of a robot (in our case, the cartesian-space
trajectory of the end effector)

  * •

$f(t)$ is a weighting function that allows different portions of the
trajectory to receive different weight

  * •

$Z$ is a normalizer ($Z=\sum_{G}\dfrac{\exp(-\mathcal{D}[\xi_{S\rightarrow
Q}]-V_{G}(Q))}{\exp(-V_{G}(S))}$)

  * •

$S$ and $Q$ are the starting and current positions, respectively ($Q=\xi(t)$)

  * •

$G_{R}$ is the robot’s end-effector goal

  * •

$\mathcal{D}$ is a cost functional, defined for our code as the square of the
path length

  * •

$V_{A}(X)$ is defined as the cost of the lowest-cost path from $X$ to $A$
($\texttt{min}_{\xi}\mathcal{D}[\xi]|\xi(0)=X,\xi(end)=A$).

The formulas above are implemented in CoMOTO using a summation to approximate
the integral and piecewise distances between timesteps to approximate path
lengths for $\mathcal{D}$. CoMOTO expressed the legibility cost at a timestep
$t$ as:

$\textsc{LegibilityCost}[\xi,t]=1-P(G_{R}|\xi_{S\rightarrow\xi(t)})$

The CoMOTO formulation for $P(G_{R}|\xi_{S\rightarrow\xi(t)})$ couples the
trajectory waypoints, making optimization more difficult. To enable
optimization of each waypoint independently, we decouple the waypoints by
modifying probability function:

$P(G_{R}|\xi_{S\rightarrow
Q})=\dfrac{1}{Z}\dfrac{\exp(-\hat{\mathcal{D}}(S,Q,t)-||G_{R}-Q||^{2})}{\exp(-||G_{R}-S||^{2})}$

The first change to the function has been to define
$\mathcal{D}(\xi)=||\xi||^{2}$, then substitute $V_{G_{R}}$ for its value
under our definition of $\mathcal{D}$, $V_{G_{R}}(Q)=||G_{R}-Q||^{2}$. Second,
we define $\hat{\mathcal{D}}(S,Q,t)$, an approximation of
$\mathcal{D}(\xi_{S\rightarrow Q})$ parameterized only on $S$, the starting
point, $Q$, the end of the trajectory segment, and $t$, the time (or,
equivalently, knotpoint index) at which the trajectory segment ends
($\xi_{t}=Q$). The motivation for this will be clear momentarily.

Implementing the normalization constant $Z$ similarly to
$P(G_{R}|\xi_{S\rightarrow Q})$:

$Z=\sum_{G}\dfrac{\exp(-\hat{\mathcal{D}}(S,Q,t)-||G-Q||^{2})}{\exp(-||S-G||^{2})}$

We observe that the $\hat{\mathcal{D}}(S,Q,t)$ term is not parameterized on
$G$ and can therefore be factored out of the sum, so the
$\exp(-\hat{\mathcal{D}}(S,Q,t))$ term will cancel with the one in the
definition of $P(G_{R}|\xi_{S\rightarrow\xi(t)})$. Therefore, the legibility
cost can also be written as:

$C_{leg}(x,u,t)=1-\dfrac{1}{Z}\exp(||G_{R}-S||^{2}-||G_{R}-x||^{2})$

###  II-D Nominal Cost

The nominal cost also remains the same as the CoMOTO cost; it is defined at
each waypoint as the cartesian distance between the nominal and actual end-
effector trajectories:

$C_{nom}(x,u,t)=||p_{eef}(t)-p_{eef}^{*}(t)||$

###  II-E Smoothness Cost

The smoothness cost is defined as the squared magnitude of joint-space
velocity at each timestep:

$C_{smooth}(x,u,t)=||u||^{2}$

###  II-F Goal Pose Cost

The goal pose cost is designed to incentivize the MPC solutions to approach
the goal (which it does in conjunction with the nominal cost). It is defined
as the distance between the end effector’s cartsian position at the goal pose
at a given waypoint, plus an orientation error term:

$C_{goal}(x,u,t)=||p_{goal}-p_{eef}||+1-\langle q_{goal},q_{eef}\rangle^{2}$

Where $q_{eef}$ and $q_{goal}$ are unit quaternions giving the orientation of
the end effector at the current time and goal position, respectively.

###  II-G Combined Cost and Problem Setup

The costs presented above are combined in a weighted sum to yield the
optimization objective $C$ from equation 1, where the weights were tuned by
hand to produce subjectively natural trajectories. Nonlinear solvers often
require an initial solution in the feasible region of the problem as a
starting point; our work uses a joint-space linear trajectory towards the
(joint-space) goal as the initial solution.

##  III Experiments and Metrics

To validate our new approach against CoMOTO, we applied each approach to a set
of predicted human trajectories made by the GMM from Luo et al. [9] and
associated covariances in a reaching task, where the robot had to reach across
a human to reach a desired pose. Each trajectory lasted 5 seconds, and the
NMPC was configured with a 1.25 second time horizon and replanned every 0.5
seconds. Both our method and CoMOTO were configured with a timestep of 0.25
seconds. Since we expected the approximations and moving horizon in our
approach to result in slightly suboptimal trajectories, we hypothesized that
our approach would perform slightly worse than CoMOTO, but run significantly
faster.

Our new approach is written in Julia, which is a just-in-time compiled
language. Therefore, to avoid capturing the time to JIT compile our approach
in our metrics, we solved the optimization problem once and discarded the
results before solving it again to measure the wall clock solution time. To
ensure that this didn’t provide our approach with any advantages in terms of
cache availability, we also ran CoMOTO once before testing (CoMOTO is written
in Python and C++ and so doesn’t have a JIT compiler). We evaluate on the
following metrics:

Separation distance (Dst.): the percentage of the trajectory where the
distance between the human and any part of the robot exceeds 20cm.

End effector visibility (Vis.): the percentage of the trajectory where the
robot’s end effector is within the human’s field of view, assuming they’re
looking at the object they’re reaching for.

Legbility (Leg.): average $P(G_{R}|\xi_{S\rightarrow Q})$, as given by Dragan
et al. [3].

Deviation from nominal trajectory (Nom.): the sum of square distances between
corresponding timesteps of the nominal and computed end effector trajectories.

Latency (Lat.): the average time required to obtain a robot trajectory once a
human trajectory has been predicted

The first four of these metrics are the same as those chosen for the CoMOTO
paper - scoring higher is better on all metrics except the last two (latency
and deviation from nominal). Higher deviation from the nominal trajectory
isn’t universally better or worse than lower deviation, but, all else being
equal, we take lower deviation to be better.

##  IV Results

| Ours | CoMOTO
---|---|---
Dst. |  0.83 (0.17) | 0.52 (0.47)
Vis. | 0.53 (0.01) |  0.74 (0.03)
Leg. | 0.26 (0.01) |  0.60 (0.01)
Nom. |  5.80 (0.67) | 9.79 (1.64)
Lat. (sec) |  0.318 (0.027) | 125.73 (27.22)
TABLE I: Performance of our method vs CoMOTO. Best performances are bolded,
and standard deviations are parenthesized. Figure 2: Performance of our method
vs CoMOTO. Higher metrics are better, except on nominal (Nom.)

Our results are summarized in table I and figure 2. As expected, CoMOTO
performs better than our approach on two out of three metrics. However, our
approach not only performs better on the distance metric (which is most
important for human safety), but it is more than 400 times faster than CoMOTO.
Further, our approach has lower standard deviations than CoMOTO, meaning that
it is more reliable. This is an unexpected result, as our approach does not
have access to the entire trajectory prediction ahead of time, and is
therefore capable of making planning errors that CoMOTO would avoid.

Unexpectedly, our method outperforms CoMOTO on the distance metric while
underperforming it on legibility and visibility. We believe that some of this
variation is due to differences in the cost functions used by our method.
Visually, our method produces trajectories similar to CoMOTO’s, but with less
deviation from nominal behavior.

##  V Conclusion

In this paper, we present an MPC-based approach to human-anticipatory
trajectory optimization. We formulate several cost functions which may be
relevant to natural HRC and use our approach to optimize for a weighted
combination of these costs, balancing them against each other. While prior
work required tens of seconds to solve this problem, our approach solves it in
about 300ms, which approaches real-time speeds and enables far broader
application than offline approaches.

In the future, we believe that exploring how humans move in reaction to the
movements of their robot partners may be a useful area of study. Additionally,
new solvers or formulation approaches may further reduce the solution latency,
improve quality, or provide stronger theoretical safety guarantees.

## References

  * Bhardwaj et al. [2022] Mohak Bhardwaj, Balakumar Sundaralingam, Arsalan Mousavian, Nathan D Ratliff, Dieter Fox, Fabio Ramos, and Byron Boots.  Storm: An integrated framework for fast joint-space model-predictive control for reactive manipulation.  In _Conference on Robot Learning_ , pages 750–759. PMLR, 2022.
  * Brito et al. [2020] Bruno Brito, Boaz Floor, Laura Ferranti, and Javier Alonso-Mora.  Model predictive contouring control for collision avoidance in unstructured dynamic environments, 2020.  URL https://arxiv.org/abs/2010.10190.
  * Dragan et al. [2013] Anca D. Dragan, Kenton C. T. Lee, and Siddhartha S. Srinivasa.  Legibility and predictability of robot motion.  _2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , pages 301–308, 2013.
  * Dragan et al. [2015] Anca D. Dragan, Shira Bauman, Jodi Forlizzi, and Siddhartha S. Srinivasa.  Effects of robot motion on human-robot collaboration.  In _2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , pages 51–58, 2015.
  * Faroni et al. [2019] Marco Faroni, Manuel Beschi, and Nicola Pedrocchi.  An mpc framework for online motion planning in human-robot collaborative tasks.  In _2019 24th IEEE International Conference on Emerging Technologies and Factory Automation (ETFA)_ , pages 1555–1558. IEEE, 2019.
  * Howell et al. [2019] Taylor A. Howell, Brian E. Jackson, and Zachary Manchester.  Altro: A fast solver for constrained trajectory optimization.  In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 7674–7679, 2019.  doi: 10.1109/IROS40897.2019.8967788.
  * Jain et al. [2020] Abhinav Jain, Daphne Chen, Dhruva Bansal, Sam Scheele, Mayank Kishore, Hritik Sapra, David Kent, Harish Ravichandar, and Sonia Chernova.  Anticipatory human-robot collaboration via multi-objective trajectory optimization, 2020.  URL https://arxiv.org/abs/2006.03614.
  * Lasota and Shah [2015] Przemyslaw A. Lasota and Julie A. Shah.  Analyzing the effects of human-aware motion planning on close-proximity human–robot collaboration.  _Human Factors_ , 57(1):21–33, 2015.  doi: 10.1177/0018720814565188.  URL https://doi.org/10.1177/0018720814565188.  PMID: 25790568.
  * Luo et al. [2018] Ruikun Luo, Rafi Hayne, and Dmitry Berenson.  Unsupervised early prediction of human reaching for human—robot collaboration in shared workspaces.  _Auton. Robots_ , 42(3):631–648, mar 2018.  ISSN 0929-5593.  doi: 10.1007/s10514-017-9655-8.  URL https://doi.org/10.1007/s10514-017-9655-8.
  * Ma et al. [2022] Tiezheng Ma, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li.  Progressively generating better initial guesses towards next stages for high-quality human motion prediction, 2022.  URL https://arxiv.org/abs/2203.16051.
  * Majumdar and Tedrake [2016] Anirudha Majumdar and Russ Tedrake.  Funnel libraries for real-time robust feedback motion planning, 2016.  URL https://arxiv.org/abs/1601.04037.
  * Mao et al. [2021] Wei Mao, Miaomiao Liu, and Mathieu Salzmann.  Generating smooth pose sequences for diverse human motion prediction, 2021\.  URL https://arxiv.org/abs/2108.08422.
  * Yang et al. [2022] Wei Yang, Balakumar Sundaralingam, Chris Paxton, Iretiayo Akinola, Yu-Wei Chao, Maya Cakmak, and Dieter Fox.  Model predictive control for fluid human-to-robot handovers.  _arXiv preprint arXiv:2204.00134_ , 2022.