# Self-Learning Tuning for Post-Silicon Validation
††thanks: This research was supported by Advantest as part of the Graduate
School ”Intelligent Methods for Test and Reliability” (GS-IMTR) at the
University of Stuttgart.

Peter Domanski, Dirk Pflüger  Institute for Parallel and Distributed Systems
University of Stuttgart
Stuttgart, Germany
{peter.domanski<EMAIL_ADDRESS>Jochen Rivoir, Raphaël
Latty  Applied Research and Venture Team
Advantest Europe GmbH
Böblingen, Germany
{jochen.rivoir<EMAIL_ADDRESS>
###### Abstract

Increasing complexity of modern integrated circuits makes design validation
more difficult. Existing approaches are not able anymore to cope with the
complexity of tasks such as robust performance tuning in post-silicon
validation. Therefore, we propose a novel learn-to-optimize approach based on
reinforcement learning in order to solve complex and mixed-type tuning tasks
in an efficient and robust way.

###### Index Terms:

Post-silicon validation, Robust performance tuning, Learn-to-optimize,
Reinforcement learning

##  I Introduction

Studies suggest that validation is a major bottleneck in semiconductor
integrated circuit (IC) design that occupies up to 70% of the time, efforts,
and resources that are used during the design process [1]. In general, design
validation can be divided into three main stages [2]: pre-silicon validation,
post-silicon validation (PSV), and in-field debugging. In the following, we
focus on PSV. More specifically, we describe the task of robust performance
tuning and our motivation for a novel approach to this task.

Today, PSV is largely considered as an art offering only few systematic
solutions. Moreover, the rising complexity of modern ICs increases the
difficulty of PSV even further. Existing approaches are not able to cope with
the complexity of future systems, especially given a limited time budget.
Therefore, PSV is an exciting research field offering novel opportunities and
challenges [3]. Ensuring functional correctness, checking security properties,
and meeting performance constraints are important tasks in PSV. Besides,
robustness, e.g., against process variations, is crucial. In order to ensure
the best possible performance and robustness, designs rely on tuning to
compensate impacts of process variations and of non-ideal design
implementations. Tuning consists in setting variables and registers, so-called
”tuning knobs” such as bias settings or adjustable currents and voltages,
which may be adjusted as a function of given (operating) conditions, e.g.,
temperature or operating mode of an IC. In addition, the number of tuning
knobs on ICs is rising because affordability is increasing in modern
technologies.

Setting tuning knobs such that device parameters stay within specification
limits and optimize performance goals still remains a complicated, bound-
constrained, mixed-type optimization task. Depending on the different data
types of the variables, a first naive approach is to use either gradient
methods or more flexible derivative-free optimization strategies. These
strategies are important in PSV because there is no analytical expression
describing the true behavior of manufactured ICs (similar to black-box
functions). Thus, we have no access to higher-order moments of the underlying
function. Applying state-of-the-art optimization strategies to solve mixed-
type optimization problems results in approximate, point-wise solutions
because exact solutions are intractable. Given the tight schedule in PSV,
point-wise solutions remain very inefficient. To avoid these pitfalls, we
propose a new self-learning tuning approach that aims to learn a mapping from
conditions to tuning knobs, which we call tuning law. Such a tuning law is
flexible, robust, and efficient as it can be applied either in IC-specific or
IC-independent setups. In the following sections, we show how our novel
approach relates to existing work and describe how to learn a tuning law with
a runtime fast enough even in high-dimensional setups and with the increasing
complexity of modern ICs. Finally, we show preliminary results of our approach
and discuss the experiments and future work.

##  II Related work

Learn-to-optimize Learn-to-optimize is a data-driven approach to learn a model
that aims to replace hand-crafted optimization algorithms (e.g., SGD, RMSprop,
and Adam). Many existing works, e.g., [4, 5, 6, 7, 8, 9] leverage Recurrent
Neural Networks (RNN) as (coordinate-wise) optimizers and rely on gradient
information of the objective function to output parameter updates.
Architectural details of the learned optimizer model (e.g., Long Short Term
Memory (LSTM) networks, and hierarchical RNN [7]), training methods (e.g.,
Truncated Backpropagation Through Time, reinforcement learning [5, 8], and
meta-learning), and input features can differ, but the main focus is on
continuous optimization problems, especially on training of Deep Neural
Networks. Besides learning the full update rule, learn-to-optimize was
customized to automatic hyperparameter tuning (auto-tuning), neural
architecture search, and adversarial attack scenarios.

Hyperparameter tuning A large number of hyperparameter optimization methods
have been proposed to improve the performance of learning algorithms. Many
frameworks, e.g., [10, 11, 12] exist that support most common methods like
Grid Search, Random Search, Bayesian optimization, Gradient-based
optimization, as well as Evolutionary- and Population-based optimization.
These methods differ in efficiency and assumptions with respect to properties
of the objective function. In PSV, methods like Grid- or Random Search are too
inefficient to be used in practice. Likewise, Bayesian- and Gradient-based
optimization are difficult to use due to assumptions like smoothness or
differentiability of the objective function. Such assumptions do not
necessarily hold in PSV, neither are properties of the objective function
known beforehand.

##  III Methodology

In order to learn a tuning law, we are following the path of learn-to-optimize
and leverage reinforcement learning methods as in [5, 8] to train an
optimization algorithm. In contrast to hand-crafted methods, this data-driven
approach automatizes the design of efficient optimization methods by observing
their own execution performance. Whereas most investigations focus on
continuous optimization, we make use of reinforcement learning in the training
procedure to enable learning of optimization methods for mixed-type problems.
Similar to [5], the objective function is a black box. In our application, the
objective is solely described by the given data set. Instead of optimizing
network parameters, we aim to find inputs that produce the best outputs (in
terms of device performance) without relying on gradients or approximations of
gradients. This resembles the scenario of black-box adversarial attacks, e.g.
in [13]. Fig. 1 shows an overview of the proposed approach. As illustrated, we
iteratively train an optimization algorithm on a surrogate model in the outer
reinforcement learning loop. The learned optimizer itself (iteratively)
improves the performance measure of the surrogate in the inner loop. The
performance improvement is used as feedback to the outer training loop. The
surrogate model allows us to formulate tuning as an iterative optimization
process and train the tuning law efficiently within the reinforcement learning
loop.

The general reinforcement learning setting of learn-to-optimize is shown in
Fig. 2. In this setting, an agent chooses an action, e.g. optimization
parameter values $\vec{x}_{t}$ at each iteration $t$, which changes the
environment, e.g., the objective function value $f(\vec{x}_{t})$. The agent
receives feedback, typically in the form of a reward, based on the
consequences of the action. The goal of the agent is to choose a sequence of
actions based on observations of the environment’s state in a way that the
agent maximizes the cumulative reward over all iterations [9]. The final goal
is to find $\vec{x}^{*}=\arg\max f(\vec{x})$. The reward definition we use
reflects these optimization goals. The computations of our methodology can be
summarized by the following steps:

  * 1.

Given current state
$\vec{s}_{t,\text{NN}}:=\\{(\vec{x}_{t-1},f(\vec{x}_{t-1}))\\}$ or
$\vec{s}_{t,\text{RNN}}:=\\{(\vec{x}_{t-1},f(\vec{x}_{t-1}),\vec{h}_{t-1})\\}$
propose update $\vec{x}_{t}$
($\vec{h}_{t-1}$: internal state of RNN at time step $t-1$)

  * 2.

Observe response of environment, e.g., $f(\vec{x}_{t})$

  * 3.

Update internal statistics to produce $\vec{s}_{t+1}$ and $r_{t+1}$

The update rule (e.g., agent) in 1. is defined by a (Recurrent) Neural Network
parameterized by $\theta$, such as
$f_{\theta,\text{(R)NN}}:=\vec{s}_{t}\rightarrow\vec{x}_{t}$. The architecture
consists of a two-layer LSTM network with tanh activation units. The
parameters $\theta$ are updated by applying reinforcement learning algorithms,
e.g., REINFORCE to maximize cumulative reward.

Given a similar computing budget, our approach can have superior performance
showing much faster convergence speed compared to classical methods, in
particular for difficult (e.g. non-convex) optimization problems. Thus,
learning to optimize has the potential to overcome the limits of the actual
analytical methods in PSV. Another benefit is that the solutions of
reinforcement learning can be directly used as a tuning law.

Figure 1: Proposed approach to self-learning tuning in PSV. The tuning law for
the surrogate objective function (yellow) is learned iteratively within the
reinforcement learning loop (orange). In this application, we use a
performance measure (aggregated from measurement outputs) of the device to
train and evaluate a (device-specific) tuning algorithm in form of a
(Recurrent) Neural Network (blue) on the surrogate model. After training, the
tuning algorithm runs on real-world devices. Note that the complexity of
surrogate models for the objective function could be heavily reduced by using
variable selection methods (e.g., [14] a recent work from the GS-IMTR) due to
high dimensional input data in PSV. Figure 2: Learn-to-optimize approach. The
visualization shows key components (agent, environment, state- and reward
definitions) in the reinforcement learning loop (orange).

##  IV Experiments

In this section, we provide preliminary results of our methodology. In
particular, we use a real-world data set provided by Advantest that consists
of data from multiple devices. The objective function $f$ we aim to optimize
is an aggregation of device-specific models, e.g., Neural Networks that
describe the input-output behavior of each device. The aggregation is designed
to cover the behavior across devices. The parameter $\vec{x}$ we optimize are
integer values, namely the tuning knobs of the devices. The goal is to find
the best value $\vec{x}^{*}$ which corresponds to the best possible
configuration of the tuning knobs. In this preliminary experimental setup, we
have not specified any tuning conditions. Thus, the tuning law we aim to learn
is a constant function. However, our methodology can be extended to support
arbitrary data types and condition-dependent tuning laws.

Fig. 3 shows the objective function value $f(\vec{x}^{*})$ of the described
experimental setup after training. We compare our approach (L2O) with a state-
of-the-art optimization algorithm called Powell’s method. In previous
experiments, this method has proven to be the best in our experiments compared
to other applicable (hyperparameter tuning) algorithms, e.g., Evolutionary-
and Population-based approaches. Here, we compare two versions of Powell’s
method, a version with default hyperparameters and another with a comparable
computational budget to our approach. As a baseline, we use the Tree-
structured Parzen Estimator (TPE) algorithm [15] with a median pruner (early
stopping of unpromising trials) as implemented in Optuna [10]. Similar to both
versions of Powell’s method, we set the number of maximum trials for TPE such
that computational budgets are comparable.

For both versions of Powell’s method, we observe a constant variability in the
objective function value caused by different initial values. Moreover, the
average performance does not improve with time. The TPE baseline looks similar
but shows a much larger variability in the objective function value. In
contrast, our approach benefits from additional effort during training and
improves over time, see Fig. 4. Thus, our methodology results in a very stable
tuning law almost independent of initial parameter values and reaches a high
performance. Tab. I shows the improvements in computation time.

##  V Conclusion

To summarize, the learn-to-optimize approach has appealing properties and
opportunities that match the characteristics of the tuning task in PSV very
well, e.g., the high-dimensional mixed-type problems. First solutions
outperform state-of-the-art, point-wise optimization results in final
performance, efficiency, and convergence speed. Finally, our approach allows
us to learn the tuning on actual devices by taking advantage of the data
related to the objective function of interest. There is no need to introduce
new a priori assumptions regarding the objective nor additional
hyperparameters of the resulting optimization algorithm. In the future, we aim
to study more complicated objective functions (e.g., min-max) and to learn
tuning laws that depend on given conditions. The latter means that we aim to
optimize tuning parameters as a function of conditions, such as currents,
voltages or, the operating mode of devices. With state-of-the-art optimization
methods, this gets infeasible as we have to re-run the entire optimization
procedure for all possible values of the conditions, which results in very
high time and memory consumption.

Figure 3: Box-plot visualization of the distributions of achievable objective
function values. We compare Powell’s method with default hyperparameters,
Powell’s method with a comparable computational budget, TPE (Optuna), and
learn-to-optimize (our method). For each evaluation of the objective function
values, we use 16 random initial parameter values. Figure 4: Performance
(objective value) of the agent (RNN that we use as tuning law) over the
reinforcement learning process. Training the agent for 2000 steps with the
learn-to-optimize approach took 1.5 hours on our hardware. For each
evaluation, we use 16 random initial parameter values.

| Computation time (avg.)
---|---
L2O | $\mathbf{1.118\,\textbf{s}}$
TPE (Optuna) | $108.237\,\text{s}$
Powell (default) | $129.684\,\text{s}$
Powell (approx.) | $39.196\,\text{s}$

TABLE I: Time to optimize (without training time).

## References

  * [1] P. Mishra, R. Morad _et al._ , “Post-silicon validation in the soc era: A tutorial introduction,” _IEEE Design & Test_, vol. 34, no. 3, pp. 68–92, 2017.
  * [2] P. Mishra and F. Farahmandi, _Post-Silicon Validation and Debug_. Springer, 2019.
  * [3] S. Mitra, S. A. Seshia, and N. Nicolici, “Post-silicon validation opportunities, challenges and recent advances,” in _Design Automation Conference_. IEEE, 2010, pp. 12–17.
  * [4] M. Andrychowicz, M. Denil _et al._ , “Learning to learn by gradient descent by gradient descent,” in _Advances in neural information processing systems_ , 2016, pp. 3981–3989.
  * [5] Y. Chen, M. W. Hoffman _et al._ , “Learning to learn without gradient descent by gradient descent,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 748–756.
  * [6] K. Lv, S. Jiang, and J. Li, “Learning gradient descent: Better generalization and longer horizons,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 2247–2255.
  * [7] O. Wichrowska, N. Maheswaranathan _et al._ , “Learned optimizers that scale and generalize,” in _International Conference on Machine Learning_. PMLR, 2017, pp. 3751–3760.
  * [8] B. Zoph and Q. V. Le, “Neural architecture search with reinforcement learning,” _arXiv preprint arXiv:1611.01578_ , 2016.
  * [9] K. Li and J. Malik, “Learning to optimize,” _arXiv preprint arXiv:1606.01885_ , 2016.
  * [10] T. Akiba, S. Sano _et al._ , “Optuna: A next-generation hyperparameter optimization framework,” in _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, 2019, pp. 2623–2631.
  * [11] R. Liaw, E. Liang _et al._ , “Tune: A research platform for distributed model selection and training,” _arXiv preprint arXiv:1807.05118_ , 2018.
  * [12] J. Bergstra, D. Yamins _et al._ , “Hyperopt: A python library for optimizing the hyperparameters of machine learning algorithms,” in _Proceedings of the 12th Python in science conference_ , vol. 13. Citeseer, 2013, p. 20.
  * [13] C. Guo, J. Gardner _et al._ , “Simple black-box adversarial attacks,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 2484–2493.
  * [14] Y. Liao, R. Latty, and B. Yang, “Feature selection using batch-wise attenuation and feature mask normalization,” in _2021 International Joint Conference on Neural Networks (IJCNN)_ , 2021, pp. 1–9.
  * [15] J. Bergstra, R. Bardenet _et al._ , “Algorithms for hyper-parameter optimization,” _Advances in neural information processing systems_ , vol. 24, 2011.


###### Abstract

Tropical Newton-Puiseux polynomials defined as piece-wise linear functions
with rational coefficients at the variables, play a role of tropical algebraic
functions. We provide explicit formulas for tropical Newton-Puiseux
polynomials being the tropical zeroes of a univariate tropical polynomial with
parametric coefficients.

Tropical Newton-Puiseux polynomials II

Dima Grigoriev

CNRS, Mathématique, Université de Lille, Villeneuve d’Ascq, 59655, France

e-mail<EMAIL_ADDRESS>
URL: http://en.wikipedia.org/wiki/Dima_Grigoriev

keywords: tropical Newton-Puiseux polynomial, zeroes of tropical parametric
polynomials

AMS classification: 14T05

## Introduction

One can find basic concepts of tropical algebra in [2].

Given a tropical univariate polynomial

$\displaystyle f=\min_{0\leq k\leq n}\\{x_{k}+kY\\}$ (1)

its tropical zero $y\in{\mathbb{R}}$ is such that the minimum in (1) is
attained for at least two different values of $0\leq k\leq n$. In this paper
we treat the coefficients $X=(x_{0},\dots,x_{n})$ as parameters and find the
zeroes of $f$ as a function in $x_{0},\dots,x_{n}$.

We show that $f$ has exactly $n$ such parametric zeroes $g_{1},\dots,g_{n}$.
Each $g_{k},\,1\leq k\leq n$ is a tropical Newton-Puiseux polynomial, i. e. a
piece-wise linear function with rational coefficients at the variables. One
can represent any tropical Newton-Puiseux polynomial in the form

$\min_{I}\\{a_{I}+(I,\,X)\\}-\min_{J}\\{b_{J}+(J,\,X)\\}$

of a difference (so, the tropical quotient) of two concave piece-wise linear
functions, where $I,\,J\in{\mathbb{Q}}^{n+1};\,a_{I},\,b_{J}\in{\mathbb{R}}$,
and $(I,\,X)$ denotes the inner product (cf. [3]).

Tropical Newton-Puiseux polynomials play a role of algebraic functions in the
tropical setting. Similar to Newton-Puiseux series in the classical algebra,
$g_{k}(x_{0},\dots,x_{n})$ provides a tropical zero of $f$ for any point
$(x_{0},\dots,x_{n})\in{\mathbb{R}}^{n+1}$. We note that in the classical
algebra one considers Newton-Puiseux series just in a single variable, while
an advantage of the tropical algebra is that one considers tropical Newton-
Puiseux polynomials in several variables.

Observe that if one considers a univariate tropical polynomial with its
coefficients being tropical Newton-Puiseux polynomials then in its turn, the
tropical zeroes of this tropical polynomial are again tropical Newton-Puiseux
polynomials. Thus, one can view the semi-field of tropical Newton-Puiseux
polynomials as a tropical algebraic closure of the semi-ring of tropical
polynomials.

##  1 Tropical Newton-Puiseux polynomials as tropical zeroes

We say that a tropical Newton-Puiseux polynomial $g:=g(X_{0},\dots,X_{n})$ is
a (tropical) zero of $f$ (1) if for any
$(x_{0},\dots,x_{n})\in{\mathbb{R}}^{n+1}$ the value $y=g(x_{0},\dots,x_{n})$
is a tropical zero of the tropical polynomial $f$.

First, we describe the tropical Newton-Puiseux zeroes of $f$ geometrically and
show that there are exactly $n$ of them. In the next section we provide for
them the explicit formulas.

For a point $x:=(x_{0},\dots,x_{n})\in{\mathbb{R}}^{n+1}$ its Newton polygon
$N_{x}\subset{\mathbb{R}}^{2}$ is the convex hull of the vertical rays
$\\{(k,\,c)\,:\,c\geq x_{k}\\},\,0\leq k\leq n$. Note that the slopes of the
edges of $N_{x}$ are just the tropical zeroes of $f$.

For a subset $S\subset\\{1,\dots,n-1\\}$ consider a convex polyhedron
$P_{S}\subset{\mathbb{R}}^{n+1}$ (of dimension $n+1$) consisting of points
$x=(x_{0},\dots,x_{n})$ such that its Newton polygon $N_{x}$ has the vertices
$(0,\,x_{0}),\,(n,\,x_{n}),\,\\{(s,\,x_{s})\,:\,s\in S\\}$. Thus,
$\\{P_{S}\,:\,S\subset\\{1,\dots,n-1\\}\\}$ constitute a partition of
${\mathbb{R}}^{n+1}$ into $2^{n-1}$ polyhedra.

Take the (open) polyhedron $P:=P_{\\{1,\dots,n-1\\}}$ consisting of points $x$
such that the Newton polygon $N_{x}$ has $n+1$ vertices. Then there are
exactly $n$ continuous piece-wise linear functions $g_{1},\dots,g_{n}$ on $P$
being tropical zeroes of $f$ (1). Namely,
$g_{k}(x_{0},\dots,x_{n})=x_{k-1}-x_{k},\,1\leq k\leq n$.

Observe that each $g_{k},\,1\leq k\leq n$ has a unique (continuous)
continuation on every polyhedron $P_{S}$. Namely, take the unique pair $0\leq
i\leq k-1,\,k\leq j\leq n$ such that $i,\,j\in S\cup\\{0,\,n\\}$, and there
are no $s\in S\cup\\{0,\,n\\}$ satisfying inequalities $i<s<j$.

###### Lemma 1.1

The unique continuation of $g_{k}$ on $P_{S}$ coincides with
$\frac{x_{i}-x_{j}}{j-i}$.

Proof. For any point $(x_{0},\dots,x_{n})$ which belongs to both boundaries of
$P$ and of $P_{S}$ holds $x_{s}-x_{s+1}=x_{k-1}-x_{k},\,i\leq s<j$, hence
$x_{k-1}-x_{k}=\frac{x_{i}-x_{j}}{j-i}$. $\Box$

Note that $\frac{x_{i}-x_{j}}{j-i}$ is the slope of the edge with the end-
points $(i,\,x_{i}),\,(j,\,x_{j})$. Thus, we have shown that there are exactly
$n$ tropical Newton-Puiseux polynomials on ${\mathbb{R}}^{n+1}$ being tropical
zeroes of $f$ (1).

##  2 Explicit formulas for tropical zeroes

###### Theorem 2.1

A tropical polynomial $f=\min_{0\leq k\leq n}\\{x_{k}+kY\\}$ with parametric
coefficients $(x_{0},\dots,x_{n})$ has exactly $n$ tropical zeroes
$g_{1},\dots,g_{n}$ being tropical Newton-Puiseux polynomials in
$(x_{0},\dots,x_{n})$. For each $0\leq k\leq n$ one can represent $g_{k}$ as
follows. For every $0\leq p<k$ consider a tropical Newton-Puiseux polynomial

$t_{p}:=\max_{k\leq q\leq n}\left\\{\frac{x_{p}-x_{q}}{q-p}\right\\}.$

Then $g_{k}=\min_{0\leq p<k}\\{t_{p}\\}$.

Proof. Fix for the time being a polyhedron $P_{S}$ and follow the notations
from Lemma 1.1. For any point $x:=(x_{0},\dots,x_{n})\in P_{S}$ its Newton
polygon $N_{x}$ has an edge with the end-points $(i,\,x_{i}),\,(j,\,x_{j})$.
Therefore, for every $0\leq p<k$ the following inequality for the slopes
holds:

$\frac{x_{p}-x_{j}}{j-p}\geq\frac{x_{i}-x_{j}}{j-i}.$

Hence $t_{p}\geq\frac{x_{i}-x_{j}}{j-i}$.

On the other hand, $t_{i}=\frac{x_{i}-x_{j}}{j-i}$ since for every $k\leq
q\leq n$ the following inequality for the slopes holds:

$\frac{x_{i}-x_{q}}{q-i}\leq\frac{x_{i}-x_{j}}{j-i}.$

Thus, $g_{k}$ coincides with $\frac{x_{i}-x_{j}}{j-i}$ on $P_{S}$ which
completes the proof due to Lemma 1.1. $\Box$

###### Remark 2.2

In the formula for $g_{k}$ in the theorem a tropical Newton-Puiseux polynomial
$t_{p}$ is involved in which a left-end point $(p,\,x_{p})$ of the intervals
is fixed. In a dual way one can define $r_{q}:=\min_{0\leq
p<k}\\{\frac{x_{p}-x_{q}}{q-p}\\}$ by fixing a right end-point $(q,\,x_{q})$.
Then, similarly to the theorem we get $g_{k}=\max_{k\leq q\leq n}\\{r_{q}\\}$.

Acknowledgements. The author is grateful to the grant RSF 21-11-00283 and to
MCCME for inspiring atmosphere.

##  References

  * [1] D. Grigoriev.  Tropical Newton-Puiseux polynomials.  Lect. Notes Comput. Sci., 11077:177–186, 2018.
  * [2] D. Maclagan and B. Sturmfels.  Introduction to Tropical Geometry:, volume 161 of Graduate Studies in Mathematics.  American Mathematical Society, 2015.
  * [3] D. Ovchinnikov.  Max-min representation of piecewise linear functions.  Beiträge Algebra Geom., 43:297–302, 2002.


# ELSA: Evaluating Localization of Social Activities
in Urban Streets

Maryam Hosseini1∗ Marco Cipriano2∗ Sedigheh Eslami2 Daniel Hodczak3
Liu Liu1 Andres Sevtsuk1 Gerard de Melo2
1Massachusetts Institute of Technology (MIT)
2 Hasso Plattner Institute (HPI)
3University of Illinios Chicago (UIC)
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Why do some streets attract more social activities than others? Is it due to
street design, or do land use patterns in neighborhoods create opportunities
for businesses where people gather? These questions have intrigued urban
sociologists, designers, and planners for decades. Yet, most research in this
area has remained limited in scale, lacking a comprehensive perspective on the
various factors influencing social interactions in urban settings. Exploring
these issues requires fine-level data on the frequency and variety of social
interactions on urban street. Recent advances in computer vision and the
emergence of the open-vocabulary detection models offer a unique opportunity
to address this long-standing issue on a scale that was previously impossible
using traditional observational methods. In this paper, we propose a new
benchmark dataset for Evaluating Localization of Social Activities (ELSA) in
urban street images. ELSA draws on theoretical frameworks in urban sociology
and design. While majority of action recognition datasets are collected in
controlled settings, we use in-the-wild street-level imagery, where the size
of social groups and the types of activities can vary significantly. ELSA
includes 937 manually annotated images with more than 4,300 multi-labeled
bounding boxes for individual and group activities, categorized into three
primary groups: Condition, State, and Action. Each category contains various
sub-categories, e.g., alone or group under Condition category, standing or
walking, which fall under the State category, and talking or dining with
regards to the Action category. ELSA is publicly available for the research
community111https://github.com/urban-submissions/elsa.

**footnotetext: Equal contribution.

##  1 Introduction

For it is interaction, not place, that is the essence of the city and of city
life.

(Melvin M. Webber, 1964, 147)

In recent years, increased focus on the human scale of the cities has drawn
more attention to public spaces and pedestrian facilities. Beyond their
physical form and artifacts, cities are recognized as social institutions
[34], often known by their lively streets and plazas where people can engage
in social interactions. For decades, urban scholars from various fields have
been fascinated by the complex interplay between public spaces and the social
interactions they support [31, 39, 16]. However, traditional scientific
inquiry into the distribution of social activities across urban streets has
been hampered by high data collection costs and extensive time requirements.

The emergence of advanced computer vision techniques such as object detection
and semantic segmentation together with the availability of public sources of
street-level imagery have opened new avenues for conducting comprehensive
observational studies at reduced cost and increased scale. Activity
recognition techniques are mostly designed to work with videos [22], since, by
nature, human activity involves motion and sequence of actions. However,
acquiring continuous video footage across an entire city over time involves
substantial data storage requirements and high processing costs, making it
very difficult to scale. Object detection on still images emerges as a low-
cost, efficient and applicable method, as it allows for the identification and
localization of complex social interactions in diverse settings where the
environmental context significantly influences the range of possible social
interactions and where each image can contain a large number of people engaged
in diverse activities.

While conventional object detection models are trained in closed-vocabulary
settings and rely heavily on predefined classes [20], open-vocabulary
detection (OVD) models aim to transcend traditional object detection models,
and utilize the abundance of language data in order to enable the detection of
classes with less representation in standard benchmark training data. A robust
OVD model is expected to handle a wide range of input terms and phrases that
were not explicitly part of its training set. This is crucial for models
deployed in real-world settings, such as urban streets, where unpredictable
and varied interactions are common. The absence of benchmark data for open-
vocabulary detection of social and individual actions in still-images ‘in the
wild’ hinders the development of robust models that can generalize well across
diverse and spontaneous urban scenarios, where the context and variability of
human activities are far greater than those typically encountered in
controlled environments. Furthermore, OVDs pose new challenges in both
localization and semantic understanding of unseen new categories. They often
struggle with semantic consistency across diverse inputs, demonstrate
sensitivity to slight variations in input phrasing, and the calibration of
their predictive confidence, especially in out-of-distribution scenarios,
remains suboptimal, resulting in overconfident predictions that do not
accurately reflect their actual accuracy [35, 8].

In response to these challenges, we propose ELSA, a new benchmark dataset for
Evaluating Localization of Social Activities in Urban Streets, using closed as
well as open-vocabulary object detection models in recognizing and localizing
human activity in urban streets from still images. We employ a multi-labeling
scheme and define 34 unique individual labels regarding human activities.
These labels can concurrently be associated to each annotated bounding box. As
a result, ELSA includes more than 4,300 bounding boxes annotated with 112
unique combination of human activities for 937 street view images. In order to
evaluate the robustness of object detection models, ELSA contains challenging
scenes with humans located relatively far from the camera position as well as
scenes containing pictures of people, which are likely to get falsely
recognized as genuine people by such models. To the best of our knowledge,
ELSA is the first dataset enabling evaluation of computer vision models for
the task of social activity interaction localization in urban street images.

##  2 Related Work

Social Interactions in Public Spaces. Vibrant streets rich in interpersonal
exchange have fascinated urban scholars because of their social qualities as
well as fundamental indicators of sustainable urban environments [31]. William
Whyte [39] along with Jacobs [16] highlight the intrinsic value of public
spaces in fostering vibrant social life, not only with respect to their social
qualities, but also with regards to how they constitute fundamental indicators
of sustainable urban environments [31]. The seminal work of William Whyte [39]
who conducted detailed observational studies on video footage taken from New
York City’s public spaces along with Jacobs [16] analysis of neighborhood
vibrancy and _sidewalk ballet_ , highlighted the intrinsic value public spaces
in fostering vibrant social life. Jan Gehl [10] describes activities in the
public spaces as a spectrum between optional activities, e.g., talking with
friends, and necessary activities, e.g., walking to work. The public space
observational method [11] delineates between active social group activities,
e.g., dining or talking together, and passive activities, such as strangers
sitting on a bench checking their cell-phones. Inspired by this research, we
define the target set of social activities in ELSA.

Object Detection. One of the primary goals of object detection in computer
vision is the fine-grained understanding of visual scenes by localizing the
objects in the scene [27]. Traditionally, researchers aimed at closed-
vocabulary object detection, where the models were trained with a pre-defined
closed set of objects [27, 23, 28, 24]. In contrast, with open-vocabulary
object detection (OVD), which was first introduced by Zareian et al. [42],
primarily tackles the limitation of traditional object detection models that
rely on pre-defined closed set of objects [4, 33, 20] tested on various OVD
benchmark datasets [35, 40]. At their core, vision–language contrastive loss
is often used for aligning semantics and concepts in the two modalities [19,
30, 6, 17, 32, 25] with additional soft token prediction in MDETR [19]. Using
a dual-encoder-single-decoder architecture, Grounding DINO [30] extends DINO
[43] such that given a text prompt, query selection is performed to select the
text features more relevant for the cross-modal decoder. A contrastive loss
for aligning the output of the decoder and text queries along with a
regression L1 loss and generalized union over intersection is optimized end-
to-end for the detection. Activity Localization Datasets. Activity
localization involves analyzing the activities in a sequence of images [2, 3,
9, 44, 44]. A seminal study by Choi et al. [7] focuses on in-the-wild
pedestrian action classification from videos. Recent methods [44, 38]combine
appearance and pose data with transformers in order to enhance interaction
recognition and improve the detection of complex human behaviors. Li et al.
[26] added cognitive depth with the HAKE engine, which uses logical reasoning
to analyze human-object interactions. However, all of these models are tested
on video datasets such as a volleyball dataset [15], AVA-Interaction [38],
HICO-DET [5], V-COCO [13], NTU RGB+D [36, 29] and SBU-Kinect-Interaction [41].
Among previous work, Ehsanpour et al. [9] includes annotated videos of
university campus scenes for group-based social activities and enables group-
based social activity recognition. In contrast, ELSA aims at localization of
social activities in still images, which is a more challenging problem. In
image sequences, activities can be recognized based on the object movements
across consecutive images. In contrary, for still images, localization models
need to infer activities from the snapshot of the moment shown in the image.

##  3 ELSA

Figure 1: Examples of annotations and label space in our social interaction
study.

Motivated by the lack of available benchmark data for detection of social
interactions and individual activities in still images, we propose ELSA. The
goal is to enable the evaluation of state-of-the-art object detection models
in detecting various levels and patterns of human activity and interactions.
In this section, we provide detailed description of ELSA and its unique
characteristics.

Image Resources. We chose New York City as our site, due to its famous lively
streets and public spaces. We collected street-level images from two different
sources: Microsoft Bing Side-view [21] and Google Street View [12, 1]. The
Bing imagery provides time-stamps, making it possible to choose days and times
where the probability of having more pedestrians on the streets are higher.

Image Pre-processings. To focus mainly on the events happening on sidewalks,
we employed the CitySurfaces [14] semantic segmentation model to segment
pedestrian-dedicated areas.

The segmented regions were then used to crop the street-level images to only
include the immediate areas around the sidewalks, We removed other irrelevant
information in order to ensure our dataset was finely tuned to the pedestrian
scale. An example of this pre-processing is shown in Figure 2.

Figure 2: Post-processing step: cropping images leveraging semantic
segmentation models to focus on sidewalk-level activities.

Target Labels.  We draw on the active design and urban vibrancy literature
(see Section 2) to select our primary individual labels. ELSA exhibits non-
disjoint label spaces, where multiple concurrent labels can be applied to the
same object in a multi-labeling scheme that encompasses the nuances of human
behavior and context. Labels are grouped into four categories:

  1. 1.

Condition: defines the social configuration of the subjects, _alone_ , _two
people_ , and _group_. These labels are disjoint, denoting mutually exclusive
social settings. The _Condition_ category is foundational for social
interaction detection, as it establishes the primary context for potential
interactions, highlighting solo activities, limited interactions, or group
dynamics.

  2. 2.

State: captures the physical disposition or activity mode of the subjects,
such as _walking_ or _sitting_. While disjoint for individuals, these labels
can co-occur in couple or group scenarios. The state is of great importance
for public space social analysis since _standing_ or _sitting_ may indicate
potential for stationary social engagement, while _walking_ or _biking_ could
imply transient or passing interactions.

  3. 3.

Action: reflects specific behaviors or activities, such as _dining_ ,
_talking_ and _phone interaction_. Action labels are inherently non-disjoint,
capturing the complexity of human behavior where multiple actions can co-
occur, such as _talking_ while _pushing a stroller_.

  4. 4.

Other: represents characteristics of the scene that do not fall under the
previous categories and are, yet, important for understanding the features of
the urban area. As an example, the label _kid_ can represent a family-friendly
area.

In the realm of social interaction recognition, the labels under the Action
category are instrumental in identifying engagement patterns and interaction
types, distinguishing, for example, between conversational engagement and co-
active behavior.

Figure 3: Example of rules of capture in annotation. a) two people sitting and
the stairs are not captures as an annotation. b) two people crossing a
crosswalk and one pushing a wheelchair. The wheelchair is captured in the
annotation.

Our primary objective is to evaluate the performance of object detection in
closed as well as open-vocabulary settings. ELSA does not incorporate any
direct closed-vocabulary labels. However, the main target, “person," is the
most representative label in the COCO dataset. Additionally, we identified a
series of prompts that have common COCO objects, such as strollers, bicycles,
benches, and chairs (used in the natural language prompts with “sitting,"
e.g., “sitting on a bench or chair"), dogs (used as synonyms with “pet"), and
cell phones. We account for this property in our evaluation and expect prompts
including these specific terms to perform better due to contextual cues.

Annotation Process. We customized the open source Label Studio tool [37] for
annotation and integrated YOLOv8 [18] for pre-detecting the initial objects. A
group of four people manually corrected the initial bounding boxes and
annotated the label combinations. Finally, an urban planning expert reviewed
the label and bounding box accuracy for all annotations.

Examples of ELSA’s annotations are depicted in Figure 1 and Figure 3.

As shown in Figure 3, for activities that are described with another non-
stationary object, e.g., _pushing a wheelchair_ or _biking_ , the annotated
ground truth bounding box includes the object as well as the person performing
the action(see Figure 3-a), wheras for actions without an object that is
actively a part of the action, the annotated bounding box merely captures the
person, (see Figure 3-b _sitting_).

Annotation Cleaning. After the initial annotations, we performed sanity checks
on the disjoint labels and defined a set of sanity rules, e.g., a bounding box
with just one person cannot have two contradictory states of sitting and
walking at the same time. The full list of these sanity rules are provided in
Section A. We applied the sanity rules to all the annotated bounding boxes and
re-annotated the ones that did not pass the sanity checks. We repeated this
process until all bounding boxes passed our defined sanity rules.

Dataset Statistics. ELSA includes 937 images with more than 4.3K annotated
bounding boxes for social and individual activities. In total, there exist 34
distinct single labels in ELSA. Since we have a multi-labeling scheme, each
bounding box can have 2 or more of the distinct 34 labels associated with it,
correspondingly. As a result, ELSA includes 112 unique combinations of human
activities. Figure 4 shows the distribution of the distinct labels as well as
the distribution of the multiple label combinations in ELSA.

Figure 4: Overview of the distribution of activities in ELSA. (A) Number of
bounding boxes per single activity label, (B) Distribution of the top 20
activity labels that occur together. Figure 5: Example of challenging
scenarios: a) Printed image of people that are not to be recognized as
genuine; b) Crowded scene with people standing at different distances from
camera; c) Prompts at two levels _cs_ and _csa_ for one target in the image.

Challenging Scenarios. ELSA includes still images from ‘in the wild’
scenarios, which examines the robustness and generalizability of the state-of-
the-art models across diverse and spontaneous urban scenarios where the
context and variability of human activities are far greater than those
typically encountered in controlled environments. Thus, ELSA poses two types
of challenges for activity recognition and localization models:

  1. 1.

Challenges in the visual data: ELSA include negative sets in scenes without
actual pedestrians but with printed images of people on billboards, buses, or
walls (see Figure 5-a), as well as mannequins in store fronts. There are
instances of people standing far away from the camera, making them difficult
to detect. We also have crowded scenes with obstructions, where detecting all
targets can be challenging (see Figure 5-b).

  2. 2.

Prompt level challenges: We employ a three-level benchmark to increase the
complexity of the query prompts at each level. Each level’s queries are
designed to return all instances of the target label combination that includes
these sub-category labels. Label combinations in ELSA follow one of the
following patterns: “Condition + State” (CS, e.g., “group standing”), or
“Condition + State + Activity” (CSA, e.g., “group standing talking and taking
photo”) or “Condition + State + Activity + Other” (e.g., “group standing
talking and taking photo with coffer or drink in hand”) (see Figure 5-c for a
two-level prompt example).

##  4 Conclusion

This paper introduces ELSA, a novel dataset specifically curated for the
detection of social activities from still images within urban environments.
Employing a multi-labeling scheme, ELSA comprises 937 annotated images, more
than 4,300 bounding boxes, which are annotated with 112 unique combinations of
social activities. With the goal of evaluating the robustness of the state-of-
the-art object detection models, ELSA contains challenging scenes with humans
located relatively far from the camera position as well as scenes containing
pictures of people, which are likely to get falsely recognized as genuine
people by such models. Moreover, in contrast to previous work that targets
video data, ELSA aims at localization of social activities in still images,
which is a more challenging problem. To the best of our knowledge, ELSA stands
as the pioneering dataset grounded in the theoretical foundations of urban
sociology and design. We hope that ELSA enables researchers to study
benchmarking, evaluating as well as enhancing the performance of state-of-the-
art models in detecting human interactions within public spaces.

##  5 Acknowledgments

We acknowledge the financial support from the HPI-MIT Designing for
Sustainability program as well as from the German Federal Ministry for
Education and Research (BMBF) within the project KI-Servicezentrum Berlin
Brandenburg (01IS22092).

## References

  * Anguelov et al. [2010] D. Anguelov, C. Dulong, D. Filip, C. Frueh, S. Lafon, R. Lyon, A. Ogale, L. Vincent, and J. Weaver.  Google street view: Capturing the world at street level.  _Computer_ , 43(6):32–38, 2010.
  * Bagautdinov et al. [2017] T. Bagautdinov, A. Alahi, F. Fleuret, P. Fua, and S. Savarese.  Social scene understanding: End-to-end multi-person action localization and collective activity recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * Barekatain et al. [2017] M. Barekatain, M. Martí, H.-F. Shih, S. Murray, K. Nakayama, Y. Matsuo, and H. Prendinger.  Okutama-action: An aerial view video dataset for concurrent human action detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pages 28–35, 2017.
  * Bravo et al. [2023] M. A. Bravo, S. Mittal, S. Ging, and T. Brox.  Open-vocabulary attribute detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7041–7050, 2023.
  * Chao et al. [2018] Y.-W. Chao, Y. Liu, X. Liu, H. Zeng, and J. Deng.  Learning to detect human-object interactions.  In _2018 ieee winter conference on applications of computer vision (wacv)_ , pages 381–389. IEEE, 2018.
  * Cheng et al. [2024] T. Cheng, L. Song, Y. Ge, W. Liu, X. Wang, and Y. Shan.  Yolo-world: Real-time open-vocabulary object detection.  _arXiv preprint arXiv:2401.17270_ , 2024.
  * Choi et al. [2009] W. Choi, K. Shahid, and S. Savarese.  What are they doing?: Collective activity classification using spatio-temporal relationship among people.  In _2009 IEEE 12th international conference on computer vision workshops, ICCV Workshops_ , pages 1282–1289. IEEE, 2009.
  * Dave et al. [2021] A. Dave, P. Dollár, D. Ramanan, A. Kirillov, and R. Girshick.  Evaluating large-vocabulary object detectors: The devil is in the details.  _arXiv preprint arXiv:2102.01066_ , 2021.
  * Ehsanpour et al. [2022] M. Ehsanpour, F. Saleh, S. Savarese, I. Reid, and H. Rezatofighi.  Jrdb-act: A large-scale dataset for spatio-temporal action, social group and activity detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20983–20992, 2022.
  * Gehl [1968] J. Gehl.  People on foot.  _Architecture_ , 20:429–446, 1968.
  * Gehl and Svarre [2013] J. Gehl and B. Svarre.  _How to study public life_ , volume 2.  Springer, 2013.
  * [12] Google Maps Platform.  Google street view static api.  URL https://developers.google.com/maps/documentation/streetview/overview.
  * Gupta and Malik [2015] S. Gupta and J. Malik.  Visual semantic role labeling.  _arXiv preprint arXiv:1505.04474_ , 2015.
  * Hosseini et al. [2022] M. Hosseini, F. Miranda, J. Lin, and C. T. Silva.  Citysurfaces: City-scale semantic segmentation of sidewalk materials.  _Sustainable Cities and Society_ , 79:103630, 2022.
  * Ibrahim et al. [2016] M. S. Ibrahim, S. Muralidharan, Z. Deng, A. Vahdat, and G. Mori.  A hierarchical deep temporal model for group activity recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1971–1980, 2016.
  * Jacobs [1961] J. Jacobs.  _The death and life of American cities_.  Random House, New York, 1961.
  * Jia et al. [2023] D. Jia, Y. Yuan, H. He, X. Wu, H. Yu, W. Lin, L. Sun, C. Zhang, and H. Hu.  Detrs with hybrid matching.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 19702–19712, 2023.
  * Jocher et al. [2023] G. Jocher, A. Chaurasia, and J. Qiu.  Ultralytics yolov8, 2023.  URL https://github.com/ultralytics/ultralytics.
  * Kamath et al. [2021] A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, I. Misra, and N. Carion.  Mdetr-modulated detection for end-to-end multi-modal understanding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1780–1790, 2021.
  * Kim et al. [2023] D. Kim, A. Angelova, and W. Kuo.  Detection-oriented image-text pretraining for open-vocabulary detection.  _arXiv preprint arXiv:2310.00161_ , 2023.
  * Kopf et al. [2010] J. Kopf, B. Chen, R. Szeliski, and M. Cohen.  Street slide: browsing street level imagery.  _ACM Transactions on Graphics (TOG)_ , 29(4):1–8, 2010.
  * Lan et al. [2012] T. Lan, L. Sigal, and G. Mori.  Social roles in hierarchical models for human activity recognition.  In _2012 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1354–1361. IEEE, 2012.
  * Law and Deng [2018] H. Law and J. Deng.  Cornernet: Detecting objects as paired keypoints.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 734–750, 2018.
  * Li et al. [2023] J. Li, D. Li, S. Savarese, and S. Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In _International conference on machine learning_ , pages 19730–19742. PMLR, 2023.
  * Li et al. [2022] L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang, et al.  Grounded language-image pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10965–10975, 2022.
  * [26] Y.-L. Li, X. Liu, X. Wu, Y. Li, Z. Qiu, L. Xu, Y. Xu, H.-S. Fang, and C. Lu.  HAKE: A knowledge engine foundation for human activity understanding.  URL http://arxiv.org/abs/2202.06851.
  * Lin et al. [2014] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Lin et al. [2017] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2980–2988, 2017.
  * Liu et al. [2019] J. Liu, A. Shahroudy, M. Perez, G. Wang, L.-Y. Duan, and A. C. Kot.  Ntu rgb+ d 120: A large-scale benchmark for 3d human activity understanding.  _IEEE transactions on pattern analysis and machine intelligence_ , 42(10):2684–2701, 2019.
  * Liu et al. [2023] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv preprint arXiv:2303.05499_ , 2023.
  * Mehta and Bosson [2021] V. Mehta and J. K. Bosson.  Revisiting lively streets: Social interactions in public space.  _Journal of Planning Education and Research_ , 41(2):160–172, 2021.
  * Minderer et al. [2022] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Mahendran, A. Arnab, M. Dehghani, Z. Shen, X. Wang, X. Zhai, T. Kipf, and N. Houlsby.  Simple open-vocabulary object detection.  Springer-Verlag, 2022.  ISBN 978-3-031-20079-3.
  * Minderer et al. [2024] M. Minderer, A. Gritsenko, and N. Houlsby.  Scaling open-vocabulary object detection.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Mumford [1961] L. Mumford.  _The city in history: Its origins, its transformations, and its prospects_ , volume 67.  Houghton Mifflin Harcourt, 1961.
  * Schulter et al. [2023] S. Schulter, Y. Suh, K. M. Dafnis, Z. Zhang, S. Zhao, D. Metaxas, et al.  Omnilabel: A challenging benchmark for language-based object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11953–11962, 2023.
  * Shahroudy et al. [2016] A. Shahroudy, J. Liu, T.-T. Ng, and G. Wang.  Ntu rgb+ d: A large scale dataset for 3d human activity analysis.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1010–1019, 2016.
  * Tkachenko et al. [2020-2022] M. Tkachenko, M. Malyuk, A. Holmanyuk, and N. Liubimov.  Label Studio: Data labeling software, 2020-2022.  URL https://github.com/heartexlabs/label-studio.  Open source software available from https://github.com/heartexlabs/label-studio.
  * [38] Z. Wang, K. Ying, J. Meng, and J. Ning.  Human-to-human interaction detection.  (arXiv:2307.00464).  URL http://arxiv.org/abs/2307.00464.
  * Whyte et al. [1980] W. H. Whyte et al.  The social life of small urban spaces.  1980\.
  * Yao et al. [2024] Y. Yao, P. Liu, T. Zhao, Q. Zhang, J. Liao, C. Fang, K. Lee, and Q. Wang.  How to evaluate the generalization of detection? a benchmark for comprehensive open-vocabulary detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 38, pages 6630–6638, 2024.
  * Yun et al. [2012] K. Yun, J. Honorio, D. Chattopadhyay, T. L. Berg, and D. Samaras.  Two-person interaction detection using body-pose features and multiple instance learning.  In _2012 IEEE computer society conference on computer vision and pattern recognition workshops_ , pages 28–35. IEEE, 2012.
  * Zareian et al. [2021] A. Zareian, K. D. Rosa, D. H. Hu, and S.-F. Chang.  Open-vocabulary object detection using captions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14393–14402, 2021.
  * Zhang et al. [2022] H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. Ni, and H.-Y. Shum.  Dino: Detr with improved denoising anchor boxes for end-to-end object detection.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * [44] J. Zhou, Z. Wang, J. Meng, S. Liu, J. Zhang, and S. Chen.  Human interaction recognition with skeletal attention and shift graph convolution.  In _2022 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8. IEEE.  ISBN 978-1-72818-671-9.  doi: 10.1109/IJCNN55064.2022.9892292.  URL https://ieeexplore.ieee.org/document/9892292/.

##  Appendix A Appendix

###  A.1 Sanity Rules for Annotation Cleaning

In order to make sure that all the annotated labels for bounding boxes are
correct, we performed a sanity-check using a predefined set of sanity rules.
In the following, we summarize the full set of rules we considered at this
stage:

  1. 1.

Each bounding box must have a condition label, unless it is a “pet”,

  2. 2.

Each bounding box must have at least one state label, unless it is a “pet”,

  3. 3.

Each bounding box can only have one condition label associated, e.g., “alone”
and “group” cannot appear together,

  4. 4.

If a bounding box is associated with the “alone” condition, then it can only
have one state label associated, e.g., “alone walking running” is not allowed,

  5. 5.

If a bounding box is associated with the “couple/two person” condition, then
it can only have two state labels associated, e.g., “couple walking sitting
running” is not allowed,

  6. 6.

If a bounding box is associated with the “shopping” activity, then state
should include either one of “sitting” or “standing” labels,

  7. 7.

If a bounding box is associated with the “street vendors” activity, then state
should include either one of “sitting” or “standing” labels,

  8. 8.

If a bounding box is associated with the “load/unload packages” activity, then
state should include either one of “sitting” or “standing” labels,

  9. 9.

If a bounding box is associated with the “waiting in bus station” activity,
then state should include either one of “sitting” or “standing” labels.