# Self-Supervised Learning for Modeling Gamma-ray Variability in Blazars

Aryeh Brill

###### Abstract

Blazars are active galactic nuclei with relativistic jets pointed almost
directly at Earth. Blazars are characterized by strong, apparently stochastic
flux variability at virtually all observed wavelengths and timescales, from
minutes to years, the physical origin of which is still poorly understood. In
the high-energy gamma-ray band, the Large Area Telescope aboard the Fermi
space telescope (Fermi-LAT) has conducted regular monitoring of thousands of
blazars since 2008. Deep learning can help uncover structure in gamma-ray
blazars’ complex variability patterns that traditional methods based on
parametric statistical modeling or manual feature engineering may miss. In
this work, we propose using a self-supervised Transformer encoder architecture
to construct an effective representation of blazar gamma-ray variability.
Measurement errors, upper limits, and missing data are accommodated using
learned encodings. The model predicts a set of quantiles for the flux
probability distribution at each time step, an architecture naturally suited
for describing data generated by a stochastic process. As a proof of concept
for how the model output can be analyzed to extract scientifically relevant
information, a preliminary search for weekly-timescale time-reversal asymmetry
in gamma-ray blazar light curves was conducted, finding no significant
evidence for asymmetry.

## Introduction

Blazars are the most extreme class of active galactic nuclei (AGN), energetic
phenomena powered by accretion onto supermassive black holes in the centers of
a few percent of galaxies. A fraction of AGN host relativistic jets that
accelerate particles close to the speed of light. AGN with jets oriented close
to the line of sight towards Earth are called blazars (Urry and Padovani
1995). Blazars are the most luminous long-lasting sources of electromagnetic
radiation in the gamma-ray sky. Blazar emission extends across the
electromagnetic spectrum, including gamma rays in the high-energy (HE;
$\sim$0.1-100 GeV) band and even higher energies.

HE gamma-ray observations of blazars have been conducted since 2008 by the
Large Area Telescope on board the Fermi Gamma-Ray Space Telescope (Fermi-LAT).
Fermi-LAT detects gamma rays from 20 MeV to above 500 GeV using a pair-
conversion technique (Atwood et al. 2009). Located in low Earth orbit, Fermi-
LAT primarily operates in survey mode, during which it scans the entire sky
every 3 hr. Among other data products, Fermi-LAT generates binned gamma-ray
flux light curves (time series) for each detected source. The time bin
duration (typically chosen between 1 day and several months) is a trade-off,
with a shorter duration improving time resolution but worsening sensitivity.

Blazars can be divided into two main classes based on their optical spectra,
flat spectrum radio quasars (FSRQs) and BL Lacertae-type objects (BL Lacs).
The brightest and most variable objects in HE gamma rays are typically FSRQs.
Over 3000 blazars have been detected by Fermi-LAT (Ajello et al. 2022). Of the
Fermi-LAT blazars, 755 are FSRQs, 1379 are BL Lacs, and 1208 are of an unknown
type.

The physical processes that cause gamma-ray blazar variability are poorly
understood. Long-timescale ($\gtrsim 1$ year) gamma-ray variability may be
connected to processes in the accretion disk, while short-timescale ($\lesssim
1$ day) variability may result from emission occurring in compact structures
in the jet (Rieger 2019). Some gamma-ray blazars exhibit apparent trends (e.g.
Valverde et al. 2020) or (quasi-)periodic oscillations (e.g. Peñil et al.
2020; Rueda, Glicenstein, and Brun 2022) on timescales of years.

Of particular physical and observational interest is intermediate-timescale
variability, ranging from days to months. Most strikingly, on these timescales
blazars can undergo flares, or short-lived flux increases of as much as two
orders of magnitude (e.g. Adams et al. 2022). The physical origin of gamma-ray
flares in blazars, or even whether flares should be understood as a physical
process distinct from ordinary gamma-ray emission, is not known. For example,
a model of intermediate-timescale variability as the interaction between an
unresolved short-timescale burst process and long-timescale stochastic
variations can yield flares as an emergent property (Brill 2022). An
understanding of to what extent flares are asymmetric in time is also
valuable, as their rise and decay timescales may be connected to the
timescales of the particle acceleration and cooling processes in the emission
region (e.g. Abdo et al. 2010). For these reasons, a better understanding of
blazar variability at intermediate timescales, including the shape of the flux
probability distribution and whether it changes over time, can give crucial
insight into physical processes.

Blazar emission typically has a power spectral density (PSD) with a power-law
shape (e.g. Abdo et al. 2010), possibly with spectral breaks, indicating that
it is wholly or partly stochastic. Characteristic gamma-ray variability
timescales can be discovered through time-series modeling (e.g. Kelly,
Bechtold, and Siemiginowska 2009; Ryan et al. 2019; Brill 2022). However,
variability analysis based on PSD fitting or autoregressive modeling has
several important limitations. Model parameters, such as the shape of the flux
distribution, must be imposed. Depending on the method used, it can be
difficult to deal with missing time bins or upper limits. Methods based on
second-order statistics are restricted to modeling time-symmetric
autocorrelation. A key assumption underlying many time-series models is
stationarity, or time invariance of statistical properties (possibly after
filtering out trends or periodicities). The extent to which blazar light
curves can be properly described as stationary is still under investigation.
For example, Duda and Bhatta (2021) reported transient non-stationarity
features in gamma-ray blazar light curves. One potential form of apparent
nonstationarity could be a tendency for flaring activity to increase (or
suppress) further flux variability, manifesting in time-asymmetric conditional
heteroskedasticity. In this work, we explore how the expressive power of deep
neural networks can complement traditional analysis tools and enable novel
studies by surfacing variability patterns in a model-independent way.

Several different approaches have been used to analyze gamma-ray blazar
variability with deep neural networks. Some works have focused on the
supervised task of classifying sources of unknown spectral type as FSRQs or BL
Lacs using variability features manually extracted from light curves (e.g.
Doert and Errando 2014; Kaur et al. 2019). In most contexts, however, light
curves are unlabeled, making it natural to adopt an unsupervised approach
capable of automatically extracting information of scientific interest for
further analysis, classification, and interpretation. Furthermore, light
curves may contain rich variability patterns that manually determined features
do not adequately describe.

One possible approach, increasingly applied in optical astronomy (Huertas-
Company and Lanusse 2022), has used recurrent autoencoders in a semi-
supervised or anomaly detection framework to search for transient events such
as supernovae (e.g. Villar et al. 2020, 2021) or anomalous variability
behavior in AGN (Sánchez-Sáez et al. 2021). Using a recurrent autoencoder,
Tachibana et al. (2020) found evidence of temporal asymmetry in the optical
variability of quasars, though such an asymmetry could have been produced by a
selection bias induced at the time of sample selection (Shen and Burke 2021).
Autoencoders can efficiently represent a light curve’s overall long-term
variability structure. However, because their compression objective leads them
to discard short-term fluctuations, they are less well suited for modeling
variability on shorter time scales.

Instead, we propose an approach to uncovering variability patterns in gamma-
ray light curves based on self-supervised learning (SSL), in which some of the
data points in the input light curves are randomly masked, and a deep neural
network is trained to predict the missing data based on its context. We employ
a model closely based on Bidirectional Encoder Representations from
Transformers (BERT; Devlin et al. 2018), widely used in natural language
processing. BERT is built around the Transformer architecture (Vaswani et al.
2017), which employs a self-attention mechanism to efficiently model complex
long-term dependencies in data. In astronomy, SSL using Transformers has been
used to remove noise and outliers from optical light curves (Morvan et al.
2022) and to perform pre-training for classifying the light curves of variable
stars (Donoso-Oliva et al. 2022).

In this work, we have applied SSL to extract variability information from
gamma-ray light curves of blazars. In order to naturally represent the
variability structure of stochastic light curves, we introduce a training
objective in which the network predicts a probability distribution, rather
than a scalar value, for each missing data point, represented non-
parametrically as a set of quantiles. After training, the resulting
probability distributions can be analyzed to obtain quantities of scientific
interest. As a proof of concept, we applied SSL to search for weekly-timescale
time-reversal asymmetry in gamma-ray blazar light curves.

## Data

For our gamma-ray dataset, we used the publicly available light curves from
the Fermi-LAT Light Curve
Repository111https://fermi.gsfc.nasa.gov/ssc/data/access/lat/LightCurveRepository/
(LCR; Fermi Large Area Telescope Collaboration 2023). The LCR consists of
light curves for 1525 sources estimated to have a less than 1% chance of being
a steady source over 10 time intervals in the source catalog of the first 10
years of Fermi-LAT observations (Ballet et al. 2020). Of these, about 93% are
blazars.

The LCR provides light curves with time bin sizes of 3 days, 7 days, and 30
days. The light curves are generated by a maximum likelihood analysis. For
each time bin, the significance is estimated by a likelihood test statistic
($TS$; Mattox et al. 1996). For time bins with a significant detection, the
flux $F$ and $1\sigma$ errors $\sigma_{F}$ are provided; otherwise a 95%
photon flux upper limit $UL$ is reported. Time bins may be missing entirely if
insufficient data were collected during a time bin or if the analysis
otherwise failed to converge. We downloaded the 7 day photon flux light curves
corresponding to a fixed spectral index for all sources for the time interval
from August 5, 2008 through March 14, 2022, for a maximum of 710 time bins per
source.

Guided by the LCR usage
notes222https://fermi.gsfc.nasa.gov/ssc/data/access/lat/LightCurveRepository/about.html,
we applied a number of quality cuts to the data. Potentially non-convergent
bins were removed by excluding bins with a nonzero return code; $TS\leq 0$;
$UL<0$; $(F/\sigma_{F})^{2}/TS\geq 1$; or $(F/\sigma_{F})^{2}/TS\leq 0.1$.
Upper limits were selected for all time bins with $TS<4$, and low-quality data
points with $F\leq\sigma_{F}$ were excluded.

On March 16, 2018, the Fermi spacecraft experienced an anomaly in its drive
system, after which it adopted a modified orbital
profile333https://fermi.gsfc.nasa.gov/ssc/observations/types/post˙anomaly/.
This resulted in occasional periods of low exposure for some sources, leading
to erroneous photon flux estimates. To remove any data points potentially
affected by this anomaly, for each source $i$, data points with
$\sigma_{F}/\mu_{i}<1.5$ and upper limits with $UL/2\mu_{i}<1.5$ were
excluded, where $\mu_{i}$ is that source’s mean photon flux.

Finally, any source with fewer than 100 flux data points; fewer than 300
combined data points and upper limits; or not classified as an FSRQ, BL Lac,
or blazar candidate of unknown type was excluded, leaving 689 sources. Fig. 1
shows the light curves after all cuts of the most and least variable sources
in the final dataset, as determined by the 4FGL-DR2 variability index (Ballet
et al. 2020). Each time bin has either an associated flux $F$ and error
$\sigma_{F}$, or an upper limit $UL$, all in units of photons cm-2 s-1, as
well as an associated time index $0\leq t<710$.

Figure 1: Light curves of the most (top) and least (bottom) variable sources
in the LCR dataset after all cuts. Upper limits are indicated by orange
triangles.

In order to develop and validate the network, we also created a simulated
dataset intended to roughly emulate the general statistical features of the
LCR dataset. 1378 light curves were simulated, of which 689 were used for
training and 689 were reserved for validation. Each light curve consisted of
710 time steps and was generated following a first-order autoregressive
(AR(1)) process,

$y_{t+1}=\phi y_{t}+\epsilon_{t},$ (1)

where $\epsilon_{t}\sim\mathcal{N}(0,\sigma)$. The light curves were
exponentiated to obtain a lognormal flux distribution and scaled by a random
normalization parameter $\mu_{N}$. The AR(1) parameters of each light curve
were randomly selected following the distribution parameters in Table 1.
Simulated measurement errors were applied to the light curves and time bins
were randomly removed using a flux/error correspondence and missing-bin
probability empirically estimated from the LCR data. Time bins with
$F<2\sigma_{F}$ were replaced with upper limits such that
$UL=\mathrm{max}(2\sigma_{F}+F,2\sigma_{F})$. A comparison of some statistical
properties of the real and simulated datasets is shown in Fig. 2.

Param. | Distribution | $\mu_{\mathrm{dist}}$ | $\sigma_{\mathrm{dist}}$ | Min | Max
---|---|---|---|---|---
$\phi$ | Trunc. Norm. | 0.70 | 0.30 | 0 | 0.975
$\sigma$ | Trunc. Norm. | 0.45 | 0.05 | 0 | -
$\mu_{N}$ | Lognormal | -17.4 | 0.8 | - | -
Table 1: Distributions of parameters used to generate the simulated light
curves. Figure 2: A comparison of statistical properties of the real and
simulated datasets, showing, from left to right, the mean photon flux, the
standard deviation of the natural logarithm of the flux, and the lag-1
autocorrelation function of the natural logarithm of the flux. Upper limits
were excluded from all calculations. No adjustments were made to account for
missing time bins.

## Methodology

The core of our model consisted of a BERT-style Transformer encoder network
with $L=4$, $H=32$, and $A=4$, with $L$ the number of layers, $H$ the hidden
size, and $A$ the number of self-attention heads. The feedforward dimension is
set to $4H$ and dropout was used with probability 0.1. Adam was used to train
the model with a learning rate of $1\times 10^{-3}$, $\beta_{1}=0.9$, and
$\beta_{2}=0.999$ (Kingma and Ba 2014).

We applied several preprocessing steps to transform the data into an input
format suitable for the network. First, for each light curve, a segment of 101
time bins was extracted, with the start point chosen randomly. The time
indices were shifted to make the index of the central data point 0. Time steps
were randomly selected for masking with 20% probability. Of those selected, a
mask was actually applied with 90% probability, otherwise, the data point was
unaltered. The natural logarithm was applied to the flux points, $1\sigma$
flux lower and upper bounds, and flux upper limits. The mean of the unmasked
flux points in the segment was subtracted. To create the input vector for the
network, the time indices and flux values were converted to vectors of length
32 using fixed trigonometric encodings (Vaswani et al. 2017). Since the flux
values and time index both used fixed encodings, the vectors were concatenated
rather than summed, yielding input vectors of length 128. For upper limits,
the flux upper and lower limits were replaced by a single learned upper limit
encoding, while for masked time steps all three flux values were replaced by a
single learned mask encoding (superseding any upper limit encoding). This
process was repeated in batches, with a batch size of 64.

To generate a non-parametric probability distribution for the flux at each
time step, the output encoding was fed through a Quantile Head, illustrated in
Fig. 3. Three independent stacks of fully-connected networks computed the
median; quantiles less than 0.5; and quantiles greater than 0.5, respectively.
The Quantile Head efficiently prevents quantile crossing, i.e., non-
monotonicity of the estimated quantiles, by applying a softplus activation
followed by a cumulative sum to the estimated quantile vectors. The vector for
quantiles less than 0.5 was flipped and negated, the median added to the other
quantiles, and all of the quantiles finally concatenated. We used the set of
19 quantiles for probabilities 0.05 to 0.95, in increments of 0.05.

Figure 3: Diagram of the Quantile Head applied to the output embedding at each
time step.

The SSL objective was the quantile score or pinball loss function,

$\mathcal{L}=2\sum_{p}\max(-p(q_{p}-y),(1-p)(q_{p}-y)),$ (2)

where $q_{p}$ is the quantile corresponding to probability $p\in(0,1)$ and $y$
is the measured value of $\log(F)$. For $p=0.5$, the quantile score reduces to
the absolute loss. Only masked flux points were included in the loss; any
masked upper limits were ignored.

## Results

The network was first trained on the simulated dataset for 1000 epochs,
monitoring the validation loss throughout training to confirm that it remained
consistent with the training loss. The network was then retrained from scratch
on the real LCR dataset using the same procedure. Each model was subsequently
used to generate predictions on its respective corresponding dataset only.

An example of the output of the model trained on real data is shown in Fig 4.
For each data point of each source, predicted quantiles were generated by
using as input the interval of length 101 centered on that data point,
replacing the data for that point but no others with a mask encoding. No
quantiles were generated for the first and last 50 data points.

Figure 4: Example of quantiles generated by the trained model, for a portion
of the light curve of the most variable LCR source (Fig. 1). Gray triangles
indicate upper limits.

The trained models were then used to perform a search for time-reversal
asymmetry in flux variability on weekly timescales. We considered the simple
case in which the distribution of $\log(F)$ is assumed to be normal with a
standard deviation that may vary at different time steps. To do so, a time-
reversed set of predicted quantiles was generated as above, except that the
time indices within each interval were reversed. The distributions were
centered at 0 by subtracting their respective median. The standard deviation
of $\log(F)$ was estimated from each set of forward and time-reversed
quantiles as the mean of their values divided by the respective values for a
normal distribution. For each data point, the relative difference
$\Delta_{\mathrm{rel}}$ between the forward standard deviation
$\sigma_{\mathrm{for}}$ and time-reversed one $\sigma_{\mathrm{rev}}$ was
defined as

$\Delta_{\mathrm{rel}}=2\frac{\sigma_{\mathrm{for}}-\sigma_{\mathrm{rev}}}{\sigma_{\mathrm{for}}+\sigma_{\mathrm{rev}}}.$
(3)

The distributions of the mean relative difference
$\mu_{\Delta_{\mathrm{rel}}}$ for each source in the simulated and real
datasets are shown in Fig. 5. The average value of
$\mu_{\Delta_{\mathrm{rel}}}$ for the simulated dataset was $-0.002\pm 0.003$
and the average for the real dataset was $0.002\pm 0.004$. No significant
deviation from 0 was observed for either dataset. By construction, the mean
relative difference for the simulated dataset should be 0 for every source.
One possible explanation for a negative relative difference could be the
network overfitting on the time orientation it was trained on, causing it to
be slightly more confident in the forward direction. For the real data, we
therefore adopted 0.002 as an additional systematic uncertainty on the mean
relative difference. Incorporating this systematic uncertainty, a preliminary
upper bound was placed on the time asymmetry of the amplitude of weekly
variability of $\left|\mu_{\Delta_{\mathrm{rel}}}\right|\lesssim 0.01$ at the
95% confidence level.

Figure 5: Distributions of the mean relative difference in standard deviation
of $\log(F)$ for the simulated and real datasets.

## Discussion

In this work, we investigated self-supervised learning, a potentially powerful
new tool for astrophysical data analysis. As a proof of concept, we used
predicted flux distributions generated by a self-supervised Transformer
encoder network to perform a novel search for weekly-timescale time-reversal
asymmetry in gamma-ray blazar light curves. No significant time asymmetry in
the amplitude of the flux distribution was found, suggesting that the
amplitude of gamma-ray variability in blazars does not itself vary on weekly
timescales. This finding is relevant for many statistical methods commonly
used in high-energy astrophysics that assume time-reversal symmetry, such as
autoregressive modeling.

More generally, the self-supervised framework has a number of advantages
relative to standard time series analysis methods used with astrophysical
data. Upper limits, measurement errors, and missing data can be naturally
accommodated using learned encodings. The neural network output is independent
of any statistical model or parameterization and can be used for a variety of
tasks, making it an unbiased basis for comparison between models or for
exploratory searches. In particular, although we did not rely on this property
in this work, the Quantile Head generates non-parametric probability
distributions. It can therefore be used as a foundation for a variety of
studies, such as comparing different intermediate-timescale flux probability
distributions. In addition, the trained network encodes knowledge of and
generalizes variability patterns learned from an entire dataset, as opposed to
a statistical model which must be fit to one light curve at a time. Finally,
although we have focused exclusively on gamma-ray data in this work, the
network architecture presented here would naturally extend to accommodate data
from multiple wavebands at once, potentially allowing novel multiwavelength
variability analyses to be conducted.

Multiple avenues exist for future studies to build on the method discussed
here. The flux distributions predicted by the network in the present
architecture mix together several sources of variability, incorporating not
only variability due to physical processes in the blazar but also measurement
error; nuisance parameters such as luminosity and redshift; and estimation
error due to the finite length of the light curves. One potential way to
eliminate the variability caused by measurement error from the predicted
distributions would be to add a secondary self-supervised objective to predict
the measurement error at each time step. The predicted error distribution
could then be deconvolved from the predicted flux distribution using a Monte
Carlo or other approach. In addition, physically-motivated data augmentations,
such as stretch and scale transformations to mimic the effect of changing
redshift, could be applied to help the network disentangle the effects of
cosmological nuisance parameters, as well as to enhance the effective size of
the dataset. However, because measurement error is nonlinearly correlated with
flux, among other factors, doing so would likely require a more detailed model
of the Fermi-LAT sensitivity. These methods may provide an improved estimation
of the intrinsic variability, enabling more detailed studies of the predicted
flux probability distributions. Unsupervised clustering and correlation
studies using the generated output encodings could also be performed.

## References

  * Abdo et al. (2010) Abdo, A. A.; Ackermann, M.; Ajello, M.; Antolini, E.; Baldini, L.; Ballet, J.; Barbiellini, G.; Bastieri, D.; Bechtol, K.; Bellazzini, R.; Berenji, B.; Blandford, R. D.; Bloom, E. D.; Bonamente, E.; Borgland, A. W.; Bouvier, A.; Bregeon, J.; Brez, A.; Brigida, M.; Bruel, P.; Buehler, R.; Burnett, T. H.; Buson, S.; Caliandro, G. A.; Cameron, R. A.; Caraveo, P. A.; Carrigan, S.; Casandjian, J. M.; Cavazzuti, E.; Cecchi, C.; Çelik, Ö.; Chekhtman, A.; Cheung, C. C.; Chiang, J.; Ciprini, S.; Claus, R.; Cohen-Tanugi, J.; Cominsky, L. R.; Conrad, J.; Costamante, L.; Cutini, S.; Dermer, C. D.; de Angelis, A.; de Palma, F.; Silva, E. d. C. e.; Drell, P. S.; Dubois, R.; Dumora, D.; Farnier, C.; Favuzzi, C.; Fegan, S. J.; Focke, W. B.; Fortin, P.; Frailis, M.; Fukazawa, Y.; Funk, S.; Fusco, P.; Gargano, F.; Gasparrini, D.; Gehrels, N.; Germani, S.; Giebels, B.; Giglietto, N.; Giommi, P.; Giordano, F.; Glanzman, T.; Godfrey, G.; Grenier, I. A.; Grondin, M. H.; Grove, J. E.; Guiriec, S.; Hadasch, D.; Hayashida, M.; Hays, E.; Healey, S. E.; Horan, D.; Hughes, R. E.; Itoh, R.; Jóhannesson, G.; Johnson, A. S.; Johnson, W. N.; Kamae, T.; Katagiri, H.; Kataoka, J.; Kawai, N.; Knödlseder, J.; Kuss, M.; Lande, J.; Larsson, S.; Latronico, L.; Lemoine-Goumard, M.; Longo, F.; Loparco, F.; Lott, B.; Lovellette, M. N.; Lubrano, P.; Madejski, G. M.; Makeev, A.; Massaro, E.; Mazziotta, M. N.; McEnery, J. E.; Michelson, P. F.; Mitthumsiri, W.; Mizuno, T.; Moiseev, A. A.; Monte, C.; Monzani, M. E.; Morselli, A.; Moskalenko, I. V.; Mueller, M.; Murgia, S.; Nolan, P. L.; Norris, J. P.; Nuss, E.; Ohno, M.; Ohsugi, T.; Omodei, N.; Orlando, E.; Ormes, J. F.; Ozaki, M.; Panetta, J. H.; Parent, D.; Pelassa, V.; Pepe, M.; Pesce-Rollins, M.; Piron, F.; Porter, T. A.; Rainò, S.; Rando, R.; Razzano, M.; Reimer, A.; Reimer, O.; Ritz, S.; Rodriguez, A. Y.; Romani, R. W.; Roth, M.; Ryde, F.; Sadrozinski, H. F. W.; Sander, A.; Scargle, J. D.; Sgrò, C.; Shaw, M. S.; Smith, P. D.; Spandre, G.; Spinelli, P.; Starck, J. L.; Strickman, M. S.; Suson, D. J.; Takahashi, H.; Takahashi, T.; Tanaka, T.; Thayer, J. B.; Thayer, J. G.; Thompson, D. J.; Tibaldo, L.; Torres, D. F.; Tosti, G.; Tramacere, A.; Uchiyama, Y.; Usher, T. L.; Vasileiou, V.; Vilchez, N.; Vitale, V.; Waite, A. P.; Wallace, E.; Wang, P.; Winer, B. L.; Wood, K. S.; Yang, Z.; Ylinen, T.; and Ziegler, M. 2010.  Gamma-ray Light Curves and Variability of Bright Fermi-detected Blazars.  _The Astrophysical Journal_ , 722(1): 520–542.
  * Adams et al. (2022) Adams, C. B.; Batshoun, J.; Benbow, W.; Brill, A.; Buckley, J. H.; Capasso, M.; Cavins, B.; Christiansen, J. L.; Coppi, P.; Errando, M.; Farrell, K. A.; Feng, Q.; Finley, J. P.; Foote, G. M.; Fortson, L.; Furniss, A.; Gent, A.; Giuri, C.; Hanna, D.; Hassan, T.; Hervet, O.; Holder, J.; Houck, M.; Humensky, T. B.; Jin, W.; Kaaret, P.; Kertzman, M.; Kieda, D.; Krennrich, F.; Kumar, S.; Lundy, M.; Maier, G.; McGrath, C. E.; Moriarty, P.; Mukherjee, R.; Nieto, D.; Nievas-Rosillo, M.; O’Brien, S.; Ong, R. A.; Oppenheimer, A.; Otte, A. N.; Patel, S.; Pfrang, K.; Pohl, M.; Prado, R. R.; Pueschel, E.; Quinn, J.; Ragan, K.; Reynolds, P. T.; Rhatigan, A.; Ribeiro, D.; Roache, E.; Ryan, J. L.; Santander, M.; Sembroski, G. H.; Williams, D. A.; Williamson, T. J.; Valverde, J.; Horan, D.; Buson, S.; Cheung, C. C.; Ciprini, S.; Gasparrini, D.; Ojha, R.; van Zyl, P.; and Sironi, L. 2022.  Variability and Spectral Characteristics of Three Flaring Gamma-Ray Quasars Observed by VERITAS and Fermi-LAT.  _The Astrophysical Journal_ , 924(2): 95.
  * Ajello et al. (2022) Ajello, M.; Baldini, L.; Ballet, J.; Bastieri, D.; Becerra Gonzalez, J.; Bellazzini, R.; Berretta, A.; Bissaldi, E.; Bonino, R.; Brill, A.; Bruel, P.; Buson, S.; Caputo, R.; Caraveo, P.; Cheung, T.; Chiaro, G.; Cibrario, N.; Ciprini, S.; Crnogorcevic, M.; Cutini, S.; D’Ammando, F.; De Gaetano, S.; Di Lalla, N.; Di Venere, L.; Dominguez, A.; Fallah Ramazani, V.; Ferrara, E.; Fiori, A.; Fukazawa, Y.; Funk, S.; Fusco, P.; Gammaldi, V.; Gargano, F.; Garrappa, S.; Gasparrini, D.; Giglietto, N.; Giordano, F.; Giroletti, M.; Green, D.; Grenier, I.; Guiriec, S.; Horan, D.; Hou, X.; Kayanoki, T.; Kuss, M.; Larsson, S.; Latronico, L.; Lewis, T.; Li, J.; Liodakis, I.; Longo, F.; Loparco, F.; Lott, B.; Lovellette, M.; Lubrano, P.; Madejski, G.; Maldera, S.; Manfreda, A.; Marti-Devesa, G.; Mazziotta, M. N.; Mereu, I.; Michelson, P.; Mirabal, N.; Mitthumsiri, W.; Mizuno, T.; Monzani, M. E.; Morselli, A.; Moskalenko, I.; Negro, M.; Ojha, R.; Orienti, M.; Orlando, E.; Ormes, J. F.; Pei, Z.; Pena-Herazo, H. A.; Persic, M.; Pesce-Rollins, M.; Petrosian, V.; Pillera, R.; Poon, H.; Porter, T.; Principe, G.; Raino, S.; Rando, R.; Rani, B.; Razzano, M.; Razzaque, S.; Reimer, A.; Reimer, O.; Scargle, J.; Scotton, L.; Serini, D.; Sgro, C.; Siskind, E. J.; Spandre, G.; Spinelli, P.; Suson, D.; Tajima, H.; Torres, D. F.; Valverde, J.; Yassin, H.; and Zaharijas, G. 2022.  The Fourth Catalog of Active Galactic Nuclei Detected by the Fermi Large Area Telescope – Data Release 3.  _arXiv e-prints_ , arXiv:2209.12070.
  * Atwood et al. (2009) Atwood, W. B.; Abdo, A. A.; Ackermann, M.; Althouse, W.; Anderson, B.; Axelsson, M.; Baldini, L.; Ballet, J.; Band, D. L.; Barbiellini, G.; Bartelt, J.; Bastieri, D.; Baughman, B. M.; Bechtol, K.; Bédérède, D.; Bellardi, F.; Bellazzini, R.; Berenji, B.; Bignami, G. F.; Bisello, D.; Bissaldi, E.; Blandford, R. D.; Bloom, E. D.; Bogart, J. R.; Bonamente, E.; Bonnell, J.; Borgland, A. W.; Bouvier, A.; Bregeon, J.; Brez, A.; Brigida, M.; Bruel, P.; Burnett, T. H.; Busetto, G.; Caliandro, G. A.; Cameron, R. A.; Caraveo, P. A.; Carius, S.; Carlson, P.; Casandjian, J. M.; Cavazzuti, E.; Ceccanti, M.; Cecchi, C.; Charles, E.; Chekhtman, A.; Cheung, C. C.; Chiang, J.; Chipaux, R.; Cillis, A. N.; Ciprini, S.; Claus, R.; Cohen-Tanugi, J.; Condamoor, S.; Conrad, J.; Corbet, R.; Corucci, L.; Costamante, L.; Cutini, S.; Davis, D. S.; Decotigny, D.; DeKlotz, M.; Dermer, C. D.; de Angelis, A.; Digel, S. W.; do Couto e Silva, E.; Drell, P. S.; Dubois, R.; Dumora, D.; Edmonds, Y.; Fabiani, D.; Farnier, C.; Favuzzi, C.; Flath, D. L.; Fleury, P.; Focke, W. B.; Funk, S.; Fusco, P.; Gargano, F.; Gasparrini, D.; Gehrels, N.; Gentit, F. X.; Germani, S.; Giebels, B.; Giglietto, N.; Giommi, P.; Giordano, F.; Glanzman, T.; Godfrey, G.; Grenier, I. A.; Grondin, M. H.; Grove, J. E.; Guillemot, L.; Guiriec, S.; Haller, G.; Harding, A. K.; Hart, P. A.; Hays, E.; Healey, S. E.; Hirayama, M.; Hjalmarsdotter, L.; Horn, R.; Hughes, R. E.; Jóhannesson, G.; Johansson, G.; Johnson, A. S.; Johnson, R. P.; Johnson, T. J.; Johnson, W. N.; Kamae, T.; Katagiri, H.; Kataoka, J.; Kavelaars, A.; Kawai, N.; Kelly, H.; Kerr, M.; Klamra, W.; Knödlseder, J.; Kocian, M. L.; Komin, N.; Kuehn, F.; Kuss, M.; Landriu, D.; Latronico, L.; Lee, B.; Lee, S. H.; Lemoine-Goumard, M.; Lionetto, A. M.; Longo, F.; Loparco, F.; Lott, B.; Lovellette, M. N.; Lubrano, P.; Madejski, G. M.; Makeev, A.; Marangelli, B.; Massai, M. M.; Mazziotta, M. N.; McEnery, J. E.; Menon, N.; Meurer, C.; Michelson, P. F.; Minuti, M.; Mirizzi, N.; Mitthumsiri, W.; Mizuno, T.; Moiseev, A. A.; Monte, C.; Monzani, M. E.; Moretti, E.; Morselli, A.; Moskalenko, I. V.; Murgia, S.; Nakamori, T.; Nishino, S.; Nolan, P. L.; Norris, J. P.; Nuss, E.; Ohno, M.; Ohsugi, T.; Omodei, N.; Orlando, E.; Ormes, J. F.; Paccagnella, A.; Paneque, D.; Panetta, J. H.; Parent, D.; Pearce, M.; Pepe, M.; Perazzo, A.; Pesce-Rollins, M.; Picozza, P.; Pieri, L.; Pinchera, M.; Piron, F.; Porter, T. A.; Poupard, L.; Rainò, S.; Rando, R.; Rapposelli, E.; Razzano, M.; Reimer, A.; Reimer, O.; Reposeur, T.; Reyes, L. C.; Ritz, S.; Rochester, L. S.; Rodriguez, A. Y.; Romani, R. W.; Roth, M.; Russell, J. J.; Ryde, F.; Sabatini, S.; Sadrozinski, H. F. W.; Sanchez, D.; Sander, A.; Sapozhnikov, L.; Parkinson, P. M. S.; Scargle, J. D.; Schalk, T. L.; Scolieri, G.; Sgrò, C.; Share, G. H.; Shaw, M.; Shimokawabe, T.; Shrader, C.; Sierpowska-Bartosik, A.; Siskind, E. J.; Smith, D. A.; Smith, P. D.; Spandre, G.; Spinelli, P.; Starck, J. L.; Stephens, T. E.; Strickman, M. S.; Strong, A. W.; Suson, D. J.; Tajima, H.; Takahashi, H.; Takahashi, T.; Tanaka, T.; Tenze, A.; Tether, S.; Thayer, J. B.; Thayer, J. G.; Thompson, D. J.; Tibaldo, L.; Tibolla, O.; Torres, D. F.; Tosti, G.; Tramacere, A.; Turri, M.; Usher, T. L.; Vilchez, N.; Vitale, V.; Wang, P.; Watters, K.; Winer, B. L.; Wood, K. S.; Ylinen, T.; and Ziegler, M. 2009.  The Large Area Telescope on the Fermi Gamma-Ray Space Telescope Mission.  _The Astrophysical Journal_ , 697(2): 1071–1102.
  * Ballet et al. (2020) Ballet, J.; Burnett, T. H.; Digel, S. W.; and Lott, B. 2020.  Fermi Large Area Telescope Fourth Source Catalog Data Release 2.  _arXiv e-prints_ , arXiv:2005.11208.
  * Brill (2022) Brill, A. 2022.  Variability Signatures of a Burst Process in Flaring Gamma-Ray Blazars.  _The Astrophysical Journal_ , 936(2): 147.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  _arXiv e-prints_ , arXiv:1810.04805.
  * Doert and Errando (2014) Doert, M.; and Errando, M. 2014.  Search for Gamma-ray-emitting Active Galactic Nuclei in the Fermi-LAT Unassociated Sample Using Machine Learning.  _The Astrophysical Journal_ , 782(1): 41.
  * Donoso-Oliva et al. (2022) Donoso-Oliva, C.; Becker, I.; Protopapas, P.; Cabrera-Vives, G.; Vishnu, M.; and Vardhan, H. 2022.  ASTROMER: A transformer-based embedding for the representation of light curves.  _arXiv e-prints_ , arXiv:2205.01677.
  * Duda and Bhatta (2021) Duda, J.; and Bhatta, G. 2021.  Gamma-ray blazar variability: new statistical methods of time-flux distributions.  _Monthly Notices of the Royal Astronomical Society_ , 508(1): 1446–1458.
  * Fermi Large Area Telescope Collaboration (2023) Fermi Large Area Telescope Collaboration. 2023.  The Fermi-LAT Light Curve Repository.  _arXiv e-prints_ , arXiv:2301.01607.
  * Huertas-Company and Lanusse (2022) Huertas-Company, M.; and Lanusse, F. 2022.  The DAWES review 10: The impact of deep learning for the analysis of galaxy surveys.  _arXiv e-prints_ , arXiv:2210.01813.
  * Kaur et al. (2019) Kaur, A.; Falcone, A. D.; Stroh, M. D.; Kennea, J. A.; and Ferrara, E. C. 2019.  Classification of New X-Ray Counterparts for Fermi Unassociated Gamma-Ray Sources Using the Swift X-Ray Telescope.  _The Astrophysical Journal_ , 887(1): 18.
  * Kelly, Bechtold, and Siemiginowska (2009) Kelly, B. C.; Bechtold, J.; and Siemiginowska, A. 2009.  Are the Variations in Quasar Optical Flux Driven by Thermal Fluctuations?  _The Astrophysical Journal_ , 698(1): 895–910.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A Method for Stochastic Optimization.  _arXiv e-prints_ , arXiv:1412.6980.
  * Mattox et al. (1996) Mattox, J. R.; Bertsch, D. L.; Chiang, J.; Dingus, B. L.; Digel, S. W.; Esposito, J. A.; Fierro, J. M.; Hartman, R. C.; Hunter, S. D.; Kanbach, G.; Kniffen, D. A.; Lin, Y. C.; Macomb, D. J.; Mayer-Hasselwander, H. A.; Michelson, P. F.; von Montigny, C.; Mukherjee, R.; Nolan, P. L.; Ramanamurthy, P. V.; Schneid, E.; Sreekumar, P.; Thompson, D. J.; and Willis, T. D. 1996.  The Likelihood Analysis of EGRET Data.  _The Astrophysical Journal_ , 461: 396.
  * Morvan et al. (2022) Morvan, M.; Nikolaou, N.; Hou Yip, K.; and Waldmann, I. 2022.  Don’t Pay Attention to the Noise: Learning Self-supervised Representations of Light Curves with a Denoising Time Series Transformer.  _arXiv e-prints_ , arXiv:2207.02777.
  * Peñil et al. (2020) Peñil, P.; Domínguez, A.; Buson, S.; Ajello, M.; Otero-Santos, J.; Barrio, J. A.; Nemmen, R.; Cutini, S.; Rani, B.; Franckowiak, A.; and Cavazzuti, E. 2020.  Systematic Search for $\gamma$-Ray Periodicity in Active Galactic Nuclei Detected by the Fermi Large Area Telescope.  _The Astrophysical Journal_ , 896(2): 134.
  * Rieger (2019) Rieger, F. 2019.  Gamma-Ray Astrophysics in the Time Domain.  _Galaxies_ , 7(1): 28.
  * Rueda, Glicenstein, and Brun (2022) Rueda, H.; Glicenstein, J.-F.; and Brun, F. 2022.  Search for Periodicities in High Energy AGNs with a Time Domain Approach.  _The Astrophysical Journal_ , 934(1): 6.
  * Ryan et al. (2019) Ryan, J. L.; Siemiginowska, A.; Sobolewska, M. A.; and Grindlay, J. 2019\.  Characteristic Variability Timescales in the Gamma-Ray Power Spectra of Blazars.  _The Astrophysical Journal_ , 885(1): 12.
  * Sánchez-Sáez et al. (2021) Sánchez-Sáez, P.; Lira, H.; Martí, L.; Sánchez-Pi, N.; Arredondo, J.; Bauer, F. E.; Bayo, A.; Cabrera-Vives, G.; Donoso-Oliva, C.; Estévez, P. A.; Eyheramendy, S.; Förster, F.; Hernández-García, L.; Arancibia, A. M. M.; Pérez-Carrasco, M.; Sepúlveda, M.; and Vergara, J. R. 2021.  Searching for Changing-state AGNs in Massive Data Sets. I. Applying Deep Learning and Anomaly-detection Techniques to Find AGNs with Anomalous Variability Behaviors.  _The Astronomical Journal_ , 162(5): 206.
  * Shen and Burke (2021) Shen, Y.; and Burke, C. J. 2021.  A Sample Bias in Quasar Variability Studies.  _The Astrophysical Journal Letters_ , 918(1): L19.
  * Tachibana et al. (2020) Tachibana, Y.; Graham, M. J.; Kawai, N.; Djorgovski, S. G.; Drake, A. J.; Mahabal, A. A.; and Stern, D. 2020.  Deep Modeling of Quasar Variability.  _The Astrophysical Journal_ , 903(1): 54.
  * Urry and Padovani (1995) Urry, C. M.; and Padovani, P. 1995.  Unified Schemes for Radio-Loud Active Galactic Nuclei.  _Publications of the Astronomical Society of the Pacific_ , 107: 803\.
  * Valverde et al. (2020) Valverde, J.; Horan, D.; Bernard, D.; Fegan, S.; Fermi-LAT Collaboration; Abeysekara, A. U.; Archer, A.; Benbow, W.; Bird, R.; Brill, A.; Brose, R.; Buchovecky, M.; Buckley, J. H.; Christiansen, J. L.; Cui, W.; Falcone, A.; Feng, Q.; Finley, J. P.; Fortson, L.; Furniss, A.; Gent, A.; Gillanders, G. H.; Giuri, C.; Gueta, O.; Hanna, D.; Hassan, T.; Hervet, O.; Holder, J.; Hughes, G.; Humensky, T. B.; Kaaret, P.; Kelley-Hoskins, N.; Kertzman, M.; Kieda, D.; Krause, M.; Krennrich, F.; Lang, M. J.; Maier, G.; Moriarty, P.; Mukherjee, R.; Nieto, D.; Nievas-Rosillo, M.; O’Brien, S.; Ong, R. A.; Otte, A. N.; Park, N.; Petrashyk, A.; Pfrang, K.; Pichel, A.; Pohl, M.; Prado, R. R.; Pueschel, E.; Quinn, J.; Ragan, K.; Reynolds, P. T.; Ribeiro, D.; Richards, G. T.; Roache, E.; Sadeh, I.; Santander, M.; Scott, S. S.; Sembroski, G. H.; Shahinyan, K.; Shang, R.; Sushch, I.; Vassiliev, V. V.; Weinstein, A.; Wells, R. M.; Wilcox, P.; Wilhelm, A.; Williams, D. A.; Williamson, T. J.; VERITAS Collaboration; Noto, G.; Edwards, P. G.; Piner, B. G.; Fallah Ramazani, V.; Hovatta, T.; Jormanainen, J.; Lindfors, E.; Nilsson, K.; Takalo, L.; Kovalev, Y. Y.; Lister, M. L.; Pushkarev, A. B.; Savolainen, T.; Kiehlmann, S.; Max-Moerbeck, W.; Readhead, A. C. S.; Lähteenmäki, A.; and Tornikoski, M. 2020.  A Decade of Multiwavelength Observations of the TeV Blazar 1ES 1215+303: Extreme Shift of the Synchrotron Peak Frequency and Long-term Optical-Gamma-Ray Flux Increase.  _The Astrophysical Journal_ , 891(2): 170.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention Is All You Need.  _arXiv e-prints_ , arXiv:1706.03762.
  * Villar et al. (2021) Villar, V. A.; Cranmer, M.; Berger, E.; Contardo, G.; Ho, S.; Hosseinzadeh, G.; and Lin, J. Y.-Y. 2021.  A Deep-learning Approach for Live Anomaly Detection of Extragalactic Transients.  _The Astrophysical Journal Supplement Series_ , 255(2): 24.
  * Villar et al. (2020) Villar, V. A.; Hosseinzadeh, G.; Berger, E.; Ntampaka, M.; Jones, D. O.; Challis, P.; Chornock, R.; Drout, M. R.; Foley, R. J.; Kirshner, R. P.; Lunnan, R.; Margutti, R.; Milisavljevic, D.; Sanders, N.; Pan, Y.-C.; Rest, A.; Scolnic, D. M.; Magnier, E.; Metcalfe, N.; Wainscoat, R.; and Waters, C. 2020.  SuperRAENN: A Semisupervised Supernova Photometric Classification Pipeline Trained on Pan-STARRS1 Medium-Deep Survey Supernovae.  _The Astrophysical Journal_ , 905(2): 94.

## Acknowledgments

A.B. is supported by the NASA Postdoctoral Program at NASA Goddard Space
Flight Center, administered by Oak Ridge Associated Universities. Thanks to
Daniel Kocevski, Michela Negro, and Janeth Valverde for helpful discussions
regarding the Fermi-LAT Light Curve Repository. This research has made use of
NASA’s Astrophysics Data System Bibliographic Services.


# Towards Zero-shot Human-Object Interaction Detection via Vision-Language
Integration

Weiying Xue, Qi Liu*, Qiwei Xiong, Yuxiao Wang, Zhenao Wei,
Xiaofen Xing, Xiangmin Xu * Corresponding author: Qi Liu
(drliuqi@scut.edu.cn)Weiying Xue (202320163283@mail.scut.edu.cn), Qi Liu,
Qiwei Xiong (xiongqiwei@vip.qq.com), Yuxiao Wang
(ftwangyuxiao@mail.scut.edu.cn), Zhenao Wei (wza@scut.edu.cn), Xiaofen Xing
(xfxing@scut.edu.cn), and Xiangmin Xu<EMAIL_ADDRESS>are with the School
of Future Technology, South China University of Technology, China 511400.

###### Abstract

Human-object interaction (HOI) detection aims to locate human-object pairs and
identify their interaction categories in images. Most existing methods
primarily focus on supervised learning, which relies on extensive manual HOI
annotations. In this paper, we propose a novel framework, termed Knowledge
Integration to HOI (KI2HOI), that effectively integrates the knowledge of
visual-language model to improve zero-shot HOI detection. Specifically, the
verb feature learning module is designed based on visual semantics, by
employing the verb extraction decoder to convert corresponding verb queries
into interaction-specific category representations. We develop an effective
additive self-attention mechanism to generate more comprehensive visual
representations. Moreover, the innovative interaction representation decoder
effectively extracts informative regions by integrating spatial and visual
feature information through a cross-attention mechanism. To deal with zero-
shot learning in low-data, we leverage a priori knowledge from the CLIP text
encoder to initialize the linear classifier for enhanced interaction
understanding. Extensive experiments conducted on the mainstream HICO-DET and
V-COCO datasets demonstrate that our model outperforms the previous methods in
various zero-shot and full-supervised settings.

###### Index Terms:

Human-object interaction, Knowledge Integration, Zero-shot, Weakly
Supervision.

††publicationid: pubid: 0000–0000/00$00.00 © 2021 IEEE

##  I INTRODUCTION

Human-object interaction (HOI) detection is a process of detecting interaction
between a human and an object in an image [1]. Precisely estimating human-
object interactions can greatly improve various visual understanding tasks,
such as image retrieval [2], visual question answering [3], and scene graph
generation [4, 5]. Given a series of $\langle$“Human”, “Object”,
“Verb”$\rangle$ triples, an HOI detector is needed to locate human-object
pairs and identify their interactions. However, most HOI detectors typically
require a significant number of pre-defined HOI categories. Considering the
diversity and complexity of human-object interaction in the real world, it is
time-consuming and laborious to define all-natural interaction annotations in
advance manually.

Figure 1: Comparison of HOI detection. Conventional HOI detection required
manually annotated datasets for training. Previous HOI detection with language
model (LM) employed limited knowledge distillation to visual detectors, but it
is limited to handling potential interactions among unseen human-object pairs.
Our model fully leverages visual-language model (VLM) and verb queries for
effective knowledge integration, to promote unseen interaction recognition.

The development of pre-trained visual-language models on large-scale data has
significantly accelerated the progress in studying zero-shot learning [6, 7],
especially CLIP [8] demonstrates remarkable transfer ability in various
subsequent tasks. The most recent zero-shot HOI detectors leverage the
comprehensive visual and linguistic knowledge of CLIP to detect novel HOIs.
For example, the knowledge of pre-trained visual language model is transferred
to EOID [9] and DOQ [10] via knowledge distillation to achieve zero-shot HOI
detection. Previous approaches utilize cross-modal knowledge in a too-
restrictive manner, failing to fully capitalize on the potential of cross-
modal information and Language Models in the field of Human-Object Interaction
(HOI) detection. Knowledge distillation depends on the quality of the teacher
model and the used training data. When the training process does not include
unknown categories, the distillation process may be biased towards known
category samples, thereby the generalization ability is limited.

Considering the recent outstanding performance of visual-language models in
various fundamental tasks, as depicted in Figure 1, we propose a novel one-
stage zero-shot framework for Human-Object Interaction detection, named
Knowledge Integration to HOI (KI2HOI). We harness the capabilities of
foundational models to enhance the comprehension of complex interactive
semantics within visual data. Instead of using knowledge distillation on CLIP
for detectors, KI2HOI learns a priori knowledge from CLIP and integrates
compositional features extracted from multiple perspectives in HOI detection.
On the one hand, leveraging the off-the-shelf object detector, Ho-Pair Encoder
is designed to extract rich local and global visual features. On the other
hand, motivated by the Query2label [11], a transformer-based model utilized
for multi-label image classification, we extend to extract verb
representations based on visual semantics for HOI classification in
conjunction with CLIP, where each query is associated with a specific
interaction category in training and inference. Such queries adaptively focus
on relevant interactive regions to understand and represent each interaction,
yielding better intricate. Furthermore, to address the long-tail distribution
problem, we combine the verb features with interaction semantic
representations and our interaction representation decoder consists of a verb
representation head and a set of class weights computed via text semantics
algorithm. While traditional interaction classification heads work well in
handling visible interactions, they fall short in tackling invisible ones.
Thus, we combine the prompt-based verb classification head with the redesigned
verb representation head to enhance the HOI prediction.

To summarize, our contributions are as follows:

  * •

We propose a novel framework, named KI2HOI, for zero-shot HOI detection that
directly retrieves visual and linguistic knowledge of CLIP. Our KI2HOI
effectively utilizes a priori knowledge and achieves superior zero-shot
transferability.

  * •

We develop a verb feature extraction strategy to explicitly match verb queries
and interactions for more expressive intricate representations between verbs
and their associated interactions.

  * •

We conduct extensive experiments on HICO-Det for the zero-shot learning task
and perform additional comprehensive experiments on HICO-Det and V-COCO for
the supervised learning task. Our model outperforms the state-of-the-art
methods in zero-shot and full-supervised settings, establishing a new state-
of-the-art.

##  II RELATED WORKS

###  II-A Human-Object Interaction Detection

HOI detection methods can be roughly categorized as two-stage or one-stage
solutions. Two-stage methods [12, 13, 14, 15, 16, 17] first detect all
candidate interaction pairs and then feed to CNN network to predict the
interaction relationships between candidate human-object pairs. Most two-stage
detection models use existing object detection models and prioritize improving
interaction prediction models. FCL [18] proposed an object fabricator to
generate effective object representations, which were then combined with verbs
to compose new HOI samples, thus increasing the diversity of training data.
UPT [19] applied a unary-pairwise transformer to represent each target’s
instance details as unary and pairwise representations. In comparison to two-
stage methods, one-stage solutions [20, 21, 22, 9, 23, 24, 25, 26] captured
context information during the early stage of feature extraction, leading to
improved HOI detection performance. The success of DETR [27] has inspired many
researchers in studying HOI detection QPIC [28] applied additional detection
heads and relied on a bipartite graph matching algorithm to locate HOI
instances and identify interactions. EOID [9] developed a teacher-student
model and designed a two-stage Hungarian matching algorithm. RR-Net [26]
introduced a relation-aware frame to build progressive structure for
interaction inference, which imitates the human visual mechanism of
recognizing HOI by comprehending visual instances and interactions coherently.
HOICLIP [29] proposed a new transfer strategy that used visual semantic
algorithms to represent verbs. Our work belongs to a one-stage end-to-end
approach to study HOI detection.

###  II-B Vision-and-Language Pre-training

The advanced vision-and-language pre-training (VLP) multimodal learning
framework can acquire generalized multimodal representations from large-scale
image and text data [30]. It has a wide application in the fields of
multimodal retrieval [31, 32], visual and language navigation [33], image
description [34], and so on. Through effective cross-modal semantic alignment,
particularly fine-grained semantic alignment, VLP contributes to cross-modal
learning and generalization. Visual-language models succeed in enabling zero-
shot open-vocabulary tasks from natural language supervision [35], which
inspires us to apply visual-language models for zero-shot HOI detection tasks.

###  II-C Zero-shot HOI

Zero-shot HOI detection aims to generalize to unseen HOI categories during
training effectively. Since the majority of HOI exhibit a long-tail
distribution, attributed to the compositional nature of HOIs, prior research
[36, 18, 37, 38, 8, 23] on zero-shot HOI detection focuses on transferring
knowledge from known HOI concepts to unseen classes. They can be categorized
into three scenarios: unseen object, unseen action, and unseen combination.
There exist primarily two research streams for addressing this task. One
stream [18, 37, 38] employed a combined learning approach for zero-shot HOI
detection, which entailed separating HOI representations and combining known
features to identify unseen HOI concepts. ConsNet [37], for instance,
constructed a consistency graph with both visual features of potential human-
object pairs and word embeddings of HOI labels. With the advancement of
multimodal learning, there is a growing interest in transferring knowledge
from pre-trained visual language models, e.g. CLIP is used to extract text
embeddings of HOI descriptions for HOI detection tasks [8]. RLIP [23] proposed
a transferable HOI detector via natural language supervision. Building upon
GEN-VLKT [20], HOICLIP [29] mapped image and text encodings to a joint visual-
semantic space, to capture their correlations and effectively transfer
knowledge from CLIP. Our work seeks to explore a more effective framework to
make full use of CLIP for improving zero-shot HOI detection performance.

##  III METHODS

In this section, we introduce the details of the proposed framework to improve
its generalization ability in zero-shot HOI detection. In Section III-A, the
overall architecture is presented. In Section III-B, we introduce our visual
encoder and a strategy for enhancing global feature extraction. Section III-C
presents verb feature learning based on verb queries. In section III-D, we
design an interaction semantic representation for transferring knowledge to
HOI detection. Section III-E describes the training and inference procedures
of our model.

Figure 2: Overview of KI2HOI pipeline. It consists of four parts: visual
encoder, verb feature learning, instance interactor, and interaction semantic
representation (ISR). Given an image, firstly, we obtain the feature map
through the backbone and then use our dedicated visual encoder to extract
contextual global features. The instance interactor injects CLIP spatial
information and global features to locate human-object pairs and classify
object categories. In the verb feature learning module, associated verb
queries are fed to the verb extraction decoder to obtain fine-grained verb
features. The interaction semantic representation model inputs the verb
features and the interaction features from encoders to extract the interaction
representation.

###  III-A Overall Architecture

As shown in Figure 2, our model consists of four primary components: visual
encoder, verb feature learning, instance interactor, and interaction semantic
representation. Given an input image $I$, we initially extract feature maps
via the backbone DETR [27]. The feature maps are subsequently input into the
HO-pair Encoder to generate global visual feature $\bm{V}_{g}$, similar to
GEN-VLKT [20]. Human query $\bm{Q}_{h}$ and object query $\bm{Q}_{o}$ are
inputted into the instance interactor to compute the mean for both types of
queries in the corresponding decoder layer. These outputs queries in the last
decoder are then fed to classifiers which initialize by label’s text weights
from CLIP to predict the interacting human bounding box
$\bm{B}_{h}{\in}\mathbb{R}^{N\times 4}$ and object bounding box
$\bm{B}_{o}{\in}\mathbb{R}^{N\times 4}$, where $N$ is the number of queries,
and object class $\bm{{C}_{o}}{\in}\mathbb{R}^{N\times{C}}$, where $C$ denotes
the object category.

Furthermore, possible types of interactions depend on action classes. For
example, humans are more likely to catch or play sports ball than to bite a
sports ball. To reflect this characteristic of HOIs, our verb feature learning
module uses a novel verb extraction decoder to perform verb recognition, as
auxiliary information. We extract the spatial feature $\bm{V}_{sp}$ from the
pre-trained CLIP Visual Encoder as memory, feed into the interaction
representation decoder by the cross-attention mechanism to augment the
interaction representation and recognition. Finally, the HOI prediction
categories are generated by the output of a linear classifier. The details of
each component are explained in the following sections.

Figure 3: Structure of Ho-Pair Encoder. The local encoder is specifically
engineered to encode efficient local characteristics, followed by $3\times 3$
depth-wise convolution and two $1\times 1$ convolutions for channel blending.
The global context former is intended to capture comprehensive local-global
representations by extracting local features from the local convolutional
layers, an efficient additive attention module, and linear layers.

###  III-B Visual Encoder

Our Ho-Pair Encoder is designed to significantly enhance performance by
effectively addressing two critical design constraints inherent in
EfficientFormer [39]. The original EfficientFormer leverages 4D MetaBlocks,
inspired by PoolFormer, for efficient learning of local image representations
and 3D MetaBlocks, which utilize self-attention for capturing global context.
However, its performance is compromised by suboptimal token mixing and the
restricted use of 3D MetaBlocks in the final stage due to the quadratic
complexity of the multi-head self-attention (MHSA) mechanism, resulting in a
potentially inconsistent and inadequate representation of context. To surmount
these limitations, the Ho-Pair Encoder incorporates a Global Context Former
that significantly enhances token mixing. This modification is designed to
enhance the model’s ability to integrate and process visual information across
different scales and contexts, thereby providing a more robust and
comprehensive understanding of the input data. We resize the cropped feature
map obtained from backbone to a $7\times 7$ grid using ROI-Align and then feed
them into the Ho-Pair Encoder to acquire comprehensive high-level contextual
information, as depicted in Figure 3. The Ho-Pair Encoder comprises a local
encoder and a global context former. Specifically, the feature map
$\bm{V}_{F}$ is input to a $3\times 3$ depth-wise convolution (DWConv) with
batch normalization (BN). The results pass through two convolutional (Conv)
layers with GeLU activation. Subsequently, a skip connection is implemented to
facilitate the information flow throughout the entire network. The global
context former encoder consists of a $3\times 3$ DWConv and a $1\times 1$ Conv
to learn spatial information and encode local representations. The resulting
feature map is fed into an efficient additive attention block to learn
contextual information at each scale of the input size. Finally, the output
global features $\bm{V}_{G}$ is processed via a linear block, comprising two
$1\times 1$ Conv layers, a normalization layer, and GeLU activation. The Ho-
Pair Encoder is defined as:

$\displaystyle\bm{{V_{F}^{{}^{\prime}}}}=Conv_{1}(Conv_{1},DWConv_{3}(BN(\bm{V_{F}}))+\bm{V_{F}},$
(1)
$\displaystyle\bm{{V_{F}^{{}^{\prime}}}}=Conv_{1}(Conv_{1},DWConv_{3}(BN(\bm{V_{F}^{{}^{\prime}}})),$
(2)
$\displaystyle\bm{{V_{G}}}=Linear(QK({\bm{V_{F}^{{}^{\prime}}}})+{\bm{V_{F}^{{}^{\prime}}}}).$
(3)

Figure 4: Structure of interaction representation decoder. Interaction queries
$\bm{Q}_{inter}$, visual-spatial features $\bm{V}_{sp}$, and global visual
features$\bm{V}_{G}$ are passed through the multi-head cross-attention block
before being fed into the multi-head self-attention block. Then, the outputs
are concatenated with the verb features $\bm{V}_{verb}$ before being fed into
the feed-forward network.

###  III-C Verb Feature Learning

Inspired by the Query2Label [11], we establish a novel approach termed
learnable verb-specific query, where
$\bm{{Q}_{v}}{\in}\mathbb{R}^{N\times{A}}$, A is the number of interaction
categories. $\bm{{Q}_{v}}$ differs from traditional transformer queries by
establishing a one-to-one correspondence between each query and a particular
instance throughout the training and inference phases. Such queries interact
with global visual feature $\bm{{V}_{g}}$ through a verb extraction decoder
and become verb-specific queries. We design a self-attention and multi-head
attention combination module and a feed-forward network (FFN) layer as verb
extraction decoder, consisting of two layers. A set of verb queries
$\bm{{Q}_{v}}$ aggregates the global feature information $\bm{{V}_{G}}$
through verb extraction decoder and is updated to ${\bm{V}_{verb}}$. In this
way, the queries are learned to capture the verb priors and become good
feature representations for these interactions. namely:

$\displaystyle\hat{\bm{V}_{G}}=\text{selfAtten}(\hat{\bm{V}_{G}}),$ (4)
$\displaystyle\bm{V}_{verb}=\text{MultiheadAttn}(\hat{\bm{V}_{G}},\bm{Q}_{V}),$
(5) $\displaystyle\bm{V}_{verb}=\text{FFN}(\bm{V}_{verb}).$ (6)

###  III-D Interaction Semantic Representation (ISR)

Conventional HOI detection uses an Interaction Classification Head to predict
the confidence of each action for each pair of interactions and to judge the
interaction. If the training sample for one of the actions is assumed to be
too few or non-existent, the prediction for that action will be very
inaccurate. This situation is more obvious in the zero-shot learning tasks,
which is one of the reasons why traditional methods fail in the field of zero-
shot learning. HOI detector maps visual features and tag-generated text
features into the same space through CLIP, which performs quite well in the
zero-shot learning tasks. We design the HOI labels and object labels are
represented as “A photo of a person [verb] a/an [object]” and “A photo of a/an
[object]” to assign different tokens to each HOI instance. To fully explore
the CLIP knowledge, we propose to retrieve the text embeddings from the CLIP
to better align them with the prior knowledge in the classifier weights.

Interaction Representation Decoder. We introduce an interaction semantic
representation (ISR) module to extract the interactive representations of
human-object interaction pairs. First, We add a learnable position guided
embedding $\bm{P}{\in}\mathbb{R}^{N\times C}$ for human queries and object
queries, viz., $\bm{Q}_{h}{\in}\mathbb{R}^{N\times C}$ and
$\bm{Q}_{o}{\in}\mathbb{R}^{N\times C}$ at the same position as interaction
pairs. We compute the interaction queries
$\bm{Q}_{inter}{\in}\mathbb{R}^{N\times C}$ by taking the average of the
concatenation of $\bm{Q}_{h}$ and $\bm{Q}_{o}$. That is:

$\displaystyle\bm{Q}_{inter}=\text{Cat}(\bm{Q}_{h}+\bm{P},\bm{Q}_{o}+\bm{P})/2.$
(7)

To guide interaction queries $\bm{Q}_{inter}$ to explore informative regions
in both $\bm{V}_{G}$ and $\bm{V}_{sp}$, we design an interaction
representation decoder with multiple cross-attention, as shown in Figure 4.
Each decoder consists of an attention block, a self-attention block, and a
forward feedback network. $\bm{Q}_{inter}$ is first input to the self-
attention block and the corresponding output is fed into the cross-attention
mechanism with $\bm{V}_{G}$ and $\bm{V}_{sp}$. Subsequently, the final output
is:

$\displaystyle\bm{Q}_{inter}=\text{MHSA}(\bm{Q}_{inter}),$ (8)
$\displaystyle\bm{Q}_{inter}^{{}^{\prime}}=\text{MHCA}(\bm{Q}_{inter},\bm{V}_{sp}),$
(9)
$\displaystyle\bm{Q}_{inter^{{}^{\prime\prime}}}=\text{MHCA}(\bm{Q}_{inter},\bm{V}_{G}),$
(10)
$\displaystyle\bm{Q}_{inter}=\text{FFN}(\bm{Q}_{inter}^{{}^{\prime}}+\bm{Q}_{inter}^{{}^{\prime\prime}}),$
(11)

where MHSA denotes a multi-head self-attention operation and MHCA denotes a
multi-head cross-attention operation. Since, the obtained
$\bm{Q}_{inter}{\in}\mathbb{R}^{N_{q}\times C}$ integrates the knowledge of
CLIP and visual features, enabling the detection of fine-grained HOI.

By leveraging object and human information from the instance Interactor, we
can concatenate interaction representations from the spatial feature map of
CLIP and visual features from the detector to efficiently retrieve
corresponding interaction representations and achieve strong generalization
capabilities.

Verb Predictor via Knowledge Retrieval. To align verb features $V_{verb}$ with
$\bm{Q}_{inter}$, we employ projection operation to map both $\bm{Q}_{inter}$
and $\bm{V}_{verb}$ into the CLIP feature space. After projection, multi-layer
perceptron (MLP) is applied to extract $\bm{D}_{verb}$ from $\bm{Q}_{inter}$
and $\bm{C}_{verb}$ from $\bm{V}_{verb}$ from $\bm{Q}_{inter}$ and
$\bm{V}_{verb}$. Finally, the verb score is obtained as:

$\displaystyle\bm{Q}_{inter}=\text{Proj}(\bm{Q}_{inter}),\bm{V}_{verb}=\text{Proj}(\bm{V}_{verb}),$
(12)
$\displaystyle\bm{C}_{verb}=\text{MLP}(\bm{V}_{verb}),\bm{D}_{verb}=\text{MLP}(\bm{Q}_{inter}),$
(13) $\displaystyle\bm{C}_{verb}=\text{Cat}(\bm{C}_{verb},\bm{D}_{verb}),$
(14) $\displaystyle\bm{S}_{verb}=\bm{C}_{verb}\bm{W^{T}}_{v},$ (15)

where the verb score is computed by the cosine similarity between the verb
features and the text weight of the verb representations. As well, a
reconstruction loss function that quantifies the dissimilarity between
features. That is:

$\displaystyle\bm{L}_{re}=L_{1}(\bm{Q}_{inter},\bm{V}_{sp}),$ (16)

where $L1$ loss is used to minimize the distance between features and visual
embeddings.

TABLE I: Performance comparison for zero-shot HOI detection on HICO-DET. RF-UC
indicates rare first setting, and NF-UC represents non-rare first unseen
combination setting. UC, UO, and UV denote unseen composition, unseen object,
and unseen verb settings, respectively.

Method | Type | Full | Seen | Unseen
---|---|---|---|---
VCL [40] | RF-UC | 21.43 | 24.28 | 10.06
ATL [38] | RF-UC | 21.57 | 24.67 | 9.18
FCL [18] | RF-UC | 22.01 | 24.23 | 13.16
SCL [41] | RF-UC | 28.08 | 30.39 | 19.07
RLIP [23] | RF-UC | 30.52 | 33.35 | 19.19
EoID [9] | RF-UC | 29.52 | 31.39 | 22.04
GEN-VLKT [20] | RF-UC | 30.56 | 32.91 | 21.36
HOICLIP [29] | RF-UC | 32.99 | 34.85 | 25.53
KI2HOI | RF-UC | 34.10 | 35.79 | 26.33
VCL [40] | NF-UC | 16.22 | 18.52 | 18.06
ATL [38] | NF-UC | 18.25 | 18.78 | 18.67
FCL [18] | NF-UC | 18.66 | 19.55 | 19.37
SCL [41] | NF-UC | 24.34 | 25.00 | 21.73
RLIP [23] | NF-UC | 26.19 | 27.67 | 20.27
GEN-VLKT [20] | NF-UC | 23.71 | 23.38 | 25.05
EoID [9] | NF-UC | 26.69 | 26.66 | 26.77
HOICLIP [29] | NF-UC | 27.75 | 28.10 | 26.39
KI2HOI | NF-UC | 27.77 | 28.31 | 28.89
FG [42] | UC | 12.26 | 12.60 | 10.93
ConsNet [37] | UC | 19.81 | 20.51 | 16.99
EoID [9] | UC | 28.91 | 30.39 | 23.01
HOICLIP [29] | UC | 32.99 | 34.85 | 25.53
KI2HOI | UC | 34.56 | 35.76 | 27.43
FCL [18] | UO | 19.87 | 20.74 | 15.54
ATL [38] | UO | 20.47 | 21.54 | 15.11
GEN-VLKT [20] | UO | 25.63 | 28.92 | 10.51
HOICLIP [29] | UO | 28.53 | 30.99 | 16.20
KI2HOI | UO | 28.84 | 31.70 | 16.50
ConsNet [37] | UV | 19.04 | 20.02 | 14.12
GEN-VLKT [20] | UV | 28.74 | 30.23 | 20.96
EoID [9] | UV | 29.61 | 30.73 | 22.71
HOICLIP [29] | UV | 31.09 | 32.19 | 24.30
KI2HOI | UV | 31.85 | 32.95 | 25.20

TABLE II: Comparison with state-of-the-art methods on HICODET and V-COCO. All methods employ the ResNet-50 backbone network. Method | Detector | Backbone | mAP Default | mAP Know Object | V-COCO
---|---|---|---|---|---
Non-Rare | Full | Rare | Non-Rare | Full | Rare | $AP_{role}^{S1}$ | $AP_{role}^{S2}$
IDN [43] | COCO | ResNet-50 | 23.36 | 22.47 | 23.63 | 26.43 | 25.01 | 26.85 | 53.3 | 60.3
HOTR [21] | HICO-Det | ResNet-50 | 25.10 | 17.34 | 27.42 | - | - | - | 55.2 | 64.4
ATL [38] | COCO | ResNet-50 | 28.53 | 21.64 | 30.59 | 31.18 | 24.15 | 33.29 | - | -
QPIC [28] | HICO-Det | ResNet-50 | 29.07 | 21.85 | 31.23 | 31.68 | 24.14 | 33.93 | 58.8 | 61.0
FCL [18] | COCO | ResNet-50 | 29.12 | 23.67 | 30.75 | 31.31 | 25.62 | 33.02 | 52.4 | -
SCG [44] | COCO | ResNet-50-FPN | 31.33 | 24.72 | 33.31 | 34.37 | 27.18 | 36.52 | 54.2 | 60.9
UPT [19] | COCO | ResNet-50 | 31.66 | 25.94 | 33.36 | 35.05 | 29.27 | 36.77 | 59.0 | 64.5
CDN [45] | HICO-DET | ResNet-50 | 31.78 | 27.55 | 33.05 | 34.53 | 29.73 | 35.96 | 62.3 | 64.4
Iwin [46] | HICO-Det | ResNet-50-FPN | 32.03 | 27.62 | 34.14 | 35.17 | 28.79 | 35.91 | 60.5 | -
[22] | COCO | ResNet-50 | 33.51 | 30.30 | 34.46 | 36.28 | 33.16 | 37.21 | 63.0 | 65.2
GEN-VLKT [20] | HICO-Det | ResNet-50+ViT-B | 33.75 | 29.25 | 35.10 | 36.78 | 32.75 | 37.99 | 62.4 | 64.4
ADA-CM [47] | COCO | ResNet-50+ViT-B | 33.80 | 31.72 | 34.42 | - | - | - | 56.1 | 61.5
OpenCat [48] | - | ResNet-50 | 32.68 | 28.42 | 33.75 | - | - | - | 61.9 | 63.2
HOICLIP [29] | HICO-Det | ResNet-50+ViT-B | 34.69 | 31.12 | 35.74 | 37.61 | 34.47 | 38.54 | 63.5 | 64.8
KI2HOI | HICO-Det | ResNet-50+ViT-B | 34.20 | 32.26 | 36.10 | 37.85 | 35.89 | 38.78 | 63.9 | 65.0
TABLE III: Robustness to different distribution data. | Non-Rare HICO-DET
---|---
Percentage | 100$\%$ | 50$\%$ | 25$\%$ | 15$\%$
GEN-VLKT | 33.75 | 26.55 | 22.14 | 20.40
KI2HOI | 34.20 | 31.25 | 30.06 | 27.20
mAP gain($\%$) | +1.3 | +19.2 | +35.77 | +33.3
| Rare HICO-DET
Percentage | 100$\%$ | 50$\%$ | 25$\%$ | 15$\%$
GEN-VLKT | 29.25 | 18.94 | 14.04 | 13.84
KI2HOI | 36.10 | 26.68 | 25.05 | 22.51
mAP gain($\%$) | +23.41 | +40.86 | +78.41 | +38.51
TABLE IV: Network architecture analysis. Ablation studies are conducted on HICO-DET under the Unseen Verb (UV) setting. Method | Full | Seen | Unseen
---|---|---|---
Baseline | 28.20 | 30.49 | 9.57
+CLIP | 30.45 | 31.65 | 23.37
+Ho-Pair Encoder | 30.99 | 32.02 | 23.96
+Verb Feature Learning | 31.85 | 32.95 | 25.20
TABLE V: Reconstruction loss setting. $L_{1}$ | $L_{2}$ | Full | Rare | Non-Rare
---|---|---|---|---
- | - | 33.31 | 31.83 | 35.05
✓ | - | 34.20 | 32.26 | 36.10
- | ✓ | 34.06 | 30.65 | 35.08
✓ | ✓ | 34.08 | 30.18 | 35.24
TABLE VI: The impact of different verb extraction decoder layer numbers. Layers | Full | Seen | Unseen
---|---|---|---
1 | 27.77 | 28.31 | 28.89
2 | 26.30 | 26.70 | 24.71
3 | 26.53 | 26.70 | 25.07

###  III-E Training and Inference

Training. We employ the Hungarian algorithm [9, 20, 29] for bipartite matching
between predictions and ground truths. The matching cost consists of human
bounding box regression loss $\bm{L}_{bh}$, object bounding box regression
loss $\bm{L}_{bo}$, object classification loss $\bm{L}_{o}$, interaction-over-
union loss $\bm{L}_{u}$, and interaction classification loss $L_{i}$. Combined
with the reconstruction loss function, the final loss function is as follows:

$\displaystyle\bm{L}=\lambda_{\alpha}\bm{L}_{b}+\lambda_{\beta}\bm{L}_{u}+\lambda_{\delta}\bm{L}_{i}+\lambda_{re}\bm{L}_{re},$
(17)

where $\lambda_{re}$ is weighting parameter, $\lambda_{\alpha}$,
$\lambda_{\beta}$, $\lambda_{\delta}$ are hyper-parameters for adjusting the
weights of all losses, $\bm{L}_{b}=\bm{L}_{bh}+\bm{L}_{bo}$.

Inference. Reconstruction loss is only used for training,
$\bm{S}_{h}\in[0,1]^{N}$, $\bm{S}_{o}\in[0,1]^{N}$. In inference, the final
score $\bm{S}_{final}\in[0,1]^{N}$ is summed by $\bm{S}_{h}$, $\bm{S}_{O}$ and
$\bm{S}_{verb}$.

$\displaystyle\bm{S}_{final}^{i}=\bm{S}_{h}+\bm{S}_{o}+\bm{S}_{verb},i\in[1,c].$
(18)

Figure 5: Visualization of the HOI detection results. From left to right,
column 1: HOI prediction results; column 2: attention maps from Verb
Extraction Decoder; column 3: attention maps from Interaction Representation
Decoder. Images are sampled from the HICO-DET dataset in UV test set.

##  IV EXPERIMENTS

###  IV-A Experimental Setup

person instr snowboard

One VS One

person row boat

person ride  motorcycle

person touch horse

person ride bicycle

person hold umbrella

One VS Many

person hold phone

person brush toothbrush

person scratch hair

person brush toothbrush

person sit boat

person hold umbrella

person(1) hold baseball

person(1) look person(2)

Many VS One

person ride horse

person lasso cow

person catch kite

person fly kite

person ride horse

person lasso cow

person look baseball

person(1) hold baseball

Many VS Many

person hold baseball

person dribble baseball

person(1) ride  motorcycle

person(2) ride motorcycle

person(1) ride horse

person(2) ride horse

Figure 6: Visualization of different interaction relationships. One VS One:
person interact with a single objects; One VS Many: person interact with
multiple objects; Many VS One: person interact with a single object; Many VS
Many: person interact with multiple objects.

Datasets. We evaluate our model on two widely-uesd benchmarks, HICO-DET [49]
and V-COCO [50]. HICO-Det contains 47,776 images, of which 38,118 and 9658
images are used for training. It includes 600 HOI triplets, where 138 triplets
are rare categories less than 10 training instances, and the remaining 462
categories are non-rare. HICO-Det also provides a zero-shot detection setting
by holding out 120 rare interactions. V-COCO is a subset of the COCO dataset
and consists of 10,396 images, with 5,400 for training and 4,964 for testing.
It has 29 action categories, including 4 annotations without any interaction
with objects.

Zero-shot Data Setups. We conduct experiments on the HICO-Det for zero-shot
HOI detection, mainly using the following approaches: Rare First Unseen
Combination (RF-UC), Non-rare First Unseen Combination (NF-UC), Unseen Verb
(UV), Unseen Object (UO) and Unseen Combination(UC). In the UC setting, the
training data includes all categories of objects and verbs but lacks some HOI
triplet categories. We assess 120 unseen categories and 480 seen categories
out of a total of 600 categories. The RF-UC selects the tail HOIs as unseen
categories, whereas the NF-UC prefers head categories. In the UO setting, we
select 12 unseen objects out of a total of 80 objects to define unseen HOIs.
Additionally, we introduce a UV setting, where 20 verbs from a total of 117
verbs are randomly selected to construct 84 unseen and 516 seen HOIs.

Evaluation Metric. We evaluate our model using the mean Average Precision
(mAP) as metric, a prediction is considered as true positive if the predicted
human and object bounding boxes have an IoU of at least 0.5 with the ground
truth, and the predicted interaction category matches the correct category. We
report the standard mAP for HOI detection, dividing interactions into non-
rare, rare, and unseen cases based on their occurrences in the training set.
The mAP measures HOI detection performance with a threshold of 0.5 for the IoU
between predicted and ground-truth bounding boxes.

Implementation Details. We use pre-trained DETR with ResNet-50 as the backbone
network. The visual encoder is based on ViT-32/B CLIP, and during training,
the parameters of CLIP remain unchanged. Our model’s encoder and decoder have
3 layers with 64 queries, except the verb extraction decoder has 1 layer. We
train the model using the AdamW optimizer with a specified interval for
adjusting the learning rate. The training epochs is set to 90, with a gradual
decrease in the learning rate after the 60th epoch. All experiments are
conducted with a batch size of 8 on 4 NVIDIA A6000 GPUs.

###  IV-B Effectiveness for HOI Detection

Zero-shot Detection. We conduct various zero-shot setting experiments on the
HICO-DET dataset, and the results are shown in Table I. Our model outperforms
existing state-of-the-art methods, demonstrating strong performance
competitiveness. Specifically, under the RF-UC and NF-UC settings, our model’s
relative mean average precision (mAP) for unseen categories exceeds that of
EoID [9] by 23.26$\%$ and 7.91$\%$, respectively. In the UA setting, our model
outperforms the latest work, HOICLIP [29], with a 1.03 mAP improvement for
unseen types. In both full and unseen categories, our model achieves 1.07 and
2.5 mAP improvements relative to EOID [9]. In the case of unseen objects, our
model’s performance surpasses that of GEN-VLKT [20] by 3.21 mAP.

Fully Supervised Detection. To verify the generalization ability of the model,
we conducted fully supervised experiments on HICO-DET and V-COCO. As tabulated
in Table II, our model achieves remarkable performance, exceeding GEN-VLKT and
HOICLIP [29] by 3.01 mAP and 1.14 mAP for full categories, and by 1 mAP and
0.36 mAP for rare categories. This indicates that our model can handle the
long-tailed distribution of HOI well. For the V-COCO, we achieve 63.9 role AP
on Scenario 1 and 65.0 role AP on Scenario 2 surpassing previous methods.

Robustness to Distributed Data. We investigate the robustness of the proposal
under different data quantities. We decrease the proportion of training data
from 100$\%$ to 15$\%$, and we seek to achieve less performance loss in HOI
detection. Compared to the state-of-the-art GEN-VLKT in Table III, the
proposed method achieves competitive performance in detecting both non-rare
and rare categories. Furthermore, Table III highlights the improvements
achieved by our model at various volumes of data. At 25$\%$ training data, our
model exhibits a 78.41$\%$ increase in mAP gain for rare HICO-DET.

###  IV-C Ablation Studies

Network Architecture Analysis. We conduct ablation experiments for each module
on the HICO-DET dataset under the UV setting and the results are shown in
Table IV. The GEN-VLKT without the knowledge distillation component.is
regarded as the baseline. First, we examine the effect of CLIP and the result
shows a 13.8 mAP improvement in HOI detection for Unseen categories. Thus, the
visual and linguistic knowledge extracted by CLIP enable to learn deeper
interaction understanding. Next, we replace the encoder with the proposed Ho-
Pair Encoder for using fine-grained visual features. As a result, we observe
that the results come up to 30.99 mAP and 32.02 mAP in the Full and Seen
categories, respectively. Finally, we add a verb feature learning to capture
verb-related features, and the performance is further advanced to 31.85 mAP in
full categories, which demonstrates the necessity of verb feature learning in
the zero-shot HOI task.

Reconstruction Loss Setting. As shown in Table V, we compare two loss types,
i.e., $L_{1}$ loss and $L_{2}$ loss, as the reconstruction loss. It
demonstrates that the reconstruction loss is indispensable for making
knowledge transfer effective. If exclusively employing $L_{1}$ loss, our model
has demonstrated superior performance and achieved 1.05 mAP gain, surpassing
the performance exhibited by only employing the $L_{2}$ loss. When combining
$L_{1}$ and $L_{2}$ losses through average summation, the performance is still
inferior compared to only applying $L_{1}$ loss.

Verb Extraction Decoder Layer Selection. We examine the effect of verb
extraction decoder layer number on the verb features update and conduct
experiments in the NF-UC setting. In Table VI, we find that a single layer
exhibits superior performance. While increasing the number of layers, there is
no consistent improvement.

###  IV-D Qualitative Visualization Results

As shown in Figure 5, we illustrate the characteristics of our model by
visualizing the attention feature maps of the decoder. Our framework can
effectively infer human-object interaction relationships in unseen interaction
categories. We observe the attention maps from the Interaction Representation
Decoder focus on a broader range of interaction-related regions, while the
Verb Extraction Decoder mainly emphasizes the interaction target area.
Additionally, taking the detection of “wash-bus“, as an example, it indicates
that our model is capable of perceiving interactions between indirectly
connected humans and objects. In Figure 6, we note that our model has the
capability to detect various types of interactions.

##  V CONCLUSION

The present paper introduces a novel one-stage framework, known as KI2HOI, for
zero-shot HOI detection. KI2HOI effectively integrates visual-linguistic
knowledge from CLIP to augment the interaction representation and recognition
unknown instances of HOIs. We also introduce Ho-Pair Encoder to generate
contextual spatial features via additive attention mechanism and extract verb
features from associated queries. To enhance the understanding of
interactions, we adopt a novel interaction representation decoder via text
embeddings of CLIP. Extensive experimental results demonstrate the
effectiveness of the proposed KI2HOI framework.

##  VI CONFLICT OF INTEREST STATEMENT

We declare no competing financial interests or personal relationships that
could have influenced the work presented in this paper.

## References

  * [1] M. Antoun and D. Asmar, “Human object interaction detection: Design and survey,” _Image and Vision Computing_ , p. 104617, 2022.
  * [2] J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. Shamma, M. Bernstein, and L. Fei-Fei, “Image retrieval using scene graphs,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3668–3678.
  * [3] L. Chen, X. Yan, J. Xiao, H. Zhang, S. Pu, and Y. Zhuang, “Counterfactual samples synthesizing for robust visual question answering,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 10 800–10 809.
  * [4] J. Liu and Q. Liu, “R3CD: Scene Graph to Image Generation with relation-aware Compositional Contrastive Control Diffusion,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2024.
  * [5] Z. Fu, C. Zheng, J. Feng, Y. Cai, X.-Y. Wei, Y. Wang, and Q. Li, “Drake: Deep pair-wise relation alignment for knowledge-enhanced multimodal scene graph generation in social media posts,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 33, no. 7, pp. 3199–3213, 2023.
  * [6] Y. Du, F. Wei, Z. Zhang, M. Shi, Y. Gao, and G. Li, “Learning to prompt for open-vocabulary object detection with vision-language model,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 14 084–14 093.
  * [7] D. Kim, A. Angelova, and W. Kuo, “Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers,” _CoRR_ , vol. abs/2305.07011, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2305.07011
  * [8] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [9] M. Wu, J. Gu, Y. Shen, M. Lin, C. Chen, and X. Sun, “End-to-end zero-shot hoi detection via vision and language knowledge distillation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 37, no. 3, 2023, pp. 2839–2846.
  * [10] X. Qu, C. Ding, X. Li, X. Zhong, and D. Tao, “Distillation using oracle queries for transformer-based human-object interaction detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 19 558–19 567.
  * [11] S. Liu, L. Zhang, X. Yang, H. Su, and J. Zhu, “Query2label: A simple transformer way to multi-label classification,” _arXiv preprint arXiv:2107.10834_ , 2021.
  * [12] B. Wan, Y. Liu, D. Zhou, T. Tuytelaars, and X. He, “Weakly-supervised HOI Detection via Prior-guided Bi-level Representation Learning,” _arXiv preprint arXiv:2303.01313_ , 2023.
  * [13] Z. Li, C. Zou, Y. Zhao, B. Li, and S. Zhong, “Improving human-object interaction detection via phrase learning and label composition,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 2, 2022, pp. 1509–1517.
  * [14] Y. Liu, Q. Chen, and A. Zisserman, “Amplifying key cues for human-object-interaction detection,” in _European Conference on Computer Vision_. Springer, 2020, pp. 248–265.
  * [15] Y. Cao, Q. Tang, F. Yang, X. Su, S. You, X. Lu, and C. Xu, “Re-mine, learn and reason: Exploring the cross-modal semantic correlations for language-guided hoi detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 23 492–23 503.
  * [16] Y. Cheng, Z. Wang, W. Zhan, and H. Duan, “Multi-scale human-object interaction detector,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 33, no. 4, pp. 1827–1838, 2023.
  * [17] Y. Wang, Q. Liu, and Y. Lei, “Ted-net: Dispersal attention for perceiving interaction region in indirectly-contact hoi detection,” _IEEE Transactions on Circuits and Systems for Video Technology_ , pp. 1–1, 2024.
  * [18] Z. Hou, B. Yu, Y. Qiao, X. Peng, and D. Tao, “Detecting human-object interaction via fabricated compositional learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 646–14 655.
  * [19] F. Z. Zhang, D. Campbell, and S. Gould, “Efficient two-stage detection of human-object interactions with a novel unary-pairwise transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 20 104–20 112.
  * [20] Y. Liao, A. Zhang, M. Lu, Y. Wang, X. Li, and S. Liu, “GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 20 091–20 100.
  * [21] B. Kim, J. Lee, J. Kang, E.-S. Kim, and H. J. Kim, “Hotr: End-to-end human-object interaction detection with transformers,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 74–83.
  * [22] X. Liu, Y.-L. Li, X. Wu, Y.-W. Tai, C. Lu, and C.-K. Tang, “Interactiveness field in human-object interactions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 20 113–20 122.
  * [23] H. Yuan, J. Jiang, S. Albanie, T. Feng, Z. Huang, D. Ni, and M. Tang, “Rlip: Relational language-image pre-training for human-object interaction detection,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 37 416–37 431, 2022.
  * [24] S. Kim, D. Jung, and M. Cho, “Relational Context Learning for Human-Object Interaction Detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 2925–2934.
  * [25] D. Zhou, Z. Liu, J. Wang, L. Wang, T. Hu, E. Ding, and J. Wang, “Human-object interaction detection via disentangled transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 19 568–19 577.
  * [26] D. Yang, Y. Zou, C. Zhang, M. Cao, and J. Chen, “Rr-net: Relation reasoning for end-to-end human-object interaction detection,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 6, pp. 3853–3865, 2022.
  * [27] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European conference on computer vision_. Springer, 2020, pp. 213–229.
  * [28] M. Tamura, H. Ohashi, and T. Yoshinaga, “Qpic: Query-based pairwise human-object interaction detection with image-wide contextual information,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 10 410–10 419.
  * [29] S. Ning, L. Qiu, Y. Liu, and X. He, “HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models,” in _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023, pp. 23 507–23 517.
  * [30] L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang _et al._ , “Grounded language-image pre-training,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 965–10 975.
  * [31] M. Dzabraev, M. Kalashnikov, S. Komkov, and A. Petiushko, “Mdmmt: Multidomain multimodal transformer for video retrieval,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3354–3363.
  * [32] J. Li, W. K. Wong, L. Jiang, X. Fang, S. Xie, and Y. Xu, “Ckdh: Clip-based knowledge distillation hashing for cross-modal retrieval,” _IEEE Transactions on Circuits and Systems for Video Technology_ , pp. 1–1, 2024.
  * [33] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3674–3683.
  * [34] W. Jin, Y. Cheng, Y. Shen, W. Chen, and X. Ren, “A good prompt is worth millions of parameters: Low-resource prompt-based learning for vision-language models,” _arXiv preprint arXiv:2110.08484_ , 2021.
  * [35] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui, “Open-vocabulary object detection via vision and language knowledge distillation,” _arXiv preprint arXiv:2104.13921_ , 2021.
  * [36] J. Peyre, J. Sivic, I. Laptev, and C. Schmid, “Detecting unseen visual relations using analogies,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 1981–1990.
  * [37] Y. Liu, J. Yuan, and C. W. Chen, “Consnet: Learning consistency graph for zero-shot human-object interaction detection,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 4235–4243.
  * [38] Z. Hou, B. Yu, Y. Qiao, X. Peng, and D. Tao, “Affordance transfer learning for human-object interaction detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 495–504.
  * [39] Y. Li, G. Yuan, Y. Wen, J. Hu, G. Evangelidis, S. Tulyakov, Y. Wang, and J. Ren, “Efficientformer: Vision transformers at mobilenet speed,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 12 934–12 949, 2022.
  * [40] Z. Hou, X. Peng, Y. Qiao, and D. Tao, “Visual compositional learning for human-object interaction detection,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16_. Springer, 2020, pp. 584–600.
  * [41] Z. Hou, B. Yu, and D. Tao, “Discovering human-object interaction concepts via self-compositional learning,” in _European Conference on Computer Vision_. Springer, 2022, pp. 461–478.
  * [42] A. Bansal, S. S. Rambhatla, A. Shrivastava, and R. Chellappa, “Detecting human-object interactions via functional generalization,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 07, 2020, pp. 10 460–10 469.
  * [43] Y.-L. Li, X. Liu, X. Wu, Y. Li, and C. Lu, “Hoi analysis: Integrating and decomposing human-object interaction,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 5011–5022, 2020.
  * [44] F. Z. Zhang, D. Campbell, and S. Gould, “Spatially conditioned graphs for detecting human-object interactions,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 13 319–13 327.
  * [45] A. Zhang, Y. Liao, S. Liu, M. Lu, Y. Wang, C. Gao, and X. Li, “Mining the benefits of two-stage and one-stage hoi detection,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 17 209–17 220, 2021.
  * [46] D. Tu, X. Min, H. Duan, G. Guo, G. Zhai, and W. Shen, “Iwin: Human-object interaction detection via transformer with irregular windows,” in _European Conference on Computer Vision_. Springer, 2022, pp. 87–103.
  * [47] T. Lei, F. Caba, Q. Chen, H. Jin, Y. Peng, and Y. Liu, “Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memory,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 6480–6490.
  * [48] S. Zheng, B. Xu, and Q. Jin, “Open-category Human-Object Interaction Pre-Training via Language Modeling Framework,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 19 392–19 402.
  * [49] Y.-W. Chao, Y. Liu, X. Liu, H. Zeng, and J. Deng, “Learning to detect human-object interactions,” in _2018 ieee winter conference on applications of computer vision (wacv)_. IEEE, 2018, pp. 381–389.
  * [50] S. Gupta and J. Malik, “Visual semantic role labeling,” _arXiv preprint arXiv:1505.04474_ , 2015.

|  Weiying Xue is currently pursuing an M.D degree at the School of Future
Technology, South China University of Technology (SCUT), China. Her current
research interests include human-object interaction, including human-object
interaction detection, visual relation detection, object detection, etc.
---|---
|  Qi Liu is currently a Professor with the School of Future Technology at
South China University of Technology. Dr. Liu received the Ph.D degree in
Electrical Engineering from City University of Hong Kong, Hong Kong, China, in
2019. During 2018 - 2019, he was a Visiting Scholar at University of
California Davis, CA, USA. From 2019 to 2022, he worked as a Research Fellow
in the Department of Electrical and Computer Engineering, National University
of Singapore, Singapore. His research interests include human-object
interaction, AIGC, 3D scene reconstruction, and affective computing, etc. Dr.
Liu has been an Associate Editor of the IEEE Systems Journal (2022-), and
Digital Signal Processing (2022-). He was also Guest Editor for the IEEE
Internet of Things Journal, IET Signal Processing, etc. He was the recipient
of the Best Paper Award of IEEE ICSIDP in 2019.
---|---
|  Qiwei Xiong received the M.E. degree from the University of Technology
Sydney, Sydney, Australia, in 2021. He is currently pursuing the Ph.D.degree
with electronic information from South China University of Technology,
Guangzhou, China. His main research interests include Human-computer
interaction, loT sensing and mobile computing.
---|---
|  Yuxiao Wang is currently pursuing a Ph.D degree at the School of Future
Technology, South China University of Technology (SCUT), China. His research
focuses on human-object interaction, including human-object interaction
detection, human-object contact detection, semantic segmentation, crowd
counting, etc. In addition, he has conducted research on weakly supervised
learning methods.
---|---
|  Zhenao Wei received the D.Eng. degree from the Graduate School of
Information Science and Engineering, Ritsumeikan University, Japan. He is
currently a postdoctoral fellow at South China University of Technology. His
research interests include game AIs, human-object interaction, and human-
object contact.
---|---
|  Xiaofen Xing is currently an Associate Professor with the School of Future
Technology at the South China University of Technology. Dr. Xing participated
in the key project of the National Natural Science Foundation Joint Fund
(Brain-like Affective Computing(2019-2022). In recent years, she has focused
on deep learning-based cross-media emotion computing, covering images,
expressions, speech, natural language understanding, and 3D virtual avatar
interaction. It has undertaken 1 youth project of the National Natural Science
Foundation and 3 science and technology plan projects such as the special
research and development project of artificial Intelligence in Guangdong
Province. She has published more than 20 articles in well-known journal
conferences in the field of artificial intelligence, won the best paper
nomination award of the well-known image processing conference ICIP, and was
shortlisted for the authoritative Emotion competition paper report of ACM MM
twice.
---|---
|  Xiangmin Xu is the Second-Grade Professor of South China University of
Technology, Doctoral Supervisor, Vice President of South China University of
Technology, Dean of the school of Future Technology, Member of Electronic
Information Teaching Steering Committee of Ministry of Education, Director of
Engineering Research Center of Ministry of Education on Human Body Data
Perception. His recent research focuses on computer vision, natural
interaction, affective computing and so on, and the doctoral student he
trained won the Excellent Doctoral Dissertation of the Chinese Society of
Electronics in 2020. He has undertaken more than 20 projects, published more
than 60 articles in top journals and conferences, including 1 ESI hot paper
with high citation, 50 invention patents were authorized, and the research
achievements of Computational Psychophysiology were granted with the 2nd class
of National Technology Invention Award in 2018. He has published 7 papers in
important educational journals and been awarded the 2nd class of the National
Teaching Achievements Award for three consecutive years.
---|---