# Full-Duplex-Enabled Joint Communications and Sensing with Reconfigurable
Intelligent Surfaces

Chandan Kumar Sheemar1  George C. Alexandropoulos2  Dirk Slock3  Jorge Querol1
and Symeon Chatzinotas1
1SnT  University of Luxembourg  emails: {chandankumar.sheemar  jorge.querol
<EMAIL_ADDRESS>
2 National and Kapodistrian University of Athens  Greece
<EMAIL_ADDRESS>
3EURECOM  Sophia Antipolis  France<EMAIL_ADDRESS>
###### Abstract

The full-duplex (FD) technology has the potential to radically evolve wireless
systems, facilitating the integration of both communications and radar
functionalities into a single device, thus, enabling joint communication and
sensing (JCAS). In this paper, we present a novel approach for JCAS that
incorporates a reconfigurable intelligent surface (RIS) in the near-field of
an FD multiple-input multiple-output (MIMO) node, which is jointly optimized
with the digital beamformers to enable JSAC and efficiently handle self-
interference (SI). We propose a novel problem formulation for FD MIMO JCAS
systems to jointly minimize the total received power at the FD node’s radar
receiver, while maximizing the sum rate of downlink communications subject to
a Cramér-Rao bound (CRB) constraint. In contrast to the typically used CRB in
the relevant literature, we derive a novel, more accurate, target estimation
bound that fully takes into account the RIS deployment. The considered problem
is solved using alternating optimization, which is guaranteed to converge to a
local optimum. The simulation results demonstrate that the proposed scheme
achieves significant performance improvement both for communications and
sensing. It is showcased that, jointly designing the FD MIMO beamformers and
the RIS phase configuration to be SI aware can significantly loosen the
requirement for additional SI cancellation.

###### Index Terms:

Full duplex, Reconfigurable Intelligent Surface, Joint Communication and
Sensing

2G

    Second Generation
3G

    3${}^{\text{rd}}$ Generation
3GPP

    3${}^{\text{rd}}$ Generation Partnership Project
4G

    4${}^{\text{th}}$ Generation
5G

    5${}^{\text{th}}$ Generation
AA

    Antenna Array
AC

    Admission Control
ACL

    adjacent channel leakage
AD

    Attack-Decay
ADC

    analog-to-digital converter
ADSL

    Asymmetric Digital Subscriber Line
AHW

    Alternate Hop-and-Wait
AMC

    Adaptive Modulation and Coding
AP

    Access Point
APA

    Adaptive Power Allocation
AR

    autoregressive
ARMA

    Autoregressive Moving Average
ATES

    Adaptive Throughput-based Efficiency-Satisfaction Trade-Off
AWGN

    additive white Gaussian noise
A/D

    analog and digital
BB

    branch and bound
BD

    block diagonalization
BER

    bit error rate
BF

    Best Fit
BLER

    BLock Error Rate
BPC

    Binary power control
BPSK

    binary phase-shift keying
BPA

    Best pilot-to-data power ratio (PDPR) Algorithm
BRA

    Balanced Random Allocation
BS

    base station
CAP

    Combinatorial Allocation Problem
CAPEX

    Capital Expenditure
CBF

    Coordinated Beamforming
CBR

    Constant Bit Rate
CBS

    Class Based Scheduling
CC

    Congestion Control
CDF

    Cumulative Distribution Function
CDMA

    Code-Division Multiple Access
CL

    Closed Loop
CLI

    cross-link interference
CLPC

    Closed Loop Power Control
CNR

    Channel-to-Noise Ratio
CPA

    Cellular Protection Algorithm
CPICH

    Common Pilot Channel
CoMP

    Coordinated Multi-Point
CQI

    Channel Quality Indicator
CRM

    Constrained Rate Maximization
CRN

    Cognitive Radio Network
CS

    Coordinated Scheduling
CSI

    channel state information
CSIR

    channel state information at the receiver
CSIT

    channel state information at the transmitter
CUE

    cellular user equipment
DAC

    digital-to-analog converter
D2D

    device-to-device
DCA

    Dynamic Channel Allocation
DE

    Differential Evolution
DFT

    Discrete Fourier Transform
DIST

    Distance
DL

    downlink
DMA

    Double Moving Average
DMRS

    Demodulation Reference Signal
D2DM

    D2D Mode
DMS

    D2D Mode Selection
DOCSIS

    Data Over Cable Service Interface Specification
DPC

    Dirty Paper Coding
DRA

    Dynamic Resource Assignment
DSA

    Dynamic Spectrum Access
DSM

    Delay-based Satisfaction Maximization
EBD

    electrical balance duplexer
ECC

    Electronic Communications Committee
EFLC

    Error Feedback Based Load Control
EI

    Efficiency Indicator
eNB

    Evolved Node B
EPA

    Equal Power Allocation
EPC

    Evolved Packet Core
EPS

    Evolved Packet System
E-UTRAN

    Evolved Universal Terrestrial Radio Access Network
ES

    Exhaustive Search
FD

    full-duplex
FDD

    frequency division duplexing
FDM

    Frequency Division Multiplexing
FER

    Frame Erasure Rate
FF

    Fast Fading
FSB

    Fixed Switched Beamforming
FST

    Fixed SNR Target
FTP

    File Transfer Protocol
GA

    Genetic Algorithm
GBR

    Guaranteed Bit Rate
GLR

    Gain to Leakage Ratio
GOS

    Generated Orthogonal Sequence
GPL

    GNU General Public License
GRP

    Grouping
HARQ

    Hybrid Automatic Repeat Request
HD

    half-duplex
HMS

    Harmonic Mode Selection
HOL

    Head Of Line
HSDPA

    High-Speed Downlink Packet Access
HSPA

    High Speed Packet Access
HTTP

    HyperText Transfer Protocol
ICMP

    Internet Control Message Protocol
ICI

    Intercell Interference
ID

    Identification
IETF

    Internet Engineering Task Force
ILP

    Integer Linear Program
JRAPAP

    Joint RB Assignment and Power Allocation Problem
UID

    Unique Identification
IBFD

    in-band full-duplex
IID

    Independent and Identically Distributed
IIR

    Infinite Impulse Response
ILP

    Integer Linear Problem
IMT

    International Mobile Telecommunications
INV

    Inverted Norm-based Grouping
IoT

    Internet of Things
IP

    Internet Protocol
IPv6

    Internet Protocol Version 6
I/Q

    in-phase/quadrature
IRS

    Intelligent Reflective Surface
ISAC

    integrated sensing and communications
ISD

    Inter-Site Distance
ISI

    Inter Symbol Interference
ITU

    International Telecommunication Union
JCAS

    joint communications and sensing
JOAS

    Joint Opportunistic Assignment and Scheduling
JOS

    Joint Opportunistic Scheduling
JP

    Joint Processing
JS

    Jump-Stay
KF

    Kalman filter
KKT

    Karush-Kuhn-Tucker
L3

    Layer-3
LAC

    Link Admission Control
LA

    Link Adaptation
LC

    Load Control
LNA

    low-noise amplifier
LOS

    Line of Sight
LP

    Linear Programming
LS

    least squares
LTE

    Long Term Evolution
LTE-A

    LTE-Advanced
LTE-Advanced

    Long Term Evolution Advanced
M2M

    Machine-to-Machine
MAC

    Medium Access Control
MANET

    Mobile Ad hoc Network
MC

    Modular Clock
MCS

    Modulation and Coding Scheme
MDB

    Measured Delay Based
MDI

    Minimum D2D Interference
MF

    Matched Filter
MG

    Maximum Gain
MH

    Multi-Hop
MIMO

    multiple-input multiple-output
MINLP

    Mixed Integer Nonlinear Programming
MIP

    Mixed Integer Programming
MISO

    Multiple Input Single Output
ML

    maximum likelihood
MLWDF

    Modified Largest Weighted Delay First
mmWave

    millimeter wave
MME

    Mobility Management Entity
MMSE

    minimum mean squared error
MOS

    Mean Opinion Score
MPF

    Multicarrier Proportional Fair
MRA

    Maximum Rate Allocation
MR

    Maximum Rate
MRC

    maximum ratio combining
MRT

    Maximum Ratio Transmission
MRUS

    Maximum Rate with User Satisfaction
MS

    mobile station
MSE

    mean squared error
MSI

    Multi-Stream Interference
MTC

    Machine-Type Communication
MTSI

    Multimedia Telephony Services over IMS
MTSM

    Modified Throughput-based Satisfaction Maximization
MU-MIMO

    multiuser multiple input multiple output
MU

    multi-user
NAS

    Non-Access Stratum
NB

    Node B
NE

    Nash equilibrium
NCL

    Neighbor Cell List
NLP

    Nonlinear Programming
NLOS

    Non-Line of Sight
NMSE

    Normalized Mean Square Error
NORM

    Normalized Projection-based Grouping
NP

    Non-Polynomial Time
NR

    New Radio
NRT

    Non-Real Time
NSPS

    National Security and Public Safety Services
O2I

    Outdoor to Indoor
OFDMA

    orthogonal frequency division multiple access
OFDM

    orthogonal frequency division multiplexing
OFPC

    Open Loop with Fractional Path Loss Compensation
O2I

    Outdoor-to-Indoor
OL

    Open Loop
OLPC

    Open-Loop Power Control
OL-PC

    Open-Loop Power Control
OPEX

    Operational Expenditure
ORB

    Orthogonal Random Beamforming
JO-PF

    Joint Opportunistic Proportional Fair
OSI

    Open Systems Interconnection
PAIR

    D2D Pair Gain-based Grouping
PAPR

    Peak-to-Average Power Ratio
PBCH

    physical broadcast channel
P2P

    Peer-to-Peer
PC

    Power Control
PCI

    Physical Cell ID
PDF

    Probability Density Function
PDPR

    pilot-to-data power ratio
PER

    Packet Error Rate
PF

    Proportional Fair
P-GW

    Packet Data Network Gateway
PL

    Pathloss
PMN

    Perceptive Mobile Network
PPR

    pilot power ratio
PRB

    physical resource block
PROJ

    Projection-based Grouping
ProSe

    Proximity Services
PS

    Packet Scheduling
PSAM

    pilot symbol assisted modulation
PSO

    Particle Swarm Optimization
PZF

    Projected Zero-Forcing
QAM

    Quadrature Amplitude Modulation
QoS

    Quality of Service
QPSK

    Quadri-Phase Shift Keying
RAISES

    Reallocation-based Assignment for Improved Spectral Efficiency and Satisfaction
RAN

    Radio Access Network
RA

    Resource Allocation
RAT

    Radio Access Technology
RATE

    Rate-based
RB

    resource block
RBG

    Resource Block Group
REF

    Reference Grouping
RF

    radio frequency
RIS

    reconfigurable intelligent surface
RLC

    Radio Link Control
RM

    Rate Maximization
RNC

    Radio Network Controller
RND

    Random Grouping
RRA

    Radio Resource Allocation
RRM

    Radio Resource Management
RSCP

    Received Signal Code Power
RSRP

    Reference Signal Receive Power
RSRQ

    Reference Signal Receive Quality
RR

    Round Robin
RRC

    Radio Resource Control
RSSI

    Received Signal Strength Indicator
RT

    Real Time
RU

    Resource Unit
RUNE

    RUdimentary Network Emulator
RV

    Random Variable
RX

    receiver
SAC

    Session Admission Control
SBFD

    sub-band full-duplex
SCM

    Spatial Channel Model
SC-FDMA

    Single Carrier - Frequency Division Multiple Access
SD

    Soft Dropping
S-D

    Source-Destination
SDPC

    Soft Dropping Power Control
SDMA

    Space-Division Multiple Access
SER

    Symbol Error Rate
SES

    Simple Exponential Smoothing
S-GW

    Serving Gateway
SI

    self-interference
SINR

    signal-to-interference-plus-noise ratio
SIC

    self-interference cancellation
SIP

    Session Initiation Protocol
SISO

    single-input single-output
SIMO

    Single Input Multiple Output
SIR

    signal-to-interference ratio
SLNR

    Signal-to-Leakage-plus-Noise Ratio
SMA

    Simple Moving Average
SNR

    signal-to-noise ratio
SORA

    Satisfaction Oriented Resource Allocation
SORA-NRT

    Satisfaction-Oriented Resource Allocation for Non-Real Time Services
SORA-RT

    Satisfaction-Oriented Resource Allocation for Real Time Services
SPF

    Single-Carrier Proportional Fair
SRA

    Sequential Removal Algorithm
SRS

    Sounding Reference Signal
STAR

    simultaneous transmit-and-receive
SU-MIMO

    single-user multiple input multiple output
SU

    Single-User
SVD

    Singular Value Decomposition
TCP

    Transmission Control Protocol
TDD

    time division duplexing
TDMA

    Time Division Multiple Access
TETRA

    Terrestrial Trunked Radio
TP

    Transmit Power
TPC

    Transmit Power Control
TTI

    Transmission Time Interval
TTR

    Time-To-Rendezvous
TSM

    Throughput-based Satisfaction Maximization
TU

    Typical Urban
TX

    transmit
RX

    receive
UE

    User Equipment
UEPS

    Urgency and Efficiency-based Packet Scheduling
UL

    uplink
UMTS

    Universal Mobile Telecommunications System
URI

    Uniform Resource Identifier
URM

    Unconstrained Rate Maximization
UT

    user terminal
VR

    Virtual Resource
VoIP

    Voice over IP
WAN

    Wireless Access Network
WCDMA

    Wideband Code Division Multiple Access
WF

    Water-filling
WiMAX

    Worldwide Interoperability for Microwave Access
WINNER

    Wireless World Initiative New Radio
WLAN

    Wireless Local Area Network
WMPF

    Weighted Multicarrier Proportional Fair
WPF

    Weighted Proportional Fair
WSN

    Wireless Sensor Network
WWW

    World Wide Web
XIXO

    (Single or Multiple) Input (Single or Multiple) Output
XDD

    cross-division duplex
ZF

    zero-forcing
ZMCSCG

    Zero Mean Circularly Symmetric Complex Gaussian

##  I Introduction

Full-duplex (FD) systems refer to wireless systems in which the same frequency
band is used for both transmitting and receiving signals simultaneously,
allowing for bi-directional communications[1]. This is in contrast to
traditional half-duplex systems, in which a separate frequency band is used
for transmitting and receiving signals, and the system can only perform one
function at a time [2]. FD systems have the potential to significantly improve
the performance and capacity of wireless networks, by allowing for more
efficient use of the available spectrum, thus, enabling higher data rates.

Self-interference (SI) is a major challenge for FD systems and SI cancellation
(SIC) is viable to make FD a reality [3, 4]. Due to simultaneous transmission
and reception, FD is currently receiving significant interest as it has the
potential to enable joint communication and sensing (JCAS), implying that
next-generation base stations (BSs) and radar functionalities could be
integrated into a single device. In [5], a novel JSAC system transceiver
design for an FD MIMO base station (BS) equipped with hybrid analog and
digital beamformers is presented. Recently, reconfigurable intelligent
surfaces (RISs) JCAS has gained significant interest due to its potential to
increase communications and sensing performance [6]. In [7], RIS-assisted JCAS
under the Cramér-Rao bound (CRB) constraint was derived. However, the derived
CRB neglected the RIS contribution in enhancing the sensing performance. In
[8], RIS-assisted JCAS to maximize the sum rate under a radar beam-pattern
similarity constraint was investigated. In [9], a CRB minimization-based
beamforming design for non-line-of-sight (LoS) RIS-assisted JCAS was studied.
Note that the designs presented above do not consider the effect of SI on JCAS
which may overwhelm the receiver. Moreover, the CRB derived in [7] did not
consider any effect of the RIS and, in [9], only the CRB for the non-LoS
sensing case was derived.

In this paper, we focus on the optimization of an FD JCAS system comprising
one MIMO downlink (DL) user and $1$ target to be detected by the FD node’s
radar receiver. Firstly, we derive the exact CRB for RIS-assisted FD JCAS, by
considering both the LoS and the non-LoS contributions. Then, we propose a
novel formulation to jointly minimizing the effective SI power received in
uplink (UL) at the radar receiver while maximizing the sum rate for the DL
user, subject to the derived exact CRB constraint. However, we remark that the
optimization of RIS with exact CRB constraint is extremely challenging and
would require significant additional space to be elaborated in detail. Due to
space limitations, we consider imposing the CRB constraint only on the digital
beamformer. The considered joint optimization problem is transformed into its
equivalent minimization of the mean squared error (MMSE) problem [10], which
results to be composed of two terms: the SI power at the radar and the MSE at
the DL user. A novel alternating optimization method is devised to minimize
the overall objective function. Our goal is to tackle SI with both the digital
beamformer at the MIMO transmitter and the passive beamforming offered by the
RIS, hence, enabling close to ideal FD operation for JCAS.

Simulation results show that the proposed scheme achieves significant
performance gain compared to the conventional JCAS scheme with no RIS.
Moreover, we also show that RIS can be beneficial to improve both the
communications performance and the sensing performance, while also assisting
in lowering the SI.

The rest of the paper is organized as follows. Section II presents the system
model and the exact CRB for FD JCAS. Section III presents a novel joint
beamforming design. Finally, Sections IV and V present simulations results and
conclusions, respectively.111 _Notations:_ Boldface lower and upper case
characters denote vectors and matrices, respectively. $\mathbb{E}\\{\cdot\\}$,
$\mbox{Tr}\\{\cdot\\}$, and $\mathbf{I}$ denote expectation, trace, and
identity matrix, respectively. The superscripts $(\cdot)^{T}$ and
$(\cdot)^{H}$ denote transpose and conjugate-transpose (Hermitian) operators,
respectively.

##  II System Model

We consider an FD JCAS system consisting of one FD MIMO BS $b$ equipped with
$M_{b}$ and $N_{b}$ transmit and receive antennas, respectively, which
communicates with one MIMO DL user $j$ having $N_{j}$ antenna elements, as
shown in Fig. 1. The DL signal is also used to detect targets/scatterers (via
their induced reflections) randomly distributed within the
communication/sensing environment at the radar receiver of the FD MIMO node.
We assume that the BS is assisted by a RIS placed within its near-field
region, whose role is to jointly improve communication and sensing performance
while offering strong SIC. Let $\mathbf{V}_{j}\in\mathbb{C}^{N_{j}\times
M_{j}}$ denote the digital beamformer for the unit-variance data stream
$\mathbf{s}_{j}\in\mathbb{C}^{d_{j}\times 1}$ intended for the DL user $j$.
The RIS is assumed to have $R$ rows and $C$ columns of reflection-tunable
elements, whose phase response is denoted by the diagonal matrix
$\mathbf{\Phi}_{i}\in\mathbb{C}^{RC\times RC}$, containing
$\mathbf{\phi}_{i}\in\mathbb{C}^{RC\times 1}$ on its main diagonal.

Figure 1: The proposed FD MIMO JCAS system with RIS.

Let $\mathbf{H}_{j,b}\in\mathbb{C}^{N_{j}\times M_{b}}$ and
$\mathbf{H}_{j,i}\in\mathbb{C}^{N_{j}\times RC}$ denote the channels from the
BS and the RIS to the DL user $j$, respectively. The channel from the RIS to
the BS and from the BS to RIS are denoted with
$\mathbf{H}_{b,i}\in\mathbb{C}^{N_{b}\times RC}$ and
$\mathbf{H}_{i,b}\in\mathbb{C}^{RC\times M_{b}}$, respectively. Let
$\mathbf{H}_{b,b}\in\mathbb{C}^{N_{b}\times M_{b}}$ denote the SI channel for
the MIMO FD JCAS node $b$ which can be decomposed as
$\mathbf{H}_{b,b}=\mathbf{H}_{b,b}^{l}+\mathbf{H}_{b,b}^{r}$, where
$\mathbf{H}_{b,b}^{l}$ and $\mathbf{H}_{b,b}^{r}$ denote the LoS and non-LoS
contributions for the SI channel, respectively. Since the transmit and receive
antennas of the FD JCAS are in the near-field, we consider a spherical
wavefront and model each element of $\mathbf{H}_{b,b}^{l}$ as

$\vspace{-2mm}\mathbf{H}_{b,b}^{l}(m,n)=\frac{\rho}{r_{m,n}}e^{-i2\pi\frac{d_{m,n}}{\lambda}},\quad\forall
m,n,\vspace{-1mm}$ (1)

where the scalars $\lambda$ and $\rho$ denote the wavelength and the power
normalization constant to assure
$\mathbb{E}(||\mathbf{H}_{b,b}^{l}||_{F}^{2})=M_{b}N_{b}$, respectively, and
the scalar $d_{m,n}$ denotes the distance between $m$-th receive and $n$-th
transmit antenna. Depending on the size of RIS, the channels
$\mathbf{H}_{b,i}$ and $\mathbf{H}_{i,b}$ can also be in the near-field.
Therefore, we consider modelling them similarly as (1). We assume perfect CSI,
which can be obtained by exploiting channel reciprocity via time division
duplexing (TDD).

###  II-A Communication Model

Let $\mathbf{y}_{j}$ denote the received signal at the DL user $j$, which can
be written as

$\mathbf{y}_{j}=(\mathbf{H}_{j,b}+\mathbf{H}_{j,i}\mathbf{\Phi}\mathbf{H}_{i,b})\mathbf{V}_{j}\mathbf{s}_{j}+\mathbf{n}_{j},$
(2)

where $\mathbf{n}_{j}\sim\mathcal{CN}(\mathbf{0},\sigma_{j}^{2}\mathbf{I})$
denote the noise with variance $\sigma_{j}^{2}$ at the DL user $j$. Let
$\mathbf{R}_{j}=\mathbb{E}[\mathbf{y}_{j}\mathbf{y}_{j}^{H}]$ denote the
signal-plus noise covariance matrix at the DL user $j$. Let
$\mathbf{R}_{\overline{j}}=\mathbb{E}[\mathbf{n}_{j}\mathbf{n}_{j}^{H}]$
denote its noise covariance matrix. The rate at the DL user $j$ is given by

$\mathcal{R}_{j}=\mbox{log}\big{[}\;\mbox{det}(\mathbf{R}_{\overline{j}}^{-1}\mathbf{R}_{j})\big{]}.$
(3)

###  II-B Radar Model

We consider the radar to estimate one angle of arrival (AoA), denoted as
$\theta_{k}$. Let $\omega_{0}$ denote the angle between the FD BS and the RIS,
which is perfectly known at the BS. Let $\mathbf{y}_{b}$ denote its total
received signal, and let $\mathbf{A}$ denote a matrix, containing
contributions from all the paths, defined as

$\displaystyle\mathbf{A}=$
$\displaystyle\psi_{k}\mathbf{a}_{r}(\theta_{k})\mathbf{a}_{t}(\theta_{k})^{T}+\xi_{1,k}\mathbf{a}_{r}(\omega_{0})\mathbf{a}_{t}(\omega_{0})^{T}\mathbf{\Phi}\mathbf{a}_{i}(\theta_{k}){\mathbf{a}_{i}(\theta_{k})}^{T}$
(4)
$\displaystyle\mathbf{\Phi}\mathbf{a}_{i}(\omega_{0})\mathbf{a}_{t}(\omega_{0})^{T}+\xi_{2,k}\;\mathbf{a}_{r}(\theta_{k})\;{\mathbf{a}_{i}(\theta_{k})}^{T}\mathbf{\Phi}\mathbf{a}_{i}(\omega_{0})\mathbf{a}_{t}(\omega_{0})^{T}$
$\displaystyle+\psi_{b,i}\mathbf{a}_{r}(\omega_{0})\mathbf{a}_{t}(\omega_{0})^{T}\mathbf{\Phi}\xi_{3,k}\;\mathbf{a}_{i}(\theta_{k})\;{\mathbf{a}_{t}(\theta_{k})}^{T},$

where $\psi_{k}$ denotes the reflection coefficient for the LoS path between
the BS and target $k$,
$\mathbf{n}_{b}\sim\mathcal{CN}(\mathbf{0},\sigma_{r}^{2}\mathbf{I})$ denotes
the noise at the radar with variance $\sigma_{r}^{2}$, and $\mathbf{a}_{r}$
and $\mathbf{a}_{t}$ denote the transmit and receive antenna steering vectors
for JCAS node, respectively. The scalars $\xi_{1,k},\xi_{2,k}$ and $\xi_{3,k}$
denote the reflection coefficients for the signal from RIS to target $k$ and
back to RIS, from RIS to the radar via target $k$ and from the transmitter to
the RIS via target $k$, respectively. Given $\mathbf{A}$, we can write the
received signal $\mathbf{y}_{b}$ as

$\displaystyle\mathbf{y}_{b}=$
$\displaystyle\mathbf{A}\mathbf{V}_{j}\mathbf{s}_{j}+\mathbf{H}_{b,b}^{l}\mathbf{V}_{j}\mathbf{s}_{j}+\mathbf{H}_{b,i}\mathbf{\Phi}\mathbf{H}_{i,b}\mathbf{V}_{j}\mathbf{s}_{j}+\mathbf{n}_{b},$
(5)

which contains the effective SI. Assuming uniform linear arrays at the FD
node, the antenna steering vector $\mathbf{a}_{r}$ at the receiver, appearing
in $\mathbf{A}$, can be modelled as

$\mathbf{a}_{r}(\theta_{k})=\frac{1}{\sqrt{N_{b}}}[1,e^{j\frac{2\pi}{\lambda}dsin(\theta_{k})},...,e^{j\frac{2\pi}{\lambda}d(N_{b}-1)sin(\theta_{k})}]^{T},\vspace{-2mm}$
(6)

and similar modelling also holds for $\mathbf{a}_{t}$. The scalars $d$ and
$\lambda$ denote the distance and wavelength, respectively. Let $\phi_{k}$ and
$\varphi_{k}$ denote the elevation and the azimuth angles between the RIS and
the target $k$. The RIS response can be modelled as a uniform planer array
(UPA) as

$\mathbf{a}_{i}(\phi_{k},\varphi_{k})=\frac{1}{\sqrt{RC}}[1,e^{j\frac{2\pi}{\lambda}\varpi_{1}},...,e^{j\frac{2\pi}{\lambda}\varpi_{RC-1}}]^{T},$
(7)

where
$\varpi_{i}=d_{i_{x}}sin(\phi_{k})cos(\varphi_{k})+d_{i_{z}}sin(\varphi_{k})$
[11], $0<i<RC$, and $d_{i_{x}}$ and $d_{i_{z}}$ denote the distance of RIS’s
element $i$ from its first element on the $x$ and $z$ axis, respectively. As
the position and the orientation of RIS are known at the FD BS, the azimuth
and elevation angles can be expressed as a function of $\theta_{k}$ to be
estimated as

$\phi_{k}=arcos\big{(}\frac{l_{k}cos(\theta_{k})-r_{1}cos(\omega_{0})}{r_{2}}\big{)},$
(8a)
$\varphi_{k}=arcos(\frac{l_{k}cos(\theta_{k})-r_{1}cos(w_{0})}{r_{r}cos(\theta_{k})}\big{)},$
(8b)

where $l_{k}$ is the distance of the target $k$, $\omega_{0}$ is the angle
between the FD BS and RIS, and the scalars $r_{1},l_{k},r_{2},r_{r}$, assuming
the RIS to be placed on the (x,z) plane, denote the distance between the FD BS
and RIS with the relative angle $\omega_{0}$, the distance between RIS and
target $k$, the distance between RIS and target $k$ on the (x,y) plane and the
distance on the (x,z) plane, respectively.

To achieve an accurate estimation, the CRBs for AoA $\theta_{k}$ should be
below the threshold $\zeta_{k}$, which imposes the constraint

$\frac{1}{\zeta_{k}}\leq\frac{1}{\mbox{CRB}(\theta_{k})},$ (9)

where $\mbox{CRB}(\theta_{k})$ is derived in the Appendix.

###  II-C Problem Formulation

We embark on the task of jointly optimizing the performance of both
communication and radar systems. While the concept of communication rate is
well-defined for the DL user, there is no such concept that exists for the UL.
The sum rate maximization problem can be formulated as a function of MMSE
[10]. Conversely, even though the notion of UL rate in the context of FD JCAS
is not well-established, the challenge of SI cancellation can be framed as a
minimization problem, specifically minimizing the squared Frobenius norm of
the SI power, thereby improving the accuracy of radar detection.

For the DL user, we assume that it deploys the combiner $\mathbf{F}_{j}$ to
estimate its data streams $\mathbf{s}_{j}$ as
$\hat{\mathbf{s}}_{j}=\mathbf{F}_{j}\mathbf{y}_{j}$. Assuming that the
combiner $\mathbf{F}_{j}$ is optimized based on the minimization of the MSE
criteria, its closed-form solution is

$\displaystyle\mathbf{F}_{j}=$
$\displaystyle\mathbf{V}_{j}^{H}(\mathbf{H}_{j,b}+\mathbf{H}_{j,i}\mathbf{\Phi}\mathbf{H}_{i,b})^{H}((\mathbf{H}_{j,b}+\mathbf{H}_{j,i}\mathbf{\Phi}\mathbf{H}_{i,b})\mathbf{V}_{j}\mathbf{V}_{j}^{H}$
(10)
$\displaystyle(\mathbf{H}_{j,b}+\mathbf{H}_{j,i}\mathbf{\Phi}\mathbf{H}_{i,b})+\sigma_{r}^{2}\mathbf{I})^{-1}.$

Given $\mathbf{F}_{j}$ as (10), the MSE of the DL user $j$ becomes

$\mathbf{E}_{j}=(\mathbf{I}+\mathbf{V}_{j}^{H}\mathbf{H}_{j}^{H}\mathbf{R}_{\overline{j}}^{-1}\mathbf{H}_{j}\mathbf{V}_{j})^{-1},$
(11)

where $\mathbf{R}_{j}={\sigma_{j}^{2}}\mathbf{I}$ is the noise covariance
matrix with variance $\sigma_{j}^{2}$. Let $\mathbf{W}_{j}$ denote the weight
computed as [10]

$\mathbf{W}_{j}=\frac{w_{j}}{\mbox{ln}2}\mathbf{E}_{j}^{-1}.$ (12)

The joint minimization problem under the total sum-power, CRB and unit-modulus
constraint for the RIS can be stated as

$\underset{\begin{subarray}{c}\mathbf{V}_{j},\mathbf{\Phi}\end{subarray}}{\min}\quad\mbox{Tr}(\mathbf{E}_{SI})+\mbox{Tr}(\mathbf{W}_{j}\mathbf{E}_{j})$
(13a)
$\text{s.t.}\quad\mbox{Tr}\big{(}\mathbf{V}_{j}\mathbf{V}_{j}^{H}\big{)}\leq
p_{o},$ (13b) $\quad|\mathbf{\phi}(i)|=1,\forall i,$ (13c)
$\eqref{crb_constraint}$ (13d)

where $\mathbf{E}_{SI}$ is a matrix defined as

$\displaystyle\mathbf{E}_{SI}=$
$\displaystyle\mathbf{H}_{b,b}^{l}\mathbf{V}_{j}\mathbf{V}_{j}^{H}{\mathbf{H}_{b,b}^{l}}^{H}+\mathbf{H}_{b,b}^{l}\mathbf{V}_{j}\mathbf{V}_{j}^{H}\mathbf{H}_{i,b}^{H}\mathbf{\Phi}^{H}\mathbf{H}_{b,i}^{H}+\mathbf{H}_{b,i}\mathbf{\Phi}$
(14)
$\displaystyle\mathbf{H}_{i,b}\mathbf{V}_{j}\mathbf{V}_{j}^{H}{\mathbf{H}_{b,b}^{l}}^{H}+\mathbf{H}_{b,i}\mathbf{\Phi}\mathbf{H}_{i,b}\mathbf{V}_{j}\mathbf{V}_{j}^{H}\mathbf{H}_{i,b}^{H}\mathbf{\Phi}^{H}\mathbf{H}_{b,i}^{H}$

obtained by writing the Frobenius norm squared as a function of the trace
operator, whose diagonal elements capture the effective SI power and (13b) and
(13c) denote the total sum-power constraint $p_{0}$ and the unit-modulus
constraint on RIS.

##  III Novel Algorithm Design for FD JCAS

In this section, we provide a novel algorithm to solve the optimization
problem (13) based on alternating optimization. Let $\mathcal{L}$ denote the
Lagrangian of (13) and let $\lambda_{0}$ and $\mu_{k}$ denote the Lagrange
multipliers for the total sum-power constraint at the FD BS and for the CRB
constraint of target $k$, respectively.

###  III-A Digital Beamformer Optimization

To optimize the digital beamformer $\mathbf{V}_{j}$, which jointly optimizes
the DL rate and handles the SI for JCAS, we take the derivative of
$\mathcal{L}$ with respect to the conjugate of $\mathbf{V}_{j}$, which leads
to the following optimal WMMSE digital beamformer

$\displaystyle\mathbf{V}_{j}=$
$\displaystyle\Big{(}(\mathbf{H}_{j,b}+\mathbf{H}_{j,i}\mathbf{\Phi}\mathbf{H}_{i,b})^{H}\mathbf{F}_{j}^{H}\mathbf{W}_{j}\mathbf{F}_{j}(\mathbf{H}_{j,b}+\mathbf{H}_{j,i}\mathbf{\Phi}\mathbf{H}_{i,b})$
(15)
$\displaystyle+(\mathbf{H}_{b,b}^{l}+\mathbf{H}_{b,i}\mathbf{\Phi}\mathbf{H}_{i,b})^{H}(\mathbf{H}_{b,b}^{l}+\mathbf{H}_{b,i}\mathbf{\Phi}\mathbf{H}_{i,b})$
$\displaystyle+\mu_{k}2\bar{\mathbf{A}}_{\theta_{k}}^{H}\mathbf{\Sigma}^{-1}\bar{\mathbf{A}}_{\theta_{k}}+\lambda_{0}\mathbf{I}\Big{)}^{-1}(\mathbf{H}_{j,b}+\mathbf{H}_{j,i}\mathbf{\Phi}\mathbf{H}_{i,b})\mathbf{F}_{j}^{H}\mathbf{W}_{j}.$

where $\bar{\mathbf{A}}_{\theta_{k}}^{H}$ is defined in the Appendix. To find
the optimal values of $\lambda_{0}$ and $\mu_{k}$, a linear search method can
be adopted. In this study, we employ the Bisection method.

###  III-B RIS Optimization

We consider optimizing the RIS to jointly achieve quasi-ideal SIC on the UL
side (which can be seen as a virtual UL rate optimization problem) and also to
maximize the rate at the DL side. For such a purpose, we consider minimizing
the Frobenius Norm squared of the effective SI. Let
$\mathbf{B},\mathbf{C},\mathbf{D}$ be

$\quad\mathbf{B}=\mathbf{H}_{b,i}^{H}\mathbf{H}_{b,i}+\mathbf{H}_{j,i}^{H}\mathbf{F}_{j}^{H}\mathbf{W}_{j}\mathbf{F}_{j}\mathbf{H}_{j,i},$
(16a)
$\mathbf{C}=\mathbf{H}_{i,b}\mathbf{H}_{i,b}^{H}+\mathbf{H}_{i,b}\mathbf{V}_{j}\mathbf{V}_{j}^{H}\mathbf{H}_{i,b}^{H},$
(16b)
$\mathbf{D}=\mathbf{H}_{i,b}{\mathbf{H}_{b,b}^{l}}^{H}\mathbf{H}_{b,i}+\mathbf{H}_{i,b}\mathbf{V}_{j}\mathbf{V}_{j}^{H}\mathbf{H}_{j,b}^{H}\mathbf{F}_{j}^{H}\mathbf{W}_{j}\mathbf{F}_{j}\mathbf{H}_{j,i},$
(16c)

Given the auxiliary matrices, problem (13) can be restated with respect to
diagonal elements of RIS $\mathbf{\phi}$ as

$\underset{\begin{subarray}{c}\mathbf{\phi}\end{subarray}}{\text{min}}\quad\mathbf{\phi}^{H}\mathbf{\Lambda}\mathbf{\phi}+\mathbf{d}^{T}\mathbf{\phi}+\mathbf{\phi}^{H}\mathbf{d}^{*}$
(17a) $\eqref{c2_WSR}$ (17b)

where $\mathbf{\Lambda}=(\mathbf{B}\odot\mathbf{C})$, and $\mathbf{d}$ is a
vector made of the diagonal elements of the matrix $\mathbf{D}$. To render a
feasible solution, we adopt the majorization-maximization optimization method
[12] by constructing an upper bound denoted as $g(\cdot)$ for the objective
function (17), denoted as $f(\cdot)$. For a problem of type (17), it has been
shown in [12] that at iteration $n$ the following upper bound can be
considered

$g(\bm{\phi}|\bm{\phi}^{(n)})=2\mbox{Re}\\{{\mathbf{s}}^{H}\mathbf{q}^{(n)}\\}+c,$
(18)

where $\lambda^{max}$ is the maximum eigenvalue
$\mathbf{q}^{(n)}=(\lambda^{max}\mathbf{I}-\mathbf{\Lambda})\bm{\phi}^{(n)}-{\mathbf{d}}^{*}$.
Given the upper bound and $\mathbf{q}^{(n)}$, our problem can be restated as a
minimization of the upper bound as,

$\underset{\begin{subarray}{c}\bm{\phi}_{r}\end{subarray}}{\min}\quad
2\mbox{Re}\\{{\mathbf{d}}^{H}\mathbf{q}^{(n)}\\},$ (19a)
$\text{s.t.}\quad|\bm{\phi}(i)|=1,\quad\forall i,$ (19b)

By solving problem (19), we get the following

$\bm{\phi}^{(n+1)}=e^{i\angle\mathbf{q}^{(n)}}.$ (20)

When the digital beamformer is computed under the CRB constraint at each
iteration, the RIS optimization should be carried out by solving the
aforementioned problem iteratively until convergence. The procedure for
optimizing the phase response of RIS is given in Algorithm $1$. The overall
procedure to optimize and solve the joint optimization problem under the CRB
is formally given in Algorithm $2$.

Algorithm 1 Optimization of RIS

Initialize: iteration index $n=1$, accuracy $\epsilon$.
Evaluate: $f(\bm{\phi}(0))$.
Repeat until convergence

Calculate
$\mathbf{q}^{(n)}=(\lambda_{i}^{max}\mathbf{I}-\mathbf{\Lambda})\bm{\phi}^{(n)}-{\mathbf{d}}^{*}$

Update $\bm{\phi}^{(n+1)}$ as
$\bm{\phi}^{(n+1)}=e^{i\angle\mathbf{q}_{i}^{(n)}}$.

if
$|f(\bm{\phi}_{i}^{(n+1)})-f(\bm{\phi}_{i}^{(n)})|/f(\bm{\phi}_{i}^{(n+1)})\leq\epsilon$

Stop and return $\bm{\phi}^{(n+1)}$.

else n=n+1 and repeat.

Algorithm 2 JCAS Optimization with SIC

Initialize iteration index $n$, accuracy $\epsilon$, digital beamformer and
combiner.
Repeat until convergence

Update $\mathbf{F}_{j}$ with (10).

Update $\mathbf{W}_{j}$ with (12).

Update $\mathbf{V}_{j}$ with (15).

Search for the Lagrange multipliers.

Update $\mathbf{\Phi}_{i}$ with Algorithm 1.

if convergence condition is satisfied

Stop and return the optimized variables.

else repeat.

The convergence of the proposed scheme is straightforward by combining the
reasoning of the well-established WMMSE [10] and the majorization-maximization
technique [12]. However, due to space limitations, we omit the extended proof.

##  IV Simulation Results

In this section, we present simulation results to validate the advantage of
the proposed SI-aware FD JCAS transceiver design.

We consider the FD BS and the DL user to be equipped with uniform linear
arrays (ULA) at $80$ m from the FD BS, and the FD BS to be placed in the
center of the three-dimensional coordinate system with ULA aligned with the
z-axis. The RIS is assumed to be placed on the (x,y) plane, with a relative
angle of $30^{\circ}$ with respect to the FD BS, with its first element being
$5\leavevmode\nobreak\ $m far from the first transmit antenna of the FD BS. We
assume that the FD BS is assumed to be equipped $M_{b}=15$ transmit antennas
and $N_{b}=10$ receive antennas, and the DL user is assumed to be equipped
with $N_{j}=5$ receiving antennas. The RIS is assumed to be of size $10\times
10$. The channels between the FD BS and the DL user $j$, denoted with
$\mathbf{H}_{j,b}$ and $\mathbf{H}_{b,i}$, are modelled with the line of sight
(LoS) channel model. The number of data streams to be transmitted to the DL
user is set to be $d_{j}=2$. The digital beamformer $\mathbf{V}_{j}$ is
initialized as the dominant eigenvectors of the effective channel covariance
matrices, and the response of the NF-IRSs is initialized with random phases.
We define the signal-to-noise-ratio (SNR) of our system as
$SNR=p_{o}/\sigma_{j}^{2}$, where $p_{o}$ is the total transmit power and
$\sigma_{j}^{2}$ is the noise variance at the DL user. For the CRB constraint,
we set $\zeta_{k}=0.01$ and the AoA to be estimated to be randomly distributed
on a circle of 50$m$ with the angle range limited to $[-\pi/2,-\pi/2]$.

For comparison, we define the following benchmark schemes: 1) _RIS-
Communications Only_ \- A scheme in which the beamformers and RIS are designed
to maximize the performance of the communications and there is no sensing and
SI (half-duplex (HD) mode), 2) _No RIS-Communications Only_ \- A scheme
similar to scheme 1) but without RIS, and 3) _No RIS- With Sensing_ \- A
scheme in which FD BS performs JCAS but without the aid of RIS. We label our
scheme as _RIS-With Sensing_.

Figure 2: Communications performance for FD JCAS with SIC. Figure 3: MSE for
the AoA estimation as a function of SNR.

Fig. 2 shows the performance of the communications in terms of the user rate
as a function of the proposed novel FD JCAS transceiver design, in comparison
to the benchmark schemes. We can see that when the BS act as an HD BS and
there is no SI and sensing, the beamformer $\mathbf{V}_{j}$ and the RIS is
designed to only maximize the DL performance, leading to a higher rate.
However, in the FD JCAS case, the beamformers and RIS are designed to jointly
handle SI, enhance the sensing performance and improve the communications.

In Fig. (3), we compare the performance of the estimation of AoA $\theta_{k}$
when using a MUSIC-based estimator. Our approach demonstrates effective
management of SI, and as the SNR increases, the estimation performance
approaches the CRB. While there remains a slight difference between the
estimation performance and the CRB, this gap can be further reduced by
incorporating the CRB constraint into the optimization of the Reconfigurable
Intelligent Surface (RIS), which is a direction for future research.

##  V Conclusions

This work introduces a new method to enable FD JCAS by considering the impact
of SI. The authors derive the exact CRB for RIS-assisted JCAS and propose a
joint optimization framework based on alternating optimization that satisfies
the CRB constraint. Simulation results demonstrate that the proposed
beamforming method, which accounts for SI, leads to a substantial performance
improvement and effectively manages SI. Furthermore, the reduction in data
transmission rate compared to a communications-only approach is negligible,
which paves the path toward FD JCAS with accurate and energy-efficient SI
management with RIS.

In this section, we derive the CRB for AoA $\theta_{k}$, which can be written
as [13]

$\mbox{CRB}(\theta_{k})=\frac{1}{2}\Big{(}\mbox{Tr}\big{(}\mathbf{V}_{j}^{H}\bar{\mathbf{A}}_{\theta_{k}}^{H}\mathbf{\Sigma}^{-1}\bar{\mathbf{A}}_{\theta_{k}}\mathbf{V}_{j}\big{)}\Big{)}^{-1}.$
(21)

where $\bar{\mathbf{A}}_{\theta_{k}}=\partial\mathbf{A}/\partial\theta_{k}$.
We first define the derivatives of the antenna responses as

$\partial\mathbf{a}_{r}=\frac{1}{\sqrt{N_{b}}}[0,....,j\frac{2\pi}{\lambda}d(N_{b}-1)cos(\theta_{k})e^{j\frac{2\pi}{\lambda}d(N_{b}-1)sin(\theta_{k})}]^{T},$
(22a)
$\partial\mathbf{a}_{t}=\frac{1}{\sqrt{M_{b}}}[0,....,j\frac{2\pi}{\lambda}d(M_{b}-1)cos(\theta_{k})e^{j\frac{2\pi}{\lambda}d(M_{b}-1)sin(\theta_{k})}]^{T},$
(22b)

Let $\partial\varpi_{i}$ denote the derivative of $\varpi_{i}$, obtained by
expressing $\phi_{k}$ and $\varphi_{k}$ as a function of $\theta_{k}$. Then
the derivative of $\mathbf{a}_{r}^{RIS}$ with respect to $\theta_{k}$ can be
written as

$\partial\mathbf{a}_{t}^{RIS}=\frac{1}{\sqrt{RC}}[0,......,j\frac{2\pi}{\lambda}\partial\varpi_{i}e^{j\frac{2\pi}{\lambda}\varpi_{RC-1}}]$
(23)

By considering the complete deployment effect of RIS, including both the LoS
and non-LoS links, $\bar{\mathbf{A}}_{\theta_{k}}$ can be written as

$\displaystyle\bar{\mathbf{A}}_{\theta_{k}}=$
$\displaystyle\psi_{k}\partial\mathbf{a}_{r}(\theta_{k})\mathbf{a}_{t}(\theta_{k})^{T}+\psi_{k}\mathbf{a}_{r}(\theta_{k})\partial\mathbf{a}_{t}(\theta_{k})^{T}$
(24)
$\displaystyle+\xi_{1,k}\mathbf{a}_{r}(\omega_{0})\mathbf{a}_{t}(\omega_{0})^{T}\mathbf{\Phi}\partial\mathbf{a}_{r}^{RIS}(\theta_{k}){\mathbf{a}_{i}(\theta_{k})}^{T}\mathbf{\Phi}\mathbf{a}_{r}^{RIS}(\omega_{0})$
$\displaystyle\mathbf{a}_{t}(\omega_{0})^{T}+\xi_{1,k}\mathbf{a}_{r}(\omega_{0})\mathbf{a}_{t}(\omega_{0})^{T}\mathbf{\Phi}\mathbf{a}_{i}(\theta_{k})\partial{\mathbf{a}_{i}(\theta_{k})}^{T}\mathbf{\Phi}\mathbf{a}_{i}(\omega_{0})$
$\displaystyle\mathbf{a}_{t}(\omega_{0})^{T}+\xi_{2,k}\;\partial\mathbf{a}_{r}(\theta_{k}){\mathbf{a}_{t}^{RIS}(\theta_{k})}^{T}\mathbf{\Phi}\mathbf{a}_{i}(\omega_{0})\mathbf{a}_{t}(\omega_{0})^{T}$
$\displaystyle+\xi_{2,k}\;\mathbf{a}_{r}(\theta_{k})\partial{\mathbf{a}_{i}(\theta_{k})}^{T}\mathbf{\Phi}\mathbf{a}_{i}(\omega_{0})\mathbf{a}_{t}(\omega_{0})^{T}$
$\displaystyle+\xi_{3,k}\mathbf{a}_{r}(\omega_{0})\mathbf{a}_{t}(\omega_{0})^{T}\mathbf{\Phi}\partial\mathbf{a}_{i}(\theta_{k})\;{\mathbf{a}_{t}(\theta_{k})}^{T}$
$\displaystyle+\xi_{3,k}\mathbf{a}_{r}(\omega_{0})\mathbf{a}_{t}(\omega_{0})^{T}\mathbf{\Phi}\mathbf{a}_{i}(\theta_{k})\partial{\mathbf{a}_{t}(\theta_{k})}^{T},$

## Acknowledgement

This work has been supported by the SNS JU TERRAMETA project under EU’s
Horizon Europe research and innovation programme under Grant Agreement number
101097101.

## References

  * [1] C. K. Sheemar _et al._ , “Practical hybrid beamforming for millimeter wave massive MIMO full duplex with limited dynamic range,” _IEEE Open J. Commun. Society_ , vol. 3, pp. 127–143, 2022.
  * [2] G. Liu _et al._ , “In-band full-duplex relaying: A survey, research issues and challenges,” _IEEE Commun. Surveys Tuts._ , vol. 17, no. 2, pp. 500–524, 2015.
  * [3] G. C. Alexandropoulos _et al._ , “Full duplex hybrid A/D beamforming with reduced complexity multi-tap analog cancellation,” in _Proc. IEEE SPAWC_ , Atlanta, USA, 2020, pp. 1–6.
  * [4] C. K. Sheemar and D. Slock, “Hybrid beamforming and combining for millimeter wave full duplex massive mimo interference channel,” in _2021 IEEE Global Communications Conference (GLOBECOM)_. IEEE, 2021, pp. 1–6.
  * [5] M. A. Islam _et al._ , “Integrated sensing and communication with millimeter wave full duplex hybrid beamforming,” in _Proc. IEEE ICC_ , Seoul, South Korea, 2022, pp. 1–6.
  * [6] C. B. Barneto, E. Rastorgueva-Foi, M. F. Keskin, T. Riihonen, M. Turunen, J. Talvitie, H. Wymeersch, and M. Valkama, “Millimeter-wave mobile sensing and environment mapping: Models, algorithms and validation,” _IEEE Transactions on Vehicular Technology_ , vol. 71, no. 4, pp. 3900–3916, 2022.
  * [7] X. Wang _et al._ , “Joint waveform and discrete phase shift design for RIS-assisted integrated sensing and communication system under cramér-rao bound constraint,” _IEEE Trans. Veh. Technol._ , vol. 71, no. 1, pp. 1004–1009, 2021.
  * [8] H. Luo, R. Liu, M. Li, Y. Liu, and Q. Liu, “Joint beamforming design for RIS-assisted integrated sensing and communication systems,” _IEEE Trans. Veh. Technol._ , vol. 71, no. 12, pp. 13 393–13 397, 2022.
  * [9] X. Song _et al._ , “Intelligent reflecting surface enabled sensing: Cramér-rao lower bound optimization,” in _Proc. IEEE GLOBECOM_ , Rio de Janerio, Brazil, 2022, pp. 413–418.
  * [10] S. S. Christensen _et al._ , “Weighted sum-rate maximization using weighted MMSE for MIMO-BC beamforming design,” _IEEE Transactions on Wireless Communications_ , vol. 7, no. 12, pp. 4792–4799, 2008.
  * [11] Y. Chen _et al._ , “Hybrid spherical-and planar-wave channel modeling and spatial multiplexing analysis for terahertz integrated um-mimo and irs systems,” in _Proc. IEEE ICC_ , Seoul, South Korea, 2022, pp. 3490–3495.
  * [12] C. Pan _et al._ , “Multicell mimo communications relying on intelligent reflecting surfaces,” _IEEE Trans. Wireless Commun._ , vol. 19, no. 8, pp. 5218–5233, 2020.
  * [13] Z. Cheng _et al._ , “Transmit sequence design for dual-function radar-communication system with one-bit DACs,” _IEEE Trans. Wireless Commun._ , vol. 20, no. 9, pp. 5846–5860, 2021.

  *[PDPR]: pilot-to-data power ratio
  *[BS]: base station


# DogFLW: Dog Facial Landmarks in the Wild Dataset

George Martvel∗1 Greta Abele2 Annika Bremhorst2 Chiara Canori3 Nareed Farhat1
Giulia Pedretti3 Ilan Shimshoni1 Anna Zamansky1

###### Abstract

Affective computing for animals is a rapidly expanding research area that is
going deeper than automated movement tracking to address animal internal
states, like pain and emotions. Facial expressions can serve to communicate
information about these states in mammals. However, unlike human-related
studies, there is a significant shortage of datasets that would enable the
automated analysis of animal facial expressions. Inspired by the recently
introduced Cat Facial Landmarks in the Wild dataset, presenting cat faces
annotated with 48 facial anatomy-based landmarks, in this paper, we develop an
analogous dataset containing 3,274 annotated images of dogs. Our dataset is
based on a scheme of 46 facial anatomy-based landmarks.

The DogFLW dataset is available from the corresponding author upon a
reasonable request.

1 University of Haifa, Israel

2 Dogs and Science, Switzerland

3 University of Parma, Italy

##  1 Introduction

Due to the internal nature of affective states, including emotions and pain,
their recognition in animals is very challenging in light of the lack of a
verbal basis for communication. Nevertheless, observing subtle changes in
their facial expressions and body language shows promise as a non-intrusive
method for studying these states. Mammals produce facial expressions, which
have been linked emotional states in a variety of species [14]. Therefore, it
is unsurprising that the number of works addressing automation of animal
affect recognition tasks is rapidly growing; Broome et al. [4] provides a
comprehensive review of state-of-the-art works using computer vision to
address such recognition in animals. Despite dogs being one of the most well-
studied animals in behavior studies, only a few of these works address these
species, with solely one work [1] focusing on facial expressions and using a
dataset collected in a controlled experimental setting.

The interest in cognitive and behavioral aspects of dogs has been increasing
dramatically [35]. Dogs can serve as valuable clinical models for numerous
human disorders, owing to their large size: many canine conditions mirror
human diseases, including diabetes, cancers, epilepsy, eye diseases,
autoimmune diseases, and other rare diseases [27]. Other factors contributing
to the widespread interest in dogs include fascination with their origins in
the context of domestication, as well as their behavior and cognitive
abilities. Moreover, we must improve our understanding of and regulate dog-
human interactions and welfare impacts, including working dogs and shelter
dogs [35].

The objective measurement of dog facial expressions is crucial for their
investigation as indicators of emotional states in different contexts [3, 41].
Facial expressions have also been studied in the context of understanding dog-
human communication (e.g., the impact of dog facial phenotypes on their
communication abilities with humans [44], the impact of facial features on the
ability of humans to understand dogs [16], etc.)

The gold standard for objectively assessing changes in facial expressions in
human emotion research is the Facial Action Coding System — FACS [15]. FACS
has recently been adapted for different non-human species, including dogs
[48]. DogFACS has been applied in several studies [8, 3, 2, 41, 44] to
objectively measure facial changes. However, using this method for facial
expression analysis depends on laborious manual annotation, which also
requires extensive human training and certification and may be prone to human
error or bias [24]. Some first steps to automated DogFACS were taken by Boneh
et al. [1].

Geometric morphometrics offers an appealing alternative approach, successfully
applied in analyzing cat facial features [21, 22]. This method uses points
(landmarks) on objects as proxies for shape, allowing for the quantification
of facial shape changes. This concept is closely related to landmarks
extensively studied in the human domain.

Indeed, numerous fundamental methods for detecting, labeling, and aligning
facial and body landmarks in humans have emerged [6, 28, 25, 43, 32, 11, 23,
49]. Typically, datasets with human facial landmarks consist of thousands of
images with dozens of landmarks. This abundance of data leads to better model
performance, even in challenging scenarios such as occlusions or low-quality
images.

The animal domain, on the other hand, severely needs landmark-related
datasets. Table 1 shows the available datasets, including their size and
number of landmarks, which generally are extremely small compared to the human
domain.

Dataset | Animal | Size | Landmarks
---|---|---|---
Khan et al. [30] | Various | 21,900 | 9
Zhang et al. [51] | Cat | 10,000 | 9
Liu et al. [33] | Dog | 8,351 | 8
Cao et al. [10] | Various | 5,517 | 5
Mougeot et al. [40] | Dog | 3,148 | 3
Martvel et al. [36] | Cat | 2,091 | 48
Sun et al. [45] | Cat | 1,706 | 15
Pessanha et al. [42] | Horse (tilted) | 952 | 44
Hewitt et al. [26] | Sheep | 850 | 25
Yang et al. [50] | Sheep | 600 | 8
Pessanha et al. [42] | Horse (frontal) | 370 | 54
Pessanha et al. [42] | Horse (side) | 348 | 45
DogFLW | Dog | 3,274 | 46
Table 1: Comparison of animal facial landmarks datasets

In the case of domesticated species, which are often artificially selected for
specific features, there is a great variation in facial morphology and
appearance. It is particularly noticeable in the case of cats and even more so
in dogs, making it challenging to create versatile landmark detection models
[9, 12, 48]. With the increase in quantity and quality of animal facial
landmark datasets, it has become possible to create cross-view models [5, 10,
34], but the number of landmarks in such works is usually relatively small. To
solve the problems of determining the emotions or states of animals, several
dozens of landmarks are needed [19, 20, 17], which are unique for a particular
animal.

To tackle the shortage of datasets, the Cat Facial Landmarks in the Wild
(CatFLW) dataset was recently introduced by Martvel et al. [36]. The dataset
comprises over 2,000 images of cat faces in various environments, each with
face bounding boxes and 48 facial landmarks as detailed in [21]. Notably, the
utilized landmark scheme is based on the cat’s facial anatomy and the CatFACS
annotation system [9]. This approach has facilitated the development of models
for automated identification of pain in cats [38, 18, 37].

Dogs exhibit a wide range of facial features, making them more challenging
than cats for automated facial analysis. To fill this gap, this paper
introduces the Dog Facial Landmarks in the Wild (DogFLW) dataset, comprising a
set of 46 facial landmarks that were defined, based on and guided by the
facial anatomy of dogs and the DogFACS method. We utilized the Ensemble
Landmark Detector (ELD) [39] to provide a benchmark for this dataset.

##  2 Landmark Scheme

The development of the dog facial landmark scheme was undertaken with the aim
of creating a comprehensive framework for analyzing canine facial expressions
with the help of three experienced dog behavior researchers certified in
DogFACS coding[48]. To establish the number of landmarks and each landmark
location, the experts worked independently and then converged using expert
consensus. The obtained final landmark scheme is shown in Figure 1. All
details and landmark descriptions are available in the spreadsheet.

Figure 1: Annotated Dog’s Face. Image of a dog with a face bounding box and 46
facial landmarks.

##  3 Dataset Properties

As a source for our dataset, we used the Stanford Dog dataset [31], which
contains 20,580 images, 120 breeds and bounding boxes for dogs. Since we are
interested in bounding boxes for the dog’s face rather than the entire body,
we did not utilize the latter in the current study.

First, we selected a random subset with an equal amount of images per breed.
Then, we filtered the images according to the following criteria: the image
contains a single visible dog face, where the dog is in non-laboratory
conditions (‘in the wild’). Other dogs could be present, but their faces
shouldn’t be visible for unambiguity of detection. The resulting subset
contains 7-40 images per breed (25 on average) of all 120 breeds (3,274 images
total), ranging in size from $100\times 103$ to $1944\times 2592$ pixels. Dogs
in images have different sizes, colors, body and head poses, as well as
different environments.

Each image is annotated with 46 facial landmarks using the CVAT platform[13]
according to the scheme described above. Unlike the CatFLW [36], our dataset
has occluded landmarks, all annotated landmarks have a visibility indicator
(”0” — occluded, ”1” — visible). Each image is also annotated with a face
bounding box, which encompasses the entire face of the animal, along with
approximately 10% of the surrounding space. This margin has proven crucial for
training face detection models, as it prevents the cropping of important parts
of the dog’s face, such as the tips of the ears or the mouth. Figure 2 shows
some examples of annotated images from the DogFLW dataset.

Figure 2: Examples of annotated images from the DogFLW.

##  4 Benchmarks

For the evaluation, the dataset was randomly divided into the train (2,794
images) and test (480 images) sets. To provide correct metrics, we cropped all
the faces by their detected bounding boxes in the preprocessing stage. All the
models in this section were trained on a train set, and the final results are
provided for the test set.

Metrics. We use Normalized Mean Error ($NME_{iod}$) that preserves the
relativity of the error regardless of the size of the image or the scale of
the face on it. It is commonly utilized in landmark detection [52, 7, 46], and
uses MAE as the basis and inter-ocular distance (distance between the outer
corners of the two eyes, IOD) for normalization:

$NME_{iod}=\frac{1}{M\cdot
N}\sum_{i=1}^{N}\sum_{j=1}^{M}\frac{\left\|{x_{i}}^{j}-{x^{\prime}_{i}}^{j}\right\|_{1}}{iod_{i}},$

where $M$ is the number of landmarks in the image, $N$ is the number of images
in the dataset, ${x_{i}}^{j}$ and ${x^{\prime}_{i}}^{j}$ — the coordinates of
the predicted and ground truth landmark respectively.

Experimental Setup. We used the Ensemble Landmark Detector (ELD) [39] model as
a landmark detector, which is justified by its high performance on the CatFLW.
We used the YOLOv8 [29] model as a face detector and the EfficientNetV2S [47]
model as a backbone for the ensemble. All other parameters are similar to the
original article.

The face detection model was trained using standard YOLOv8n training
parameters for 100 epochs. The ensemble models were trained for 300 epochs
using the mean squared error loss and the ADAM optimizer with a learning rate
of $10^{-4}$ and a batch size of 16.

To train the models, we used the Google Colab cloud service with the NVIDIA
TESLA V100 GPU.

Augmentation. We randomly applied different augmentations (rotation, color
balance adjustment, brightness and contrast modification, sharpness
alteration, application of random blur masks, and addition of random noise) to
the training data, doubling the size of the training set. Additionally, when
training ensemble models, we mirrored symmetrical regions (ears and eyes) in
order to further artificially increase the amount of training data.

###  4.1 Dataset Size

In order to examine how the size of the training dataset affects the accuracy
of landmark detection, we carried out a series of experiments, training the
ELD on random subsets of the training data (averaging the results for 5
iterations) and gradually increasing their size. Figure 3 indicates that the
detection accuracy is approaching a plateau when the model is trained on 65%
of the data (1800 images). However, the accuracy ceiling has yet to be
reached, indicating potential for further work into diversification and
expanding the dataset size. The ELD model trained on the whole training set
has a NME of 6.52. The detection examples are shown on Figure 4.

Figure 3: Impact of the training subset size on the normalised mean error. The
size of the subset is measured as a fraction of the total size of the training
data. Figure 4: Landmark detection examples on pre-cropped images from the
test set. Red — ground truth, blue — predictions.

###  4.2 Ear Types

While conducting experiments, it was observed that the accuracy of localizing
landmarks in the ears is noticeably lower compared to other regions (Table 2
shows the NME for different regions). This is presumably due to the diverse
shapes and lengths of ears across different breeds, which may include
previously unseen ear types and positions in the test set, particularly for
floppy or half-floppy ear types.

Region | Eyes | Nose | Ears
---|---|---|---
NME | 2.24 | 4.22 | 13.19
Table 2: Normalised mean error of landmark detection on the test set for
different face regions

To test our hypothesis, we divided the data into three subsets: one with erect
ears (pointy), another with hanging ears (floppy), and the third with other
types (half-floppy). When dividing, we were guided by the average breed ear
type in a relaxed state, obtaining a ratio of 31:50:19 for the training set
and 29:52:19 for the test set.

Despite the dataset’s predominance of breeds with floppy ear types, Table 3
shows that the accuracy of landmark detection on a test set for such dogs is
less than for dogs with erect and half-floppy ears. This can be explained by
the variety of lengths and positions of such ears.

Ear Type | % in train set | NME total | NME ears only
---|---|---|---
Pointy | 31 | 5.31 | 9.79
Floppy | 50 | 7.18 | 15.28
Half-floppy | 19 | 6.56 | 12.77
Table 3: Normalised mean error of landmark detection on the test set for dogs
with different ear types

###  4.3 Breeds

Due to the uneven presence of breeds in the training set, the accuracy of
landmark detection for rare breeds may be lower than for frequent breeds.
However, it would be incorrect to say that only the number of samples affects
the detection error value. The influence of the breed as an additional feature
on the accuracy of detection is apparent — many breeds have distinct fur
length and texture, as well as facial anatomy.

Figure 5 shows the image distribution of dogs of each breed in the training
set and the detection error for the same breeds in the test set. Based on the
distribution, it can be observed that breeds with long fur covering facial
features or extending on ears are the most difficult to detect accurately
(Irish Water Spaniel, Briard, Standard Poodle, Bedlington Terrier, Scottish
Deerhound, Komondor, Kerry Blue Terrier). Moreover, in the Ear Types
subsection, we demonstrated that detection has a high error on dogs with long,
floppy ears, resulting in low accuracy in detecting facial landmarks for
breeds with such ears (Basset, Redbone). It is also worth noting that some
breeds have non-obvious detection results. In some cases, this could be
explained by a significant variation in the position of the ears (Ibizan
Hound, Collie, Great Dane) or they being obscure (Chow), which can cause a
significant detection error.

Breeds with short smooth facial fur (Cardigan, Kelpie, Miniature Pinscher,
Dingo, Chihuahua, Malinois, etc.), for which facial features are clearly
distinguishable, have the lowest detection error on average.

Figure 5: The distribution of images in the training set for each breed of
dogs and the detection errors for the same breeds in the test set.

##  5 Conclusion

The domestic dog is a highly social animal with a complex, elaborate
communication system via facial signaling, which also underwent the most
extreme morphological changes due to domestication and selective breeding. In
this paper, we introduce a landmark scheme that is grounded in dog facial
musculature. We further present an annotated dataset and benchmark detection
results aimed at advancing the field of dog facial analysis. As expected, the
model has the most difficulties with ear landmarks, which can be improved by
enriching the dataset with more samples with diverse ear types. A similar
strategy can be taken to improve the performance on certain breeds. Our future
work includes utilizing the presented scheme and dataset to classify the
internal states of dogs.

The presented dataset holds great promise for canine cognition, emotion,
health, and welfare research. We hope that it will aid the scientific
community in utilizing AI-driven methods to deepen our understanding of our
best friends’ behavior and emotional world.

## Acknowledgements

The research was supported by the Data Science Research Center at the
University of Haifa. We thank Ephantus Kanyugi for his contribution with data
annotation and management. We thank Yaron Yossef for his technical support.

## References

  * [1] Tali Boneh-Shitrit, Marcelo Feighelstein, Annika Bremhorst, Shir Amir, Tomer Distelfeld, Yaniv Dassa, Sharon Yaroshetsky, Stefanie Riemer, Ilan Shimshoni, Daniel S Mills, et al.  Explainable automated recognition of emotional states from canine facial expressions: the case of positive anticipation and frustration.  Scientific reports, 12(1):22611, 2022.
  * [2] A Bremhorst, DS Mills, H Würbel, and S Riemer.  Evaluating the accuracy of facial expressions as emotion indicators across contexts in dogs.  Animal cognition, pages 1–16, 2021.
  * [3] Annika Bremhorst, Nicole A Sutter, Hanno Würbel, Daniel S Mills, and Stefanie Riemer.  Differences in facial expressions during positive anticipation and frustration in dogs awaiting a reward.  Scientific reports, 9(1):1–13, 2019.
  * [4] Sofia Broomé, Marcelo Feighelstein, Anna Zamansky, Gabriel Carreira Lencioni, Pia Haubro Andersen, Francisca Pessanha, Marwa Mahmoud, Hedvig Kjellström, and Albert Ali Salah.  Going deeper than tracking: A survey of computer-vision based recognition of animal pain and emotions.  International Journal of Computer Vision, 131(2):572–590, 2023.
  * [5] Erickson R. Nascimento Bruna Vieira Frade.  A two-step learning method for detecting landmarks on faces from different domains.  In Proceedings of the 2018 IEEE International Conference on Image Processing (ICIP), 2018.
  * [6] A. Bulat and G. Tzimiropoulos.  Binarized convolutional landmark localizers for human pose estimation and face alignment with limited resources.  ICCV, 2017.
  * [7] Xavier P. Burgos-Artizzu, Pietro Perona, and Piotr Dollár.  Robust face landmark estimation under occlusion.  In 2013 IEEE International Conference on Computer Vision, pages 1513–1520, 2013.
  * [8] Cátia Caeiro, Kun Guo, and Daniel Mills.  Dogs and humans respond to emotionally competent stimuli by producing different facial actions.  Scientific reports, 7(1):1–11, 2017.
  * [9] Cátia C Caeiro, Anne M Burrows, and Bridget M Waller.  Development and application of catfacs: Are human cat adopters influenced by cat facial expressions?  Applied Animal Behaviour Science, 2017.
  * [10] Jinkun Cao, Hongyang Tang, Hao-Shu Fang, Xiaoyong Shen, Cewu Lu, and Yu-Wing Tai.  Cross-domain adaptation for animal pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9498–9507, 2019.
  * [11] X. Chu, W. Ouyang, H. Li, and X. Wang.  Structured feature learning for pose estimation.  CVPR, 2016.
  * [12] Catia Correia-Caeiro, Kathryn Holmes, and Takako Miyabe-Nishiwaki.  Extending the MaqFACS to measure facial movement in japanese macaques (macaca fuscata) reveals a wide repertoire potential.  PLOS ONE, 16(1):e0245117, 2021.
  * [13] CVAT.ai Corporation.  Computer Vision Annotation Tool (CVAT), Nov. 2023.
  * [14] Kris A Descovich, Jennifer Wathan, Matthew C Leach, Hannah M Buchanan-Smith, Paul Flecknell, David Farningham, and Sarah-Jane Vick.  Facial expression: An under-utilized tool for the assessment of welfare in mammals.  ALTEX-Alternatives to animal experimentation, 34(3):409–429, 2017.
  * [15] Paul Ekman and Wallace V. Friesen.  Facial Action Coding System: Manual.  Palo Alto, Calif: Consulting Psychologists Press, 1978.
  * [16] Petra Eretová, Quanxiao Liu, Lucie Přibylová, Helena Chaloupková, Viktória Bakos, Rita Lenkei, and Péter Pongrácz.  Can my human read my flat face? the curious case of understanding the contextual cues of extremely brachycephalic dogs.  Applied Animal Behaviour Science, 270:106134, 2024.
  * [17] Marina C Evangelista, Ryota Watanabe, Vivian SY Leung, Beatriz P Monteiro, Elizabeth O’Toole, Daniel SJ Pang, and Paulo V Steagall.  Facial expressions of pain in cats: the development and validation of a feline grimace scale.  Scientific reports, 9(1):1–11, 2019.
  * [18] Marcelo Feighelstein, Lea Henze, Sebastian Meller, Ilan Shimshoni, Ben Hermoni, Michael Berko, Friederike Twele, Alexandra Schütter, Nora Dorn, Sabine Kästner, et al.  Explainable automated pain recognition in cats.  Scientific reports, 13(1):8973, 2023.
  * [19] Marcelo Feighelstein, Ilan Shimshoni, Lauren Finka, Stelio P. Luna, Daniel Mills, and Anna Zamansky.  Automated recognition of pain in cats.  Scientific Reports, 12, 2022.
  * [20] Kim Ferres, Timo Schloesser, and Peter A Gloor.  Predicting dog emotions based on posture analysis using deeplabcut.  Future Internet, 14(4):97, 2022.
  * [21] Lauren R Finka, Stelio P Luna, Juliana T Brondani, Yorgos Tzimiropoulos, John McDonagh, Mark J Farnworth, Marcello Ruta, and Daniel S Mills.  Geometric morphometrics for the study of facial expressions in non-human animals, using the domestic cat as an exemplar.  Scientific reports, 9(1):1–12, 2019.
  * [22] Lauren R Finka, Stelio PL Luna, Daniel S Mills, and Mark J Farnworth.  The application of geometric morphometrics to explore potential impacts of anthropocentric selection on animals’ ability to communicate via the face: The domestic cat as a case study.  Frontiers in Veterinary Science, page 1070, 2020.
  * [23] X. Guo, S. Li, J. Zhang, J. Ma, L. Ma, W. Liu, and H. Ling.  Pfld: A practical facial landmark detector.  CoRR, 2019.
  * [24] Jihun Hamm, Christian G Kohler, Ruben C Gur, and Ragini Verma.  Automated facial action coding system for dynamic analysis of facial expressions in neuropsychiatric disorders.  Journal of neuroscience methods, 200(2):237–256, 2011.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  CVPR, 2016.
  * [26] Charlie Hewitt and Marwa Mahmoud.  Pose-informed face alignment for extreme head pose variations in animals.  In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 1–6. IEEE, 2019.
  * [27] Marjo K Hytönen and Hannes Lohi.  Canine models of human rare disorders.  Rare Diseases, 4(1):e1006037, 2016.
  * [28] H. Jin, S. Liao, , and L. Shao.  Pixel-in-pixel net: Towards efficient facial landmark detection in the wild.  International Journal of Computer Vision, 2021.
  * [29] G. Jocher, A. Chaurasia, and J. Qiu.  YOLO by Ultralytics, 2023.
  * [30] Muhammad Haris Khan, John McDonagh, Salman H Khan, Muhammad Shahabuddin, Aditya Arora, Fahad Shahbaz Khan, Ling Shao, and Georgios Tzimiropoulos.  Animalweb: A large-scale hierarchical dataset of annotated animal faces.  CVRR, abs/1909.04951, 2019.
  * [31] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei.  Novel dataset for fine-grained image categorization.  In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
  * [32] W. Li, Y. Lu, K. Zheng, H. Liao, C. Lin, J. Luo, C.-T. Cheng, J. Xiao, L. Lu, C.-F. Kuo, and S. Miao.  Structured landmark detection via topology-adapting deep graph learning.  ECCV, 2020.
  * [33] Jiongxin Liu, Angjoo Kanazawa, David Jacobs, and Peter Belhumeur.  Dog breed classification using part localization.  In European conference on computer vision, pages 172–185. Springer, 2012.
  * [34] Yong Jae Lee M. Rashid, Xiuye Gu.  Interspecies knowledge transfer for facial keypoint detection.  CVPR, 2017.
  * [35] Evan L MacLean, Aubrey Fine, Harold Herzog, Eric Strauss, and Mia L Cobb.  The new era of canine science: reshaping our relationships with dogs.  Frontiers in Veterinary Science, page 762, 2021.
  * [36] George Martvel, Nareed Farhat, Ilan Shimshoni, and Anna Zamansky.  Catflw: Cat facial landmarks in the wild dataset.  arXiv preprint arXiv:2305.04232, 2023.
  * [37] George Martvel, Teddy Lazebnik, Marcelo Feighelstein, Lea Henze, Sebastian Meller, Ilan Shimshoni, Friederike Twele, Alexandra Schutter, Nora Dorn, Sabine Kastner, Lauren Finka, Stelio Luna, Daniel S Mills, Holger A Volk, and Anna Zamansky.  Automated pain recognition in cats using facial landmarks: Dynamics matter.  2023\.  in review.
  * [38] George Martvel, Teddy Lazebnik, Marcelo Feighelstein, Sebastian Meller, Ilan Shimshoni, Lauren Finka, Stelio Luna, Daniel S Mills, Holger A Volk, and Anna Zamansky.  Automated landmark-based cat facial analysis and its applications.  2023\.  in review.
  * [39] George Martvel, Ilan Shimshoni, and Anna Zamansky.  Automated detection of cat facial landmarks.  International Journal of Computer Vision, pages 1–16, 2024.
  * [40] Guillaume Mougeot, Dewei Li, and Shuai Jia.  A deep learning approach for dog face verification and recognition.  Lecture Notes in Computer Science, 2019.
  * [41] Giulia Pedretti, Chiara Canori, Sarah Marshall-Pescini, Rupert Palme, Annalisa Pelosi, and Paola Valsecchi.  Audience effect on domestic dogs’ behavioural displays and facial expressions.  Scientific Reports, 12(1):9747, 2022.
  * [42] Francisca Pessanha, Albert Ali Salah, Thijs van Loon, and Remco Veltkamp.  Facial image-based automatic assessment of equine pain.  IEEE Transactions on Affective Computing, 2022.
  * [43] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  CVPR, 2018.
  * [44] Courtney L Sexton, Colleen Buckley, Jake Lieberfarb, Francys Subiaul, Erin E Hecht, and Brenda J Bradley.  What is written on a dog’s face? evaluating the impact of facial phenotypes on communication between humans and canines.  Animals, 13(14):2385, 2023.
  * [45] Yifan Sun and Noboru Murata.  Cafm: A 3d morphable model for animals.  IEEE Winter Applications of Computer Vision Workshops (WACVW), 2020.
  * [46] Yi Sun, Xiaogang Wang, and Xiaoou Tang.  Deep convolutional network cascade for facial point detection.  In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pages 3476–3483, 2013.
  * [47] Mingxing Tan and Quoc V Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  CoRR, abs/1905.11946, 2019.
  * [48] Bridget M Waller, Kate Peirce, Cátia C Caeiro, Linda Scheider, Anne M Burrows, Sandra McCune, and Juliane Kaminski.  Paedomorphic facial expressions give dogs a selective advantage.  PLoS one, 8(12):e82686, 2013.
  * [49] Z. Xu, B. Li, Y. Yuan, and M. Geng.  Anchorface: An anchor-based facial landmark detector across large poses.  Thirty-Fifth AAAI Conference on Artifi- cial Intelligence, AAAI 2021, Thirty-Third Confer- ence on Innovative Applications of Artificial Intelli- gence, IAAI 2021, The Eleventh Symposium on Ed- ucational Advances in Artificial Intelligence, EAAI 2021, 2021.
  * [50] Heng Yang, Renqiao Zhang, and Peter Robinson.  Human and sheep facial landmarks localisation by triplet interpolated features.  CVRR, abs/1509.04954, 2015.
  * [51] Weiwei Zhang, Jian Sun, and Xiaoou Tang.  Cat head detection - how to effectively exploit shape and texture features.  ECCV, 2008.
  * [52] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Learning and transferring multi-task deep representation for face alignment.  CoRR, abs/1408.3967, 2014.