# Privacy-sensitive Objects Pixelation for Live Video Streaming

Jizhe Zhou  Department of Computer and Information Science, University of
MacauAvenida da Universidade,TaipaMacauChina999078<EMAIL_ADDRESS>,  Chi-
Man Pun  Department of Computer and Information Science, University of
MacauAvenida da Universidade,TaipaMacauChina<EMAIL_ADDRESS>and  Yu Tong  vivo
AI Lab, Shenzhen & Department of Computer and Information Science, University
of MacauAvenida da Universidade,TaipaMacauChina<EMAIL_ADDRESS>
(2020)

###### Abstract.

With the prevailing of live video streaming, establishing an online pixelation
method for privacy-sensitive objects is an urgency. Caused by the inaccurate
detection of privacy-sensitive objects, simply migrating the tracking-by-
detection structure applied in offline pixelation into the online form will
incur problems in target initialization, drifting, and over-pixelation. To
cope with the inevitable but impacting detection issue, we propose a novel
Privacy-sensitive Objects Pixelation (PsOP) framework for automatic personal
privacy filtering during live video streaming. Leveraging pre-trained
detection networks as the backbone, our PsOP is extendable to any potential
privacy-sensitive objects pixelation. Employing the embedding networks and the
proposed Positioned Incremental Affinity Propagation (PIAP) clustering
algorithm, our PsOP unifies the pixelation of discriminate and indiscriminate
pixelation objects through trajectories generation. In addition to the
pixelation accuracy boosting, experiment results on the streaming video data
we built show that the proposed PsOP can significantly reduce the over-
pixelation ratio and the human intervention in privacy-sensitive object
pixelation.

object pixelation; neural networks; PIAP clustering; live video-streaming

††journalyear: 2020††copyright: acmlicensed††conference: Proceedings of the
28th ACM International Conference on Multimedia; October 12–16, 2020; Seattle,
WA, USA††booktitle: Proceedings of the 28th ACM International Conference on
Multimedia (MM ’20), October 12–16, 2020, Seattle, WA, USA††price: 15.00††doi:
10.1145/3394171.3413972††isbn: 978-1-4503-7988-5/20/10††ccs: Security and
privacy Privacy protections††ccs: Theory of computation Unsupervised learning
and clustering††ccs: Computing methodologies Tracking††ccs: Computing
methodologies Scene understanding

##  1\. Introduction

(a) Established offline pixelation pipeline.

(b) Established offline pixelation pipeline.

Figure 1. Current pixelation methods v.s. PsOP under the same scene. The
leftmost clips are a clip of a streaming scene. Potential privacy-sensitive
objects are labeled in circles and rectangular and only circled objects shall
be pixelated. Tracklets rendered in gray ground require manual association in
(a). The color of the clusters in (b) represents the corresponding colored
objects. Pixelation results are shown in the rightmost clip.

Live video streaming has never been so popular as in the past two or three
years. Live video streaming, especially the outdoors, presents audiences
unprepared, unrehearsed filming of reality, and ramps up the sense of
immersion, unpredictability or nervousness. Real scenes are recorded, then
instantly broadcast to audiences mostly through streamers’ phone cameras and
the mobile network. Streaming activities join a symbiosis with the
popularization of smartphones, 5G networks, and cheaper communication costs
(Stewart and Littau, 2016). The primary host of live streaming activities
transformed from TV stations and news bureau to ordinary individuals. Every
rose has its thorn. Without imposed censorship or filters before broadcasting,
the pervasiveness of live streaming today arouses unprecedented violations in
the personal privacy protection field. Strict law or conservative religions
even commit those violations to crimes (Faklaris et al., 2016).

Actually, apart from the indifference of streamers and the absence of imposed
censorship, the handcrafted pixelation process is the main reason that leads
to privacy infringements amid video streaming. As a giant live-streaming
hosting service provider, YouTube is already aware of such urgent needs and
published its offline auto-pixelation tool on the latest YouTube Creator
Studio. As far as we know, current studies, including YouTube Studio and
Microsoft Azure, mainly focus on processing offline videos; whilst leave the
pixelation methods on the online live video streaming field underexplored. To
ensure the privacy rights and the sound development of the streaming industry,
proper pixelation methods shall be interpolated between the filming and
broadcasting of live video streaming. Transparency in terms of user experience
is also indispensable to keep the pixelation practical. Therefore, in this
paper, we devote to establish a pixelation method that generates automatic
personal privacy filtering during unconstrained streaming activities.

Most of the few existing offline pixelation methods adopt a similar tracking-
by-detection structure that assembles privacy-sensitive objects detectors
ahead of multi-object tackers. In such a structure, a Multi-Object Tracker
(MOT) aims at continuously locating an arbitrary target in a video with a
given bounding box in the target’s initial frame, while the detector is
responsible for giving the bounding box in the target’s initial frame. As
shown in Figure 1 (a), this structure intuitively follows the manual
pixelation pipeline and works well on fine-shot videos. Once a sensitive
object is spotted by the detector, it is tracked by the tracker till its
vanishment. This action sequence loops over for every detection. Allocating
Gaussian or other filters on the trajectories reaches the final pixelation.
Essentially, the object trajectories determine the final pixelation results.

In tracking-by-detection structure, trajectories are the joint effort of
trackers and detectors, and in fact, trackers and detectors are contained
independently. Comparing with conventional fine-shot videos, live streaming
videos always involve very few shot changes and the shaky camera. Recorded
often by hand-held cameras, shaky or jerky camera stands for frequent camera
shakes, abrupt streaming condition alteration, and noisy backgrounds. Under
such conditions, trackers still struggle standing up to scattered or drifted
tracklets in the long-term. Even were the long-term tracking accuracy somehow
resolved, tracking-by-detection structure also prone to corruption as its
performance decisively in accordance with detection accuracy. Trackers are
unable to function well with poor initiations. However, due to inadequate
training samples and the lack of comprehension for video contexts, false
positives and false negatives produced by the image-based detectors spring up
in streaming videos. Consequently, besides efficiency, blinking, drifting,
missing, or excessive mosaics are allocated everywhere when migrating existing
methods to pixelate live videos. Hence, blaming unacceptable mosaics mainly on
inaccurate detections, current methods are incapable of handling live video
streaming.

The other major drawback of the current pixelation workflow is the over-
pixelation problem. Introduced by the tracking algorithms and inherited by
current tracking-by-detection pixelation methods, the over-pixelation problem
occurs when the trackers insist on generating puzzling and unnecessary mosaics
for unidentifiable privacy-sensitive objects. Heavy or fully occlusion, and
massive motion blur are the common reasons for objects’ unidentifiable.
Tracking algorithms are designed to construct seamless or long enough
trajectories for objects in avoiding frequent ID-switch. Unlike tracking, in
the meantime of blocking the privacy-sensitive information from leaking,
pixelation tasks are dedicated to preserving the audience as many originals as
possible. Over-pixelation is an intrinsic problem while migrating tracking
algorithms to pixelation tasks.

To address the aforementioned issues, our Privacy-sensitive Objects Pixelation
(PsOP) introduces a brand-new framework for generating reliable pixelation
with real-time efficiency in live video streaming. To yield reliable
trajectories for pixelation, PsOP foremost copes with the inaccurate
detections caused by deep networks insufficiency and the lack of comprehension
for video contexts. PsOP divides the potential privacy-sensitive objects into
contexts-irrelevant Indiscriminating Pixelation Objects (IPOs) and contexts-
relevant Discriminating Pixelation Objects (DPOs) to alleviate contexts’
effects on detection. As their sensitiveness barely changes in scenes, IPOs,
including erotic images, trademarks, phone numbers or car plate numbers, etc.,
are liable to be handled through well-established sensitive objects detection
algorithms (Yu et al., 2016b). Conversely, DPOs mainly made up of faces and
texts are not to be solely dealt with detection networks. When the domain of
discriminating and indiscriminating pixelation objects overlaps (e.g., the
phone number is also a kind of text), IPOs are prioritized in claiming the
overlapped bounding box for tighter privacy protection. Depicted in Figure 1
(b), detected IPOs are smoothed and then pixelated.

The remaining Discriminating Pixelation Objects (DPOs) are the primary concern
of PsOP. The sensitiveness of instances belonging to DPOs changes according to
scenes or contexts. The varying sensitiveness of DPOs along with the inbuilt
network insufficiency cannot be simply solved through training on video data.
Such training assumes the detection, recognition, and video semantics
segmentation tasks share a unified network structure as well as demands labor-
intensive video data labeling. Considering the learning-based feature vectors
are in fact the central role for instances’ trajectories association, as shown
in Figure 1 (b), PsOP abnegates tracking-by-detection structure and employs
the detection, embedding, and clustering procedure to generate trajectories
under inaccurate detections.

In specific, for every class of DPOs, pre-trained detection and embedding
networks are leveraged to yield feature vectors for instances of the class.
Subsequently, we propose the Positioned Incremental Affinity Propagation
(PIAP) algorithm for clustering under inaccurate detection and embeddings. In
classic Affinity Propagation (AP), affinities are the distances between
feature vectors. A series of messages about affinities, denoted as
availabilities and responsibilities, are propagated among vectors to reach a
final consensus on clustering results. For PsOP, since an individual instance
of DPOs cannot appear at different positions within a frame, the detected
instances positioned in the same frame serve as negative samples with minimal
affinities. These minimized affinities revise the consensus through the
propagation of availabilities and responsibilities. The preference matrix is
also computed based on availabilities and responsibilities. Weak preference
indicates the detection is relatively isolated in the feature space; thereby,
excluded as outliers through propagation. Besides, as a single instance owns
similar expanding availability and responsibility matrices in adjacent frames,
positioned affinities are propagated in an incremental way. A run of PIAP
iterates until the consensus of clusters’ results is reached for all vectors.
Link the detection within the same cluster forms the trajectory of an
instance. Comparing with classic AP, while retaining the ability of clustering
under ill-defined cluster number, PIAP solves the noise-sensitive and time-
consuming problem of AP by introducing positioned affinities and incremental
propagation. Moreover, unlike gradually fitting trackers, detection, and
embedding based PIAP simultaneously solves the inherent over-pixelation
problem.

In order to evaluate the performance of the proposed PsOP and PIAP, we crawled
raw streaming videos from YouTube Live and Facebook Live platforms. These raw
videos contents are inspected and selected in detail to ensure the dataset
diversities. Tested on the live streaming video dataset (84261 labeled boxes
and 19372 frames) we collected and manually labeled, PsOP gains significant
pixelation accuracy boosting as well as retains much more non-sensitive
originals for audiences.

In summary, the main contributions of this paper are as follows:

  * •

We build the Privacy-sensitive Objects Pixelation (PsOP) framework for the
pixelation of privacy-sensitive objects in live video streaming. As far as we
know, PsOP is the first online method adopts the detection, embedding,
clustering procedures and solves the over-pixelation problem.

  * •

We proposed the Positioned Incremental Affinity Propagation (PIAP) clustering
algorithm to generate trajectories for inaccurately detected and sensitiveness
varying discriminating pixelation objects. The proposed PIAP spontaneously
handles the cluster number generation, cluster under unbalanced sample-size
problems, and further endows the classic AP with noise-resistance and time-
saving merits.

  * •

We built a live video streaming dataset to test the proposed PsOP framework
and to be available to public. Diverse streaming videos are collected through
live streaming platforms, and dense annotations on each frame are manually
labeled for the evaluation of the proposed method.

Figure 2. The proposed PsOP framework. Purple and blue lines respectively
indicate the pixelation process of discriminate pixelation objects, and
nonsensitive objects. Dash lines and boxes presents the extendability of PsOP
through pre-training.

##  2\. Related Works

To our best knowledge, face pixelation in live video streaming has not been
well studied before. Current research of privacy-sensitive objects detection
and pixelation focus on image-level (Yu et al., 2016b, 2017). (Yu et al.,
2016b) established a multi-task learning based algorithm to extract the user-
dependent privacy-sensitive objects. User blurred images are collected and fed
to the multi-task CNNs for finding privacy-sensitive objects through a
decision-tree shaped loss-function. Put the availability of such data in
videos and accuracy aside, the relatively shallow structure along with the
decision-tree shaped loss-function are unaware of the sensitiveness variations
of discriminating pixelation objects in videos. The most widely applied
commercial offline pixelation tools developed by YouTube Studio (Help, 2020)
and Microsoft Azure (Juliako, 2020) adopts similar detection networks in their
tracking-by-detection structure. However, these works do offer us an
innovative way of spotting the IPOs.

As we mentioned in the introduction, current pixelation methods are incapable
of functioning under poor detection. Referring to existing techniques, we
firstly tried to solve the detection flaws in an end-to-end way by replacing
the 2D deep CNNs with its extension 3D CNNs (Ji et al., 2013) or CNNs with
attention mechanisms (Li et al., 2018; Zhu et al., 2018). Taking an entire
video as input, 3D CNNs and the attentions consider spatial-temporal
information simultaneously. Achieving great success in video segmentation
(Yue-Hei Ng et al., 2015; Karpathy et al., 2014), and action recognition areas
(Ji et al., 2013; Tran et al., 2015), they are expected to relief the
detection burden. However, tested on the live streaming dataset we collected,
both 3D CNNs and the attention models are quite time-costing and extremely
sensitive to the video quality and length. Furthermore, without sufficient
training data, as designed to extract high-level abstractions of video
contexts, 3D CNNs cannot precisely handle the multi-task regression on
individual frame-level. Therefore, the mosaics generated by both models always
blink during live-streaming.

Another proper way is to exploit multi-object tracking (MOT) algorithms that
can contend with poor initiations. Although MOT are claimed to be thoroughly
studied (Zhang and Gomes, 2014), their tracking accuracy is still challenging
in unconstrained videos. When reviewing state-of-the-art trackers, two main
categories of conventional trackers can be found: offline and online trackers.
The former category of trackers assumes that object detection in all frames
has already been conducted, and clean tracklets are achieved by linking
different detections and tracks in offline mode (Berclaz et al., 2011). This
property of offline trackers allows for global optimization of the path (Zamir
et al., 2012) but makes them inappropriate in dealing with live streaming.

A typical online MOT algorithm aims at continuously locating an arbitrary
target in a video with a given bounding box in the target’s initial frame.
State-of-the-art online MOT algorithms like KCF (Henriques et al., 2014) and
ECO (Danelljan et al., 2017) are implemented through continuous motion
prediction and online learning based on the correlation filter. Varieties of
tracking related works are conducted, however, they concentrate on the
association of the learning-based appearance features among shots changes.
Tracking problems give tacit consent to the bounding box precisions, and
intrinsic detection inaccuracy is seldom mentioned or discussed. Only very few
works (Yu et al., 2016a) take the detection accuracy into consideration, and
they merely acknowledge the false negatives.

##  3\. Proposed Method

The procedure of the proposed Privacy-sensitive Objects Pixelation (PsOP) is
shown in Figure 2. The process of contexts-irrelevant Indiscriminating
Pixelation Objects (IPOs) are omitted since they are pixelated without
generating trajectories. Live streaming videos are sliced into video segments
with a fixed size for better accuracy. Then, frame-wise detections are applied
through detectors pre-trained on image datasets. Detectors, along with
corresponding embedding networks for every class of contexts-relevant
Discriminating Pixelation Objects (DPOs) (not limited to face and text), are
applied in parallel. Similarly, embeddings yielded by the same detector and
embedding network are feed to the PIAP clustering. The parallel PIAP
associates the same instance across frames into the same cluster, and
sequential link within a cluster forms the trajectory. As not all DPOs are
sensitive in a particular scene, sensitive thesaurus (Bollegala et al., 2012)
or minor manual specification are appended to filter the trajectories of DPOs
further. Gaussian filters are imposed on the final trajectories for pixelation
and then streaming to the audience.

###  3.1. Video Segments

The proposed PsOP leverages image-based pre-trained detection networks. Dash
lines and boxes in Figure 2 shows the extendable detection of IPOs and DPOs
through user-defined pre-training datasets. As false positives and negatives
are common in detection, we design a buffer section at the beginning of each
live streaming to promote the accuracy of trajectories without affecting the
audience experience. With the PsOP’s real-time efficiency, we can stack every
$\mathcal{N}$ frames into a short video segment by demanding a
$(2*\mathcal{N})$ frames buffering section at the very beginning of live video
streaming without causing discontinuities in broadcasting. Video segments
slice the typical hours-long live streaming into numerous segments in seconds
level. Transmitted to a lag at the begging, the conflict between accuracy and
efficiency is greatly alleviated. Trajectories of privacy-sensitive objects
could be smoothed within and across segments. Considering the primal latency
brought by data communication and video compression, such a buffering latency
appended is acceptable to users.

###  3.2. Indiscriminate Pixelation Objects

Then, the Indiscriminating Pixelation Objects (IPOs) are firstly detected in
video segments. We adopt the (Yu et al., 2016b) as the detection network here.
Training is conducted jointly on NPDI (Avila et al., 2013), METU (Tursun et
al., 2017), and openALPR. The multi-task learning network is trained to handle
the detection of erotic image, trademark, and plate numbers simultaneously. As
in Figure 2, video segments are directly fed to the detector of indiscriminate
pixelation objects colored in purple. Gaussian smooth among five consecutive
frames is applied for compensating false negatives. Within a segment, the
detection which has less than five bounding boxes with an overlapping ratio
IOU$\leq\varepsilon$ is eliminated as false positives to avoid the blinking of
mosaics. Afterward, the trajectories of indiscriminate pixelation objects are
established and ready for pixelation. To avoid the domain overlapping of IPOs
and DPOs, bounding boxes detected by both detectors with an IOU$>\varepsilon$
are categorized to IPOs.

###  3.3. Discriminate Pixelation Objects

To elucidate the details of proposed pixelation method for Discriminate
Pixelation Objects (DPOs), faces and texts are used in the following as
tangible instances. Following the embedding then clustering pixelation process
for DPOs, in this paper, we use MTCNN (Zhang et al., 2016), and CosFace (Wang
et al., 2018) to process every frame in turn. The application of MTCNN and
CosFace considers the convenience for latter cosine similarity in PIAP. The
detection and embedding networks could be substituted by other state-of-the-
art methods. MTCNN accepts arbitrary size inputs and detects face larger than
$12*12$ pixel. CosFace generates $512$ dimension feature vector for each
detected face. Faces are aligned to the frontal pose through the affine
transformation before embedding. Similarly for texts, Textboxes (Liao et al.,
2017) and FastText (Joulin et al., 2016) are used in texts detection and
embedding. TextBoxes receives input images containing texts with an arbitrary
scale. A bit close to MTCNN, frames are rescaled to $300*300$ for efficiency,
and Non-Maximum Suppression (NMS) are also leveraged to boost the accuracy.
FastText is a C-BOW like fast text classification network, and the result of
the second last layer before softmax regression is extracted as embedding. The
embedding dimension of FastText is defined to $128$. The clustering for faces
and texts is conducted in parallel.

The proposed PIAP is then activated to connect the same faces or the same
piece of text across frames according to face or text vectors. Do notice that
the efficacy of PIAP remains the same in establishing trajectories for other
objects of this type. DBSCAN (Ester et al., 1996) and Affinity Propagation
(AP) (Frey and Dueck, 2007) are the candidates under unpredictable cluster
numbers. As noisy detection results are common in videos and the data-size for
clustering is also unbalanced, the density-based DBSCAN is excluded. To revise
the false detection, variation of embedded vectors, and boost the clustering
speed, we employ the position information and incremental clustering in the
proposed PIAP. Sequential links within each cluster form the trajectory of
each face and text, and likewise, Gaussian smooth is applied within a segment
before pixelation.

####  3.3.1. Affinity Propagation (AP)

Similar to traditional clustering algorithms, the first step of classic AP is
the measurement of the distance between data nodes, denoted as the
similarities. Following the common notation in AP, $i$ and $k$ ($i,k\in
R^{D}$, $D=512$ for face and $D=128$ for text) are two of the data nodes. $S$
is the similarity matrix stores the similarities between every two nodes.
$S(i,k)$ is the element on row $i$, column $k$ of $S$ and $S(i,k)$ denotes the
similarity between data node ${i}$ and ${k}$, thereby indicating how well the
data node $k$ is suited to be the exemplar for data point $i$. Similar
notations are also used in below.

The core of AP is that a series of responsibilities $R(i,k)$ and
availabilities $A(i,k)$ messages are passed among all data nodes to reach the
consensus on exemplars’ selection. $R$ and $A$ are the responsibility and
availability matrix. $i$ passes $R(i,k)$ to its potential exemplar $k$
indicating the current willingness of $i$ choosing $k$ as its exemplar
considering all the other potential exemplars. Correspondingly, $k$ responds
$A(i,k)$ to $i$ update the current willingness of $k$ accepting $i$ as its
member considering all the other potential members. The sum of $R(i,k)$ and
$A(i,k)$ can directly give the fitness for choosing $k$ as the exemplar of
$i$. The consensus of the sum-product message passing process ($R(i,k)$ and
$A(i,k)$ remain the same after iterations) stands for the final agreement of
all nodes in the selection of exemplars and the association of clusters is
reached. Apart from the ill-defined cluster number, AP is not sensitive to the
initialization settings; selects real data nodes as exemplars; allows
asymmetric matrix as input; is more accurate when measured in the sum of
square errors. Therefore, considering the subspace distribution (least square
regression), AP is effective in generating robust and accurate clustering
results for high dimension data like face vectors.

The governing equations for message passing in AP are:

(1) $R(i,k)\leftarrow S(i,k)-\max_{{k^{\prime}},s.t.{k^{\prime}}\neq
k}\\{A(i,k^{\prime})+S(i,{k^{\prime}})\\}$ (2)
$A(i,k)\leftarrow\min\bm{\\{}{0,R(k,k)+\sum_{{i^{\prime}},{i^{\prime}}\notin{\\{i,k\\}}}\max\\{0,R({i^{\prime}},k)\\}\bm{\\}}}$

Equation (3) is used to fill in the elements on the diagonal of the
availability matrix:

(3) $A(k,k)\leftarrow\sum_{i^{\prime},s.t.i^{\prime}\neq
k}\max\\{0,R(i^{\prime},k)\\}$

Update responsibilities and availabilities according to (1), (2) and (3) till
convergence, then the criterion matrix $C$ which holds the exemplars is the
sum of the $A$ and $R$ at each location.

(4) $C(i,k)\leftarrow R(i,k)+A(i,k)$

The highest value of each row of the criterion matrix is designated as the
exemplar.

####  3.3.2. Positioned Incremental Affinity Propagation (PIAP)

Clustering builds the connection of the same face across frames since its
result could be facilely corrected with some intervention. With the results of
face detection and recognition, the proposed PIAP process in a segment-wise
way to group the faces within and across segments simultaneously. Apparently,
the longer segments offer more information but also incur more noise and
reduce efficiency. In this paper, the proposed PsOP cuts every 150-frames into
a segment and leverages the whole existing context to reach accurate and fast
face pixelation. Define a data stream ${Z=\\{Z_{1},Z_{2},...,Z_{n}\\}}$ is
sequentially collected face/text feature vectors of the current video segment.
i.e. $Z$ is an $512*n$ matrix with $512$ dimension feature vector.

For the normalization in the PIAP, cosine similarity is brought for
measurements. As an instance of DPOs (like a person’s face) can only appear at
one position in a single frame, instances that belong to the same frame are
set to the minimum similarity value $-1$. Let $j$ stands for the other
instances that belong to the same frame as $i$. The similarity matrix can be
generated as:

(5) $S(i,k)=\left\\{\begin{array}[]{lr}\frac{i\cdot
k}{\|i\|\|k\|}-1,\quad\mbox{ if }\ k\notin{j}\\\ \\\ -1,\quad\mbox{ if }\ k\in
j\end{array}\right.$

Moreover, in (2), the left side of the equation is the accumulated evidence of
how much $k$ can stand for itself ($R(k,k)$), and how many others $k$ is also
responsible to stands for
($\sum_{{i^{\prime}},{i^{\prime}}\notin{\\{i,k\\}}}\max\\{0,R({i^{\prime}},k)\\}$).
Every positive responsibilities of $k$ contributes to $A(i,k)$. Therefore,
according to (5), $A(i,k)$ should not accumulate $j$’s choice as the evidence
for representing $i$. One step further, the choice of $j$ will actually repel
$i$ in availability, and the message passing of $A(i,k)$ shall be rewritten
as:

(6) $\displaystyle A(i,k)\leftarrow\min\bm{\\{}0,R(k,k)+$
$\displaystyle\sum_{{i^{\prime}},{i^{\prime}}\notin{\\{i,j,k\\}}}\max\\{0,R({i^{\prime}},k)\\}-\sum_{{i^{\prime}},{i^{\prime}}\in{\\{j\\}}}\max\\{0,R({i^{\prime}},k)\\}\bm{\\}}$

(5), (6) ensures $i$ and $j$ are mutual exclusive in the clustering process.

After the revision of the position information, the challenge for extending AP
into an incremental way is that the data nodes received at different
timestamps stay at varying statuses with disproportionate responsibilities and
availabilities value.

However, video frames are strongly self-correlated data. We could assign a
proper value to newly-arrived vectors according to the ones in the previous
segment without affecting the clustering purity. The embedded vectors of a
particular person or text shall stay close with each other in the feature
space across different frames. Thus, our incremental AP algorithm is proposed
based on the fact that if two detected faces/texts are in adjacent segments
and refer to the same person/the same piece of texts, they should not only be
clustered into the same group but also have the same responsibilities and
availabilities. Such fact is not well considered in past studies of
incremental affinity propagation (Sun and Guo, 2014; Wang et al., 2013).

Following the common notations in AP, the similarity matrix is denoted as
$S_{t-t^{\prime}}$ at time $t-t^{\prime}$ with
($M_{t-t^{\prime}}*M_{t-t^{\prime}}$) dimension where
$(t-t^{\prime})=\frac{\mathcal{N}}{FPS}$. And the responsibility matrix and
availability matrix at time $t-t^{\prime}$ are $R_{t-t^{\prime}}$ and
$A_{t-t^{\prime}}$ with a same dimension as $S_{t-t^{\prime}}$. Then, the
update rule of $R_{t}$ and $A_{t}$ respect to $R_{t-t^{\prime}}$ and
$A_{t-t^{\prime}}$ can be written as:

(7) $R_{t}(i,k)=\left\\{\begin{array}[]{lr}R_{t-t^{\prime}}(i,k),\quad i\leq
M_{t-t^{\prime}},k\leq M_{t-t^{\prime}}\\\ \\\
R_{t-t^{\prime}}(i^{\prime},k),\quad i>M_{t-t^{\prime}},k\leq
M_{t-t^{\prime}}\\\ \\\ R_{t-t^{\prime}}(i,k^{\prime}),\quad i\leq
M_{t-t^{\prime}},k>M_{t-t^{\prime}}\\\ \\\ 0,\quad
i>M_{t-t^{\prime}},k>M_{t-t^{\prime}}\end{array}\right.$

Note that the dimension of three matrices is increasing with time.
$R_{t}(i,k)$ is the newly arrived face vectors of a segment at time $t$.
$M_{t-t^{\prime}}$ stands for the amount of faces at time $t-t^{\prime}$.

(8) $i^{\prime}=\arg\max_{i^{\prime},i^{\prime}\leq
M_{t-t^{\prime}}}\left\\{S(i,i^{\prime})\right\\}$

Easily, the $A_{t}$ could be updated through

(9) $A_{t}(i,k)=\left\\{\begin{array}[]{lr}A_{t-t^{\prime}}(i,k),\quad i\leq
M_{t-t^{\prime}},k\leq M_{t-t^{\prime}}\\\ \\\
A_{t-t^{\prime}}(i^{\prime},k),\quad i>M_{t-t^{\prime}},k\leq
M_{t-t^{\prime}}\\\ \\\ A_{t-t^{\prime}}(i,k^{\prime}),\quad i\leq
M_{t-t^{\prime}},k>M_{t-t^{\prime}}\\\ \\\ 0,\quad
i>M_{t-t^{\prime}},k>M_{t-t^{\prime}}\end{array}\right.$

Denote $z_{p}^{q}=\\{z_{1}^{q},z_{2}^{q},z_{3}^{q}...z_{p}^{q}\\}$ as the set
of all $p$ vectors extracted in segment $q$. The full process of PIAP
algorithm can be summarized as Algorithm 1.

Algorithm 1 Positioned Incremental Affinity Propagation

Input: $R_{t-t^{\prime}}$,$A_{t-t^{\prime}}$,$C_{t-t^{\prime}},z_{p}^{q}$

Output: $R_{t},A_{t},C_{t}$

1: while $q$ is not end of a live-streaming do

2: Compute similarity matrix according to (5).

3: if $q$ is the first video segment of a live-stream then

4: Assign zeros to all responsibilities and availabilities.

5: else

6: Compute responsibilities and availabilities for $z_{p}^{q}$ according to
equation (7), (8) and (9).

7: Extend responsibilities matrix $R_{t-t^{\prime}}$ to $R_{t}$, and
availabilities $A_{t-t^{\prime}}$ to $A_{t}$.

8: end if

9: Message-passing according to equation (1), (6) and (3) until convergence.

10: Compute exemplars and clustering results $C_{t}$ as equation (4).

11: end while

Handled by PIAP, faces, texts, and alike DPOs could be intra-class
distinguished. Cluster results are smoothed similar as the IPOs for the
exclusion of false negatives and positives. Faces and texts within a cluster
are then linked sequentially to form a trajectory.

###  3.4. Pixelation on Trajectories of Privacy-sensitive Objects

In live video streaming, the appeared non-streamers are much more than
streamers but own a far smaller sample-size for each non-streamer. Thus, as in
the rightmost part of Figure 1, with trajectories built by PIAP, manual
specification (like screen touching) is involved solely on the face of a newly
appeared streamer. Also, texts are further filtered through checking the
sensitive thesaurus based on (Bollegala et al., 2012). With the rolling of
PIAP, the previously specified cluster is capable of guiding the future
aggregation of the same faces or texts.

The trajectories of the faces of the non-streamers, the sensitive texts along
with IPOs are gathered as the trajectories of privacy-sensitive objects.
Gaussian filters are applied on these trajectories for pixelation before
broadcasting to the audience.

##  4\. Experiments and Discussion

Table 1. Details of the live video streaming dataset

Dataset |  | Quantity
---
of videos
Category | Resolution |  | People occurred
---
Frames |  | Privacy-sensitive
---
objects Labels*
| IPOs
---
Labels*
| DPOs
---
Lables*
$HS$ | 4 | a,b,c,d | 720p/1080p |  $\gg$ 2 | 4133 | 27337 | 6255 | 21082
$LS$ | 8 | a,b,c,d | 360p/480p |  $\gg$ 2 | 4680 | 23869 | 3975 | 19894
$LN$ | 4 | a,b,c,d | 360p/480p |  $\leq$2 | 5692 | 16994 | 7946 | 9048
$HN$ | 4 | a,b,c,d | 720p |  $\leq$2 | 4867 | 16061 | 10805 | 5256

  * •

*: the value amounts the occurrences of the relative objects. Meaning the same object is repeatedly counted in different frames.

Table 2. Pixelation results on collected live video streaming dataset

Method | SOPA$\uparrow$ | SOPA$\uparrow$ | SOPA$\uparrow$ | SOPA$\uparrow$ | SOPP$\uparrow$ | SOPP$\uparrow$ | SOPP$\uparrow$ | SOPP$\uparrow$ | Entire Dataset
---|---|---|---|---|---|---|---|---|---
($HS$) | ($LS$) | ($HN$) | ($LN$) | ($HS$) | ($LS$) | ($HN$) | ($LN$) |  | MP$\uparrow$
---
(frames)
OPR$\downarrow$ | FPS$\uparrow$
YouTube (Help, 2020) | 0.61 | 0.57 | 0.68 | 0.57 | 0.70 | 0.64 | 0.78 | 0.63 | 238 | 0.44 | $N/A$
Azure (Juliako, 2020) | 0.63 | 0.57 | 0.66 | 0.56 | 0.71 | 0.66 | 0.77 | 0.66 | 203 | 0.43 | $N/A$
KCF (Henriques et al., 2014) | 0.59 | 0.52 | 0.68 | 0.61 | 0.69 | 0.63 | 0.79 | 0.70 | 166 | 0.47 |  $>$100
ECO (Danelljan et al., 2017) | 0.57 | 0.61 | 0.69 | 0.66 | 0.66 | 0.75 | 0.79 | 0.74 | 178 | 0.41 | 30~50
| PsOP
---
0.73 | 0.69 | 0.75 | 0.71 | 0.80 | 0.79 | 0.84 | 0.83 | 387 | 0.21 | 18 ~30

  * •

”$\uparrow$”$\&$”$\downarrow$”stand for the higher the better and the lower
the better respectively.

Table 3. Pixelation results of DPOs on live video streaming dataset

Method | SOPA$\uparrow$ | SOPA$\uparrow$ | SOPA$\uparrow$ | SOPA$\uparrow$ | SOPP$\uparrow$ | SOPP$\uparrow$ | SOPP$\uparrow$ | SOPP$\uparrow$ | Entire Dataset
---|---|---|---|---|---|---|---|---|---
($HS$) | ($LS$) | ($HN$) | ($LN$) | ($HS$) | ($LS$) | ($HN$) | ($LN$) |  | MP$\uparrow$
---
(frames)
OPR$\downarrow$
YouTube (Help, 2020) | 0.45 | 0.42 | 0.56 | 0.49 | 0.53 | 0.47 | 0.77 | 0.63 | 238 | 0.56
Azure (Juliako, 2020) | 0.43 | 0.47 | 0.54 | 0.53 | 0.50 | 0.53 | 0.70 | 0.68 | 203 | 0.54
KCF (Henriques et al., 2014) | 0.35 | 0.32 | 0.38 | 0.31 | 0.41 | 0.40 | 0.44 | 0.40 | 113 | 0.64
ECO (Danelljan et al., 2017) | 0.27 | 0.28 | 0.34 | 0.31 | 0.37 | 0.39 | 0.41 | 0.40 | 148 | 0.59
| PsOP
---
(MTCNN+CosFace)
0.63 | 0.60 | 0.71 | 0.66 | 0.80 | 0.77 | 0.86 | 0.85 | 362 | 0.34
| PsOP
---
(Pyramid+ArcFace)
0.63 | 0.62 | 0.71 | 0.65 | 0.80 | 0.79 | 0.86 | 0.85 | 387 | 0.34

  * •

”$\uparrow$”$\&$”$\downarrow$”s.tand for the higher the better and the lower
the better respectively.

###  4.1. Dataset

As the privacy-sensitive objects pixelation problem in live video streaming
has not been studied before, there is no available dataset or benchmark tests
for reference. We collected and built a video live streaming video dataset
from YouTube and Facebook platforms and manually labeled the sensitive objects
for the experiments. In total, 20 streaming videos come from 4 different
streaming categories (a:street interview; b:street streaming; c:flash
activities; d:dancing) of 19372 frames are manually labeled and will be
available to public. As demonstrated in Table 1, these videos are further
divided into four groups according to the broadcasting resolution, and the
number of people showed up in streaming. Live streaming videos with at least
720p resolution are marked as high-resolution $H$, and the rest are low-
resolution $L$. Similarly, the videos contain more than two people are
sophisticated scenes $S$, and the rest are naive ones $N$.The number of
labeled bounding boxes regarding privacy-sensitive objects, IPOs, and DPOs are
also listed to exhibit the dense annotation.

###  4.2. Parameters

All the parameters remain unchanged for entire live-streaming video tests.
$\mathcal{N}=150$ for every segment contains 150 frames. 10 seconds for
buffering as $2*\mathcal{N}/30$. CosFace resizes the cropped face to $112*96$;
weight decay is 0.0005. The learning rate is 0.005 for initialization and
drops at every 60K iterations. The damping factor for PIAP is default 0.5 as
it in AP. Detection threshold for MTCNN is [0.7,0.8,0.9]. The IOU rate
$\varepsilon$ is [0.7].

###  4.3. Evaluation Metrics

Setting our manual pixelation annotations in the collected video live
streaming dataset as the baseline, we migrated and adopt the algorithms
including ECO and KCF tracker, currently applied commercial tools like YouTube
Studio and Microsoft Azure, and the proposed PsOP for comparison. Sensitive
Objects Pixelation Accuracy (SOPA), Sensitive Objects Pixelation Precision
(SOPP), and Most Pixelated (MP) frames are the most concerned metrics.
Specifically:

${}SOPA=1-{\frac{\sum_{t}(m_{t}+fp_{t}+mm_{t})}{\sum_{t}{g_{t}}}}$

where $m_{t}$, $fp_{t}$, $mm_{t}$, $g_{t}$ correspond to the missed
pixelation, false positives in pixelation, missmatched pixelation, total
pixelation in frame $t$. SOPA reflects the overall performance of the PsOP.

$SOPP={\frac{\sum_{i,t}d_{i,t}}{\sum_{t}{c_{t}}}}$

where $d_{i,t}$ is the bounding box overlapping ratio of pixelated sensitive
object $i$ with its labelled ground truth. $c_{t}$ is the total number of
matched pixelation in frame $t$. Therefore, SOPP states the degree of drifting
and pixelation precision. Over-Pixelated Ratio (OPR) is another important
metric indicating the degree of over-pixelation problem.

$OPR=\sum_{t}(fp_{t})/{\sum_{t}{c_{t}}}$

###  4.4. Ablation Study

In Table 2, all the algorithms get their best performance on the High-
resolution Naive-scene ($HS$) sub-dataset.KCF and ECO here are optimized for
privacy object tracking tasks with detection based calibration. To conduct the
comparisons, we activated manual specification on occurrences of sensitive
objects for the first four algorithms. However, affected by the noisy
detection results, KCF styled online learning algorithms are maladjusted in
video live streaming. Although ECO applies higher dimension features than KCF,
the SOPA of ECO is lower than KCF. Overall, our PsOP achieves better
performance in terms of SOPA, SOPP, MP, and OPR. Particularly, PsOP noticeably
increases most pixelated frames and decrease the over-pixelation ratio on the
entire dataset, indicating that PsOP can construct more robust trajectories
and avoid pixelation if the object is temporally not identifiable. The high
value in SOPP suggests the trajectory drifting in PsOP is relatively rare or
happens at a later time.

Table 3 shows the pixelation results solely on DPOs. Also, manual
specification is activated for the first four methods. PsOP gains a surpassing
pixelation accuracy and proves the proposed embedding and PIAP clustering
framework is sharp in dealing with DPOs. Listed at the bottom line of Table 3,
the boxed PsOP substitutes the detection and embedding network with more
advanced PrimidBox (Tang et al., 2018), ArcFace (Deng et al., 2019). The last
section shows the network selection is actually not that important for PIAP,
as long as they are deep architectured, well-tunned networks.

Table 4. Clustering purity and efficiency for faces Dataset | Method | Purity$\uparrow$ |  | Cluster nubmer
---
(Clustered/Truth)
Time(s)$\downarrow$
$HS$ | AP | 0.81 | 19/17 | 1.82
PAP | 0.89 | 17/17 | 1.82
PIAP | 0.89 | 17/17 | 0.07
$LS$ | AP | 0.72 | 37/43 | 3.05
PAP | 0.84 | 41/43 | 3.05
PIAP | 0.83 | 41/43 | 0.13
$HN$ | AP | 0.86 | 6/6 | 0.75
PAP | 0.93 | 6/6 | 0.75
PIAP | 0.93 | 6/6 | 0.04
$LN$ | AP | 0.89 | 7/7 | 0.80
PAP | 0.92 | 7/7 | 0.80
PIAP | 0.91 | 7/7 | 0.03

$\begin{array}[]{ccc}\vspace{-0.2em}\includegraphics[width=80.8058pt]{img/11.jpg}&\includegraphics[width=80.8058pt]{img/12.jpg}&\includegraphics[width=80.8058pt]{img/15.jpg}\\\
\vspace{-0.2em}\includegraphics[width=80.8058pt]{img/11y.jpg}&\includegraphics[width=80.8058pt]{img/12y.jpg}&\includegraphics[width=80.8058pt]{img/15y.jpg}\\\
\vspace{-0.2em}\includegraphics[width=80.8058pt]{img/11k.jpg}&\includegraphics[width=80.8058pt]{img/12k.jpg}&\includegraphics[width=80.8058pt]{img/15k.jpg}\\\
\vspace{-0.2em}\includegraphics[width=80.8058pt]{img/11p.jpg}&\includegraphics[width=80.8058pt]{img/12p.jpg}&\includegraphics[width=80.8058pt]{img/15p.jpg}\end{array}$

Figure 3. Qualitative comparison of YouTube (mosaics in rectangular), KCF
(mosaics in hexagons), and PsOP (mosaics in circles). Green boxes indicate
human intervention is needed for specification, yellow are the auto-generated
mosaics, red ones are the over-pixelated area, and blue ones are the miss-
pixelated area.

Table 4 is the face clustering performance of classic AP, PAP (AP after
position revision), and PIAP on the collected live streaming dataset.
Comparing AP with PAP, the position information of face vectors significantly
boosts the purity of the clustering result. According to the results of PIAP
and AP, our incremental message passing algorithm solves the time-consuming
problem of AP almost without affecting the purity as we stated before. The
value of the Time column is the seconds needed to reach an iteration of
consensus.

###  4.5. Qualitative Comparison

Qualitative comparisons among the YouTube offline pixelation tool, migrated
KCF, and PsOP are shown in Figure 2. Snapshots are shoot from left to right.
IPOs, including the trademark and the car plate number, and DPOs, including
faces and texts, are contained in the scenes. The first row is the original
scene in which a girl streamer on the street is waving hands to greeting the
audience. The 2nd, 3rd, and 4th rows are respectively the pixelation results
after YouTube, KCF, and PsOP. Besides the significant differences in red and
blue boxes indicating accuracy, the green boxes in the leftmost snapshot
demonstrate the huge gap of required manual work on the initialization. Only
in PsoP, privacy-sensitive objects are auto-detected and categorized, and DPOs
are clustered and pixelated with minimum intervention.

###  4.6. Efficiency

Training and testing are conducted on our i7-7800X, 64G RAM and dual Nvidia
GTX 1080 machine. The main cost of FPVLS is on the face embedding algorithm.
The embedding for one face takes 10-15ms on our i7-7800X, dual GTX1080, 32G
RAM machine. The face detection, including compensation for a frame, takes
3ms. Another time-costing part is the initialization of PIAP. The
initialization process takes 10-30ms depending on the case. Furthermore, each
incremental propagation loop takes 3-5ms. So PsOP can satisfy the real-time
efficiency requirement. In the cases of extreme circumstances that the live
video streams contain many faces or other privacy sensitive objects, we can
reduce the sampling rate of the video frames to improve the efficiency of the
detection and recognition processes.

##  5\. Conclusion

In this paper, a novel extendable and unified framework using Privacy-
sensitive Objects Pixelation (PsOP) is proposed for live video streaming. The
proposed PsOP shows significant improvements in pixelation accuracy,
precision, and over-pixelation ratio compared with other offline pixelation
algorithms. Also, our PsOP requires minimum human intervention on the
pixelation of discriminating pixelation objects represented by faces and
texts. Leveraging the trained deep CNNs, the proposed PsOP manages to solve
the pixelation problems in video live streamings through the proposed
Positioned Incremental Affinity Propagation (PIAP) clustering. The PIAP deals
with the ill-defined cluster number issue and boosts the accuracy of classic
Affinity Propagation (AP) through position information. Moreover, the accuracy
of AP is not affected by the incremental processing. However, the low-
resolution video streamings are still quite challenging because the image
resolution affects much of the detection and embedding network performance.
Designed and trained on high-quality stationary images, these deep neural
networks can drop to a deficient performance and break the robustness of the
PsOP. Our future work will focus on the improvement of accuracy and robustness
under low resolution streaming scenes.

###### Acknowledgements.

This work was partly supported by the University of Macau under Grants:
MYRG2018-00035-FST and MYRG2019-00086-FST, and the Science and Technology
Development Fund, Macau SAR (File no. 041/2017/A1, 0019/2019/A).

## References

  * (1)
  * Avila et al. (2013) Sandra Avila, Nicolas Thome, Matthieu Cord, Eduardo Valle, and Arnaldo De A AraúJo. 2013\.  Pooling in image representation: The visual codeword point of view.  _Computer Vision and Image Understanding_ 117, 5 (2013), 453–465.
  * Berclaz et al. (2011) Jerome Berclaz, Francois Fleuret, Engin Turetken, and Pascal Fua. 2011.  Multiple object tracking using k-shortest paths optimization.  _IEEE transactions on pattern analysis and machine intelligence_ 33, 9 (2011), 1806–1819.
  * Bollegala et al. (2012) Danushka Bollegala, David Weir, and John Carroll. 2012\.  Cross-domain sentiment classification using a sentiment sensitive thesaurus.  _IEEE transactions on knowledge and data engineering_ 25, 8 (2012), 1719–1731.
  * Danelljan et al. (2017) Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. 2017.  Eco: Efficient convolution operators for tracking. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6638–6646.
  * Deng et al. (2019) Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019\.  Arcface: Additive angular margin loss for deep face recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4690–4699.
  * Ester et al. (1996) Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996.  Density-based spatial clustering of applications with noise. In _Int. Conf. Knowledge Discovery and Data Mining_ , Vol. 240. 6.
  * Faklaris et al. (2016) Cori Faklaris, Francesco Cafaro, Sara Anne Hook, Asa Blevins, Matt O’Haver, and Neha Singhal. 2016\.  Legal and ethical implications of mobile live-streaming video apps. In _Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct_. ACM, 722–729.
  * Frey and Dueck (2007) Brendan J Frey and Delbert Dueck. 2007.  Clustering by passing messages between data points.  _science_ 315, 5814 (2007), 972–976.
  * Help (2020) YouTube Help. 2020 (accessed Apr 17th, 2020).  _Blur your videos_.  https://support.google.com/youtube/answer/9057652?hl=en
  * Henriques et al. (2014) João F Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista. 2014.  High-speed tracking with kernelized correlation filters.  _IEEE transactions on pattern analysis and machine intelligence_ 37, 3 (2014), 583–596.
  * Ji et al. (2013) Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 2013\.  3D convolutional neural networks for human action recognition.  _IEEE transactions on pattern analysis and machine intelligence_ 35, 1 (2013), 221–231.
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016.  Fasttext. zip: Compressing text classification models.  _arXiv preprint arXiv:1612.03651_ (2016).
  * Juliako (2020) Juliako. 2020 (accessed Apr 17th, 2020).  _Redact faces with Azure Media Analytics_.  https://docs.microsoft.com/en-us/azure/media-services/previous/media-services-face-redaction
  * Karpathy et al. (2014) Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014\.  Large-scale video classification with convolutional neural networks. In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_. 1725–1732.
  * Li et al. (2018) Feng Li, Cheng Tian, Wangmeng Zuo, Lei Zhang, and Ming-Hsuan Yang. 2018.  Learning spatial-temporal regularized correlation filters for visual tracking. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4904–4913.
  * Liao et al. (2017) Minghui Liao, Baoguang Shi, Xiang Bai, Xinggang Wang, and Wenyu Liu. 2017.  Textboxes: A fast text detector with a single deep neural network. In _Thirty-First AAAI Conference on Artificial Intelligence_.
  * Stewart and Littau (2016) Daxton Râ Chipâ Stewart and Jeremy Littau. 2016.  Up, periscope: Mobile streaming video technologies, privacy in public, and the right to record.  _Journalism & Mass Communication Quarterly_ 93, 2 (2016), 312–331.
  * Sun and Guo (2014) Leilei Sun and Chonghui Guo. 2014.  Incremental affinity propagation clustering based on message passing.  _IEEE Transactions on Knowledge and Data Engineering_ 26, 11 (2014), 2731–2744.
  * Tang et al. (2018) Xu Tang, Daniel K Du, Zeqiang He, and Jingtuo Liu. 2018\.  Pyramidbox: A context-assisted single shot face detector. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 797–813.
  * Tran et al. (2015) Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015.  Learning spatiotemporal features with 3d convolutional networks. In _Proceedings of the IEEE international conference on computer vision_. 4489–4497.
  * Tursun et al. (2017) Osman Tursun, Cemal Aker, and Sinan Kalkan. 2017.  A large-scale dataset and benchmark for similar trademark retrieval.  _arXiv preprint arXiv:1701.05766_ (2017).
  * Wang et al. (2013) Chang-Dong Wang, Jian-Huang Lai, Ching Y Suen, and Jun-Yong Zhu. 2013.  Multi-exemplar affinity propagation.  _IEEE transactions on pattern analysis and machine intelligence_ 35, 9 (2013), 2223–2237.
  * Wang et al. (2018) Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. 2018\.  Cosface: Large margin cosine loss for deep face recognition. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 5265–5274.
  * Yu et al. (2016a) Fengwei Yu, Wenbo Li, Quanquan Li, Yu Liu, Xiaohua Shi, and Junjie Yan. 2016a.  Poi: Multiple object tracking with high performance detection and appearance feature. In _European Conference on Computer Vision_. Springer, 36–42.
  * Yu et al. (2017) Jun Yu, Zhenzhong Kuang, Zhou Yu, Dan Lin, and Jianping Fan. 2017.  Privacy setting recommendation for image sharing. In _2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA)_. IEEE, 726–730.
  * Yu et al. (2016b) Jun Yu, Baopeng Zhang, Zhengzhong Kuang, Dan Lin, and Jianping Fan. 2016b.  iPrivacy: image privacy protection by identifying sensitive objects via deep multi-task learning.  _IEEE Transactions on Information Forensics and Security_ 12, 5 (2016), 1005–1016.
  * Yue-Hei Ng et al. (2015) Joe Yue-Hei Ng, Matthew Hausknecht, Sudheendra Vijayanarasimhan, Oriol Vinyals, Rajat Monga, and George Toderici. 2015.  Beyond short snippets: Deep networks for video classification. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4694–4702.
  * Zamir et al. (2012) Amir Roshan Zamir, Afshin Dehghan, and Mubarak Shah. 2012\.  Gmcp-tracker: Global multi-object tracking using generalized minimum clique graphs. In _European Conference on Computer Vision_. Springer, 343–356.
  * Zhang et al. (2016) Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016\.  Joint face detection and alignment using multitask cascaded convolutional networks.  _IEEE Signal Processing Letters_ 23, 10 (2016), 1499–1503.
  * Zhang and Gomes (2014) Tong Zhang and Herman Martins Gomes. 2014.  Technology survey on video face tracking. In _Imaging and Multimedia Analytics in a Web and Mobile World 2014_ , Vol. 9027. International Society for Optics and Photonics, 90270F.
  * Zhu et al. (2018) Zheng Zhu, Wei Wu, Wei Zou, and Junjie Yan. 2018\.  End-to-end flow correlation tracking with spatial-temporal attention. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 548–557.


# Energy conditions in the $f(R,L,T)$ theory of gravity

Simran Arora 0000-0003-0326-8945<EMAIL_ADDRESS>Department of
Mathematics, Birla Institute of Technology and Science-Pilani,
Hyderabad Campus, Hyderabad-500078, India. P.H.R.S. Moraes 0000-0002-8478-546
<EMAIL_ADDRESS>Laboratório de Física Teórica e Computacional (LFTC),
Universidade Cidade de São Paulo (UNICID) - Rua Galvão Bueno 868, 01506-000
São Paulo, Brazil. P.K. Sahoo 0000-0003-2130-8832<EMAIL_ADDRESS>pilani.ac.in Department of Mathematics, Birla Institute of Technology and
Science-Pilani,
Hyderabad Campus, Hyderabad-500078, India.

(August 28, 2024)

###### Abstract

We construct the energy conditions for the recently proposed $f(R,L,T)$
gravity theory, for which $f$ is a generic function of the Ricci scalar $R$,
matter lagrangian density $L$ and trace of the energy-momentum tensor $T$. We
analyse two different forms for the $f(R,L,T)$ function within the framework
of the Friedmann-Lemâitre-Robertson-Walker universe. We constrain the model
parameters from the energy conditions. This approach allows us to assess the
feasibility of specific forms of the $f(R,L,T)$ gravity.

##  I Introduction

Since the discovery of the acceleration of the universe expansion in the late
90’s of the last century [1, 2], theoretical physicists struggle to discover
what mechanism is behind such an outstanding dynamical phenomenon. In
principle, a simple constant term, namely cosmological constant $\Lambda$, in
Einstein’s field equations of General Relativity (GR), could provide the late
acceleration dynamics. However, this approach is plagued by the so-called
cosmological constant problem, which is the extraordinarily high discrepancy
between the observational [1, 2] and theoretical values of $\Lambda$ [3].

This shortcoming has led to the consideration of the possibility that what
really drives the universe expansion to accelerate is an effect “hidden” in a
wider formalism of gravity. We already faced this kind of paradigm shift when
GR theory predicted the observed “discrepancies” in Mercury’s orbit [4], which
were not present in Newtonian formalism. We might be facing a new paradigm
shift.

An extended gravity formalism must be able to recover GR in certain regimes,
as the Solar System, but while the standard cosmological model, based on GR,
explains the cosmic acceleration through $\Lambda$, an extended gravity
cosmological model should explain it from extra terms appearing in the field
equations of such theories [5].

Nowadays, we have numerous extended formalisms of gravity, such as $f(R)$
theories [6, 7], $f(R,L)$ gravity [8, 9], $f(R,T)$ gravity [10],
$f(\mathcal{T})$ gravity [11, 12], $f(Q)$ gravity [13, 14], $f(Q,T)$ gravity
[15, 16, 17, 18, 19], among others, in which $f$ stands for a generic function
of the argument(s), and $R$ stands for the Ricci scalar, $L$ for the matter
lagrangian density, $T$ for the trace of the energy-momentum tensor,
$\mathcal{T}$ for the torsion scalar and $Q$ for the non-metricity scalar.

It is clear the necessity of distinguishing among these models and searching
for the valid ones within them. For these purposes, a well-known tool is
cosmography [20, 21, 22], which is not in the scope of the present article.

Here, we are going to use the energy conditions in order to constrain a
recently proposed gravity theory, called $f(R,L,T)$ theory [23]. The energy
conditions refer to physical restrictions imposed to the distribution of
matter and energy in space-time. They represent paths to implement the
positiveness of the energy-momentum tensor, the attractive nature of gravity
and the geodesic structure of space-time. They have been discussed in the
literature for a long time, as one can check [24, 25], among others. Some of
the references above, namely, [8, 18], are, indeed, energy conditions
applications in extended gravity.

An exemplary contribution to the field of energy conditions applications was
presented in Reference [26], in which the energy conditions were applied to
the $f(Q)$ gravity theory. An important result of [26] was obtained for a
well-known functional form of the $f(Q)$ function, namely $f(Q)=Q+mQ^{n}$,
with $m$ and $n$ being free parameters. The constraints obtained from energy
conditions application to such a model were $m\leq-1$ and $0.9\leq n\leq 2$.

In the present article, we are going to apply the energy conditions to the
very recently developed $f(R,L,T)$ gravity theory. For this purpose, we
introduce the outline of $f(R,L,T)$ theory in Section II. In Section III, we
discuss the energy condition in the context of GR. We obtain the inequalities
corresponding to the energy condition in $f(R,L,T)$ theory in Section IV. In
Section V, we formulate the expressions for the energy conditions in terms of
cosmological parameters and discuss the energy bounds for the two specific
$f(R,L,T)$ models. In the last section, we summarize our results and present
our concluding remarks.

##  II The $f(R,L,T)$ theory of gravity

Haghani and Harko proposed in [23] a generalization of two gravity theories
that allow to non-minimal coupling between geometry and matter, namely
$f(R,L)$ and $f(R,T)$ theories (check, respectively, [27] and [10]), the so-
called $f(R,L,T)$ theory of gravity.

The $f(R,L,T)$ gravity formalism starts from the action

$\mathcal{S}=\frac{1}{16\pi}\int d^{4}x\sqrt{-g}f(R,L,T)+\int
d^{4}x\sqrt{-g}L,$ (1)

in which $g$ stands for the metric determinant.

The variation of the above action with respect to the metric tensor
$g_{\mu\nu}$ yields the following field equations

$\displaystyle(R_{\mu\nu}+g_{\mu\nu}\Box-\nabla_{\mu}\nabla_{\nu})f_{R}-\frac{1}{2}fg_{\mu\nu}=8\pi
T_{\mu\nu}+$
$\displaystyle\frac{1}{2}(f_{L}+2f_{T})(T_{\mu\nu}-Lg_{\mu\nu})+f_{T}\,\tau_{\mu\nu},$
(2)

in which $R_{\mu\nu}$ is the Ricci tensor, $g_{\mu\nu}$ is the metric tensor,
$\Box=g^{\mu\nu}\nabla_{\mu}\nabla_{\nu}$, $T_{\mu\nu}$ is the energy-momentum
tensor, $f_{R}\equiv\partial f/\partial R$, $f_{L}\equiv\partial f/\partial
L$, $f_{T}\equiv\partial f/\partial T$, and $\tau_{\mu\nu}\equiv
2g^{\alpha\beta}\partial^{2}L/(\partial g^{\mu\nu}\partial g^{\alpha\beta})$.

By taking the trace of (II), we obtain

$\displaystyle(R+3\Box)f_{R}-2f=8\pi
T+(f_{L}+2f_{T})\frac{T-4L}{2}+f_{T}\tau.$ (3)

It is evident that the above field equations (II) recover the metric formalism
field equations of the $f(R)$ theory for the functional choice
$f(R,L,T)=f(R)$. Moreover, if $f(R,L,T)=f(R,L)$, it leads to the field
equations of $f(R,L)$ theory and an analogous scenario arises in the case of
$f(R,L,T)=f(R,T)$. Furthermore, when $f(R,L,T)=R$, we obtain the standard
field equations of GR.

##  III The energy conditions

An important aspect of the approach that we shall employ to deduce the energy
conditions for the $f(R,L,T)$ theory of gravity can be adopted from the
strategy used to establish the null energy condition (NEC), strong energy
condition (SEC), weak energy condition (WEC) and dominant energy condition
(DEC) within GR framework. To ensure accuracy, we will begin by providing a
concise overview of the methodology used to address these energy conditions in
GR [28]. The Raychaudhuri equation, together with the condition of attractive
gravity, serves as the fundamental source of the energy conditions [29]. For
this, let $u^{\mu}$ be the tangent vector field in case of a congruence of
timeline geodesics. The Raychaudhuri equation is

$\displaystyle\frac{d\theta}{d\tilde{\tau}}=-\frac{1}{3}\theta^{2}-\sigma_{\mu\nu}\sigma^{\mu\nu}+\omega_{\mu\nu}\omega^{\mu\nu}-R_{\mu\nu}u^{\mu}u^{\nu},$
(4)

where $\tilde{\tau}$, $\theta$, $\sigma_{\mu\nu}$ and $\omega_{\mu\nu}$ are
the conformal time, expansion parameter, shear and rotation associated with
the congruence, respectively. The equation governing the expansion of a
congruence of null geodesics, which is determined by a vector field $k^{\mu}$,
exhibits a similar structure to the Raychaudhuri equation.

Since GR field equations relate $R_{\mu\nu}$ to $T_{\mu\nu}$, the combination
of Einstein’s and Raychaudhuri’s equations can be used to restrict the energy-
momentum tensor on physical grounds. Since the shear is a spatial tensor, one
has $\sigma^{2}=\sigma_{\mu\nu}\sigma^{\mu\nu}\geq 0$. The condition for
attractive gravity, i.e., $d\theta/d\tilde{\tau}<0$, for any hypersurface
orthogonal congruences $(\omega_{\mu\nu}=0)$, reduces to

$\displaystyle R_{\mu\nu}u^{\mu}u^{\nu}\geq 0,\quad and\quad
R_{\mu\nu}k^{\mu}k^{\nu}\geq 0.$ (5)

Henceforth, we obtain the following conditions

$\displaystyle R_{\mu\nu}u^{\mu}u^{\nu}$ $\displaystyle=$
$\displaystyle\left(T_{\mu\nu}-\frac{T}{2}g_{\mu\nu}\right)u^{\mu}u^{\nu}\geq
0,$ (6) $\displaystyle R_{\mu\nu}k^{\mu}k^{\nu}$ $\displaystyle=$
$\displaystyle T_{\mu\nu}k^{\mu}k^{\nu}\geq 0.$ (7)

Equations (6) and (7) are nothing but SEC and NEC, respectively, in a
coordinate invariant way in terms of $T_{\mu\nu}$ and vector fields character.
Note that for a perfect fluid of density $\rho$ and pressure $p$, Equations
(6) and (7) reduce to $\rho+3p\geq 0$ and $\rho+p\geq 0$, respectively.

Another energy condition, namely WEC, mandates the positivity of the energy
density for any observer at any point, indicating $\rho\geq 0$ and $\rho+p\geq
0$. The requirement that energy cannot propagate faster than the speed of
light gives rise to an additional condition, namely $\rho\pm p\geq 0$, along
with the condition $\rho\geq 0$, which is called DEC.

##  IV The energy conditions in the $f(R,L,T)$ gravity

We can rewrite Eq.(II) as follows

$\displaystyle G_{\mu\nu}=R_{\mu\nu}-\frac{1}{2}g_{\mu\nu}R=8\pi
T^{eff}_{\mu\nu}\text{,}$ (8)

in which the effective energy-momentum tensor $T^{eff}_{\mu\nu}$ is

$\displaystyle T^{eff}_{\mu\nu}\equiv\frac{1}{8\pi
f_{R}}\left[\frac{1}{2}g_{\mu\nu}\left(f-f_{R}R\right)-\left(g_{\mu\nu}\Box-\nabla_{\mu}\nabla_{\nu}\right)f_{R}\right.$
$\displaystyle\left.+8\pi
T_{\mu\nu}+\frac{1}{2}\left(f_{L}+2f_{T}\right)\left(T_{\mu\nu}-Lg_{\mu\nu}\right)+f_{T}\tau_{\mu\nu}\right].$
(9)

Contracting Equation (IV), we obtain

$\displaystyle T^{eff}=\frac{1}{8\pi f_{R}}\left[2\left(f-f_{R}R\right)-3\Box
f_{R}+8\pi T\right.$
$\displaystyle\left.+\frac{1}{2}\left(f_{L}+2f_{T}\right)\left(T-4L\right)+f_{T}\,\tau\right].$
(10)

Following what was done for the $f(R,T,R_{\mu\nu}T^{\mu\nu})$ gravity in [30],
the attractive nature of gravity needs to satisfy the following additional
constraint

$\displaystyle\frac{8\pi+\frac{1}{2}(f_{L}+2f_{T})}{8\pi f_{R}}>0,$ (11)

which does not depend on the conditions derived from the Raychaudhuri
equation.

Let us take the Friedmann-Lemaître-Robertson-Walker (FLRW) metric,

$\displaystyle ds^{2}=-dt^{2}+a(t)^{2}\left(dx^{2}+dy^{2}+dz^{2}\right),$ (12)

in which $a(t)$ is the scale factor. From (12), we obtain
$R=6(2H^{2}+\dot{H})$, in which the Hubble parameter $H=\dot{a}/a$.

By using Equation (6), the SEC can be given as
$T_{\mu\nu}^{eff}u^{\mu}u^{\nu}+\frac{1}{2}T^{eff}\geq 0$, where we used
$g_{\mu\nu}u^{\mu}u^{\nu}=-1$. Taking the energy-momentum tensor of a perfect
fluid, that is, $T_{\mu\nu}=\left(\rho+p\right)u_{\mu}u_{\nu}+pg_{\mu\nu}$,
and considering the condition for SEC in $f(R,L,T)$ theory, yields

$\displaystyle\frac{1}{8\pi
f_{R}}\left[\left(\rho+3p\right)\left(8\pi+\frac{1}{2}(f_{L}+2f_{T})\right)+(f-f_{R}R)\right.$
(13)
$\displaystyle\left.+3\left(f_{RRR}\dot{R}^{2}+f_{RR}\ddot{R}+Hf_{RR}\dot{R}\right)-(f_{L}+2f_{T})L\right]\geq
0,$

where the dot denotes cosmic time differentiation. We neglect the terms
involving the second derivative of $L$ with respect to $g_{\mu\nu}$. As we are
dealing with perfect fluid, the matter lagrangian can either be $L=p$ or
$L=-\rho$, which makes it obvious to ignore such a term.

The NEC in $f(R,L,T)$ gravity can be expressed as

$\displaystyle\frac{1}{8\pi
f_{R}}\left[\left(\rho+p\right)\left(8\pi+\frac{1}{2}(f_{L}+2f_{T})\right)+\right.$
(14)
$\displaystyle\left.\left(f_{RRR}\dot{R}^{2}+f_{RR}\ddot{R}\right)\right]\geq
0.$

Note that the above expressions for the SEC and NEC are directly derived from
the Raychaudhuri equation. Using Equations (IV) and (12), the effective energy
density and the effective pressure can be derived as

$\displaystyle\rho^{eff}$ $\displaystyle=$ $\displaystyle\frac{1}{8\pi
f_{R}}\left[(8\pi+\frac{1}{2}(f_{L}+2f_{T}))\rho-\frac{1}{2}(f-f_{R}R)\right.$
(15)
$\displaystyle\left.-3Hf_{RR}\dot{R}+\frac{1}{2}(f_{L}+2f_{T})L\right]\geq 0,$
$\displaystyle p^{eff}$ $\displaystyle=$ $\displaystyle\frac{1}{8\pi
f_{R}}\left[(8\pi+\frac{1}{2}(f_{L}+2f_{T}))p+\frac{1}{2}(f-f_{R}R)+\right.$
$\displaystyle\left.f_{RRR}\dot{R}^{2}+f_{RR}\ddot{R}+2Hf_{RR}\dot{R}-\frac{1}{2}(f_{L}+2f_{T})L\right]\geq
0.$

Then, the corresponding DEC and WEC in $f(R,L,T)$ gravity can be respectively
given as

$\displaystyle\frac{1}{8\pi
f_{R}}\left[(8\pi+\frac{1}{2}(f_{L}+2f_{T}))(\rho-p)-(f-f_{R}R)\right.$ (17)
$\displaystyle\left.-f_{RRR}\dot{R}^{2}-f_{RR}\ddot{R}-5Hf_{RR}\dot{R}+(f_{L}+2f_{T})L\right]\geq
0,$ $\displaystyle\frac{1}{8\pi
f_{R}}\left[(8\pi+\frac{1}{2}(f_{L}+2f_{T}))\rho-\frac{1}{2}(f-f_{R}R)\right.$
(18)
$\displaystyle\left.-3Hf_{RR}\dot{R}+\frac{1}{2}(f_{L}+2f_{T})L\right]\geq 0.$

If we neglect the role of terms on $L$, i.e., $f(R,L,T)=f(R,T)$, the above
energy conditions reduce to that in $f(R,T)$ gravity [31, 32]. Further, by
neglecting the dependence on the trace of the energy-momentum tensor, we can
have the energy conditions in $f(R)$ gravity, which are consistent with the
results in the literature [33, 34, 35].

##  V Constraining $f(R,L,T)$ gravity

In our present approach, the energy-condition inequalities can be used to
place bounds in a given $f(R,L,T)$ function in the context of FLRW models. To
investigate such bounds, we first note that the Ricci scalar and its
derivatives for a spatially flat FLRW geometry can be expressed in terms of
the deceleration $q$, jerk $j$ and snap $s$ parameters [36, 37, 38], that is,

$\displaystyle R$ $\displaystyle=$ $\displaystyle 6H^{2}(1+q),$ (19)
$\displaystyle\dot{R}$ $\displaystyle=$ $\displaystyle 6H^{3}(j-q-2),$ (20)
$\displaystyle\ddot{R}$ $\displaystyle=$ $\displaystyle
6H^{4}(4q^{2}+15q+2j+s+9),$ (21)

such that these parameters are written in terms of the Hubble parameter, the
scale factor and its derivatives as $q=-\frac{1}{H^{2}}\frac{\ddot{a}}{a}$,
$j=\frac{1}{H^{3}}\frac{\dddot{a}}{a}$ and
$s=\frac{1}{H^{4}}\frac{\ddddot{a}}{a}$, respectively.

Therefore, the energy conditions for the $f(R,L,T)$ gravity can be rewritten
as

$\displaystyle SEC\mathrel{\mathop{\mathchar
58\relax}}\left(\rho+3p\right)+\frac{1}{8\pi+\frac{1}{2}(f_{L}+2f_{T})}\left[f-6H^{2}(1+q)f_{R}+\right.$
$\displaystyle\left.18H^{4}\left(6f_{RRR}H^{2}(j-q-2)^{2}+f_{RR}(4q^{2}+14q+3j+s+7)\right)\right.$
$\displaystyle\left.-(f_{L}+2f_{T})L\right]\geq 0,$ (22)

$\displaystyle NEC\mathrel{\mathop{\mathchar
58\relax}}\left(\rho+p\right)+\frac{6H^{4}}{8\pi+\frac{1}{2}(f_{L}+2f_{T})}\left[(6f_{RRR}H^{2}(j-q-2)^{2}\right.$
$\displaystyle\left.+f_{RR}(4q^{2}+15q+2j+s+9)\right]\geq 0,$ (23)

$\displaystyle DEC\mathrel{\mathop{\mathchar
58\relax}}\left(\rho-p\right)+\frac{1}{8\pi+\frac{1}{2}(f_{L}+2f_{T})}\left[(f_{L}+2f_{T})L-f+\right.$
$\displaystyle\left.6H^{2}(1+q)f_{R}-36H^{6}f_{RRR}(j-q-2)^{2}\right.$ (24)
$\displaystyle\left.-6H^{4}f_{RR}(4q^{2}+10q+7j+s-1)\right]\geq 0,$

$\displaystyle WEC\mathrel{\mathop{\mathchar
58\relax}}\rho+\frac{1}{8\pi+\frac{1}{2}(f_{L}+2f_{T})}\left[\frac{1}{2}(f_{L}+2f_{T})L\right.$
(25)
$\displaystyle\left.-\frac{1}{2}(f-6H^{2}(1+q)f_{R})-18H^{4}f_{RR}(j-q-2)\right]\geq
0.$

To exemplify how the above conditions can be used to place bounds on
$f(R,L,T)$ theories, we first note that, apart from the WEC, all of the above
inequalities depend on the current value of the snap parameter. Therefore,
since no reliable measurement of this parameter has been reported hitherto, in
what follows, we shall focus on the WEC requirement in the confrontation of
the energy-condition bounds with observational data.

We can consider $q_{0}=-0.503$, $H_{0}=73.5$ and $j_{0}=1$ according to
References [39, 40] to obtain the constraints or bounds on the model
parameters.

###  V.1 $f(R,L,T)=R+\gamma\,T+\lambda\,L$ with $L=-\rho$

As a first case, we consider the simplest linear functional form of
$f(R,L,T)$, namely $f(R,L,T)=R+\gamma\,T+\lambda\,L$, where $\gamma$ and
$\lambda$ are free parameters. For this case, the WEC fulfilment condition can
be written as

$\rho+\frac{1}{8\pi+\frac{1}{2}(\lambda+2\gamma)}\left[-\frac{\gamma}{2}(\rho+3p)\right]\geq
0.$ (26)

The condition ensuring attractive gravity within this model is expressed as
$8\pi+\frac{1}{2}(\lambda+2\gamma)>0$ (derived from (11)). The validity of WEC
reduces to $-\frac{\gamma}{2}(\rho+3p)\geq 0$, as it mandates that the energy
density measured by any observer remains non-negative. The latter condition
imposes a constraint on the parameter $\gamma$.

We outline constraints on the model parameters for the two distinct phases of
the universe, namely the matter-dominated era and the dark energy-dominated
era.

  * •

During the matter-dominated era, for which $p=0$, we derive constraints on the
model parameters $\lambda$ and $\gamma$ using inequalities (26) and (11).
Thus, we get

$\displaystyle\lambda>-16\pi-2\gamma\quad\text{and}\quad\gamma\leq 0.$ (27)

  * •

In the dark energy-dominated era, characterized by $\omega\simeq-1$, we deduce
further insights from the inequality

$\displaystyle\lambda>-16\pi-2\gamma\quad\text{and}\quad\gamma\geq 0.$ (28)

From equation (13), it becomes evident that achieving an accelerated expansion
necessitates the condition $\rho_{eff}+3p_{eff}<0$. It is worth noting that
one can ensure that ordinary matter adheres to all the energy conditions while
achieving acceleration for some suitable functional form of $f(R,L,T)$.

###  V.2 $f(R,L,T)=R^{n}+\alpha\,L\,T+\beta$ with $L=-\rho$

As a second example, we shall consider the form
$f(R,L,T)=R^{n}+\alpha\,L\,T+\beta$, in which $n$ is an integer and $\alpha$
and $\beta$ are constants. This functional form has been proposed by the
authors of the theory themselves [23]. For $n=1$, it has a de-Sitter type
evolutionary phase triggered by the generalized geometry-matter coupling in
the case of ordinary matter.

The inequality for the WEC fulfilment condition can be written in terms of the
$q$, $j$ and $H$ parameters as

$\displaystyle\rho+\frac{1}{8\pi+\frac{3\alpha}{2}(p-\rho)}\left[\frac{1}{2}\left[(n-1)6^{n}H^{2n}(1+q)^{n}+\alpha\rho(3p-\rho)\right.\right.$
$\displaystyle\left.\left.-\beta)\right]+\frac{3}{2}\alpha\rho(\rho-p)-3n(n-1)H^{2n}(1+q)^{n-2}(j-q-2)\right]$
$\displaystyle\geq 0.$ (29)

For simplicity, we consider the case $n=1$, so that Equation (V.2) reduces to

$\displaystyle\rho+\frac{1}{8\pi+\frac{3\alpha}{2}(p-\rho)}\left[\frac{\alpha\rho}{2}(3p-\rho)-\beta+\frac{3}{2}\alpha\rho(\rho-p)\right]\geq
0.$

That is,

$\displaystyle\rho+\frac{1}{8\pi+\frac{3\alpha}{2}(p-\rho)}\left[\frac{1}{2}(2\alpha\rho^{2}-\beta)\right]\geq
0.$ (30)

Since, the term $8\pi+\frac{3\alpha}{2}(p-\rho)>0$ using the condition (11),
we should have $\frac{1}{2}(2\alpha\rho^{2}-\beta)\geq 0$.

Below, we present the energy conditions constraints for the model parameters
in two distinct stages of the universe, namely the dominated era and the dark
energy-dominated era.

  * •

Matter-dominated era: once again, in the specific phase of matter-dominated
era, the fluid pressure is null. So, from inequality (30) and using (11), we
obtain the bounds on the model parameters $\alpha$ and $\beta$ as

$\displaystyle\alpha<\frac{16\pi}{3\rho}\quad\text{and}\quad\beta\leq\frac{32\pi\rho}{3}.$
(31)

  * •

Dark energy dominated era: for the dark energy dominated era,
$\omega\simeq-1$. From inequality (30), we obtain

$\alpha<\frac{8\pi}{3\rho}\quad\text{and}\quad\beta\leq\frac{16\pi\rho}{3}$
(32)

We want to stress that within the generalized framework outlined in this
study, there are no violations of the energy conditions of GR. Instead, what
occurs is a reinterpretation of additional degrees of freedom that arise from
the dynamics of the current theory.

##  VI Summary and Discussion

In this work, we have worked with a generalized modified theory of gravity
featuring a flexible coupling of geometry and matter. The gravitational
lagrangian is derived by incorporating an arbitrary function of $f(R,L,T)$
into the Einstein-Hilbert action. This is basically an extension and
generalization of two classes of gravitational theories with geometry-matter
coupling, that is, $f(R,L)$ and $f(R,T)$ theories.

The flexibility inherent in the Lagrangian formulation raises the question of
placing constraints on such a theory based on physical considerations. We have
addressed this issue by establishing constraints on general and specific forms
of $f(R,L,T)$ gravity. We accomplish this by examining the energy conditions.
By utilizing the Raychaudhuri equation and imposing the condition of
attractive gravity, we derived the energy conditions and focused on weak
energy conditions to obtain the constraints on model parameters. In a formal
sense, the WEC, NEC, DEC and SEC can be expressed similarly to how they are in
GR. We have explored two scenarios: one characterized by a matter-dominated
phase and the other by a dark energy phase, aiming to establish constraints on
the free parameters for the two models.

The model parameters have a significant impact on the validation of the energy
conditions. We have found that both the considered models satisfy the energy
conditions, specifically the weak energy condition, for the appropriate
choices of free parameters. These results could be a powerful tool for probing
the dark sector of the universe, as any deviations from standard GR could be
highlighted.

## Data availability

There are no new data associated with this article.

## Acknowledgements

PKS acknowledges the Science and Engineering Research Board, Department of
Science and Technology, Government of India, for financial support to carry
out the Research project No.: CRG/2022/001847.

## References

  * [1] Adam G. Riess, Alexei V. Filippenko, Peter Challis, Alejandro Clocchiatti, Alan Diercks, Peter M. Garnavich, Ron L. Gilliland, Craig J. Hogan, Saurabh Jha, Robert P. Kirshner, B. Leibundgut, M. M. Phillips, David Reiss, Brian P. Schmidt, Robert A. Schommer, R. Chris Smith, J. Spyromilio, Christopher Stubbs, Nicholas B. Suntzeff, and John Tonry.  Observational Evidence from Supernovae for an Accelerating Universe and a Cosmological Constant.  AJ, 116(3):1009–1038, September 1998.
  * [2] S. Perlmutter, G. Aldering, G. Goldhaber, R. A. Knop, P. Nugent, P. G. Castro, S. Deustua, S. Fabbro, A. Goobar, D. E. Groom, I. M. Hook, A. G. Kim, M. Y. Kim, J. C. Lee, N. J. Nunes, R. Pain, C. R. Pennypacker, R. Quimby, C. Lidman, R. S. Ellis, M. Irwin, R. G. McMahon, P. Ruiz-Lapuente, N. Walton, B. Schaefer, B. J. Boyle, A. V. Filippenko, T. Matheson, A. S. Fruchter, N. Panagia, H. J. M. Newberg, W. J. Couch, and The Supernova Cosmology Project.  Measurements of $\Omega$ and $\Lambda$ from 42 High-Redshift Supernovae.  ApJ, 517(2):565–586, June 1999.
  * [3] Steven Weinberg.  The cosmological constant problem.  Reviews of Modern Physics, 61(1):1–23, January 1989.
  * [4] Albert Einstein.  Erklärung der Perihelbewegung des Merkur aus der allgemeinen Relativitätstheorie.  Sitzungsberichte der Königlich Preussischen Akademie der Wissenschaften, pages 831–839, January 1915.
  * [5] Mark Trodden.  Cosmic Acceleration and Modified Gravity.  International Journal of Modern Physics D, 16:2065–2074, January 2007.
  * [6] Antonio De Felice and Shinji Tsujikawa.  $f(R)$ Theories.  Living Reviews in Relativity, 13(1):3, June 2010.
  * [7] A. A. Starobinsky.  Disappearing cosmological constant in $f(R)$ gravity.  Soviet Journal of Experimental and Theoretical Physics Letters, 86(3):157–163, October 2007.
  * [8] Jun Wang and Kai Liao.  Energy conditions in $f(R,L_{m})$ gravity.  Classical and Quantum Gravity, 29(21):215016, November 2012.
  * [9] Raja Solanki, Zinnat Hassan, and P. K. Sahoo.  Wormhole solutions in $f(R,L_{m})$ gravity.  Chinese Journal of Physics, 85:74–88, October 2023.
  * [10] Tiberiu Harko, Francisco S. N. Lobo, Shin’Ichi Nojiri, and Sergei D. Odintsov.  $f(R,T)$ gravity.  Phys. Rev. D, 84(2):024020, July 2011.
  * [11] Yi-Fu Cai, Salvatore Capozziello, Mariafelicia De Laurentis, and Emmanuel N. Saridakis.  $f(T)$ teleparallel gravity and cosmology.  Reports on Progress in Physics, 79(10):106901, October 2016.
  * [12] Shih-Hung Chen, James B. Dent, Sourish Dutta, and Emmanuel N. Saridakis.  Cosmological perturbations in $f(T)$ gravity.  Phys. Rev. D, 83(2):023508, January 2011.
  * [13] Jose Beltrán Jiménez, Lavinia Heisenberg, Tomi Koivisto, and Simon Pekar.  Cosmology in $f(Q)$ geometry.  Phys. Rev. D, 101(10):103507, May 2020.
  * [14] Sanjay Mandal, Deng Wang, and P. K. Sahoo.  Cosmography in $f(Q)$ gravity.  Phys. Rev. D, 102(12):124029, December 2020.
  * [15] Yixin Xu, Guangjie Li, Tiberiu Harko, and Shi-Dong Liang.  $f(Q,T)$ gravity.  European Physical Journal C, 79(8):708, August 2019.
  * [16] Snehasish Bhattacharjee and P. K. Sahoo.  Baryogenesis in $f(Q,T)$ gravity.  European Physical Journal C, 80(3):289, March 2020.
  * [17] Antonio Nájera and Amanda Fajardo.  Cosmological perturbation theory in $f(Q,T)$ gravity.  J. Cosmology Astropart. Phys., 2022(3):020, March 2022.
  * [18] Simran Arora and P. K. Sahoo.  Energy conditions in $f(Q,T)$ gravity.  Phys. Scr, 95(9):095003, September 2020.
  * [19] Simran Arora, JRL Santos, and PK Sahoo.  Constraining f (q, t) gravity from energy conditions.  Physics of the Dark Universe, 31:100790, 2021.
  * [20] Matt Visser.  Cosmography: Cosmology without the Einstein equations.  General Relativity and Gravitation, 37(9):1541–1548, September 2005.
  * [21] B. S. Sathyaprakash, B. F. Schutz, and C. Van Den Broeck.  Cosmography with the Einstein Telescope.  Classical and Quantum Gravity, 27(21):215006, November 2010.
  * [22] Jéferson A. S. Fortunato, Wiliam S. Hipólito-Ricaldi, and Marcelo V. dos Santos.  Cosmography from well-localized fast radio bursts.  Mon. Not. Roy. Astron. Soc., 526(2):1773–1782, December 2023.
  * [23] Zahra Haghani and Tiberiu Harko.  Generalizing the coupling between geometry and matter: $f(R,L_{m},T)$ gravity.  European Physical Journal C, 81(7):615, July 2021.
  * [24] Frank J. Tipler.  Energy conditions and spacetime singularities.  Phys. Rev. D, 17(10):2521–2528, May 1978.
  * [25] Michael S. Morris, Kip S. Thorne, and Ulvi Yurtsever.  Wormholes, time machines, and the weak energy condition.  Phys. Rev. Lett., 61(13):1446–1449, September 1988.
  * [26] Sanjay Mandal, P. K. Sahoo, and J. R. L. Santos.  Energy conditions in $f(Q)$ gravity.  Phys. Rev. D, 102(2):024057, July 2020.
  * [27] Tiberiu Harko and Francisco S. N. Lobo.  $f(R,L_{m})$ gravity.  European Physical Journal C, 70(1-2):373–379, November 2010.
  * [28] Sean M Carroll.  An introduction to general relativity: spacetime and geometry.  Addison Wesley, 101:102, 2004.
  * [29] Sayan Kar and Soumitra Sengupta.  The raychaudhuri equations: A brief review.  Pramana, 69:49–76, 2007.
  * [30] M. Sharif and M. Zubair.  Energy conditions in f ( R, T, R μν T μν) gravity.  Journal of High Energy Physics, 2013:79, December 2013.
  * [31] M Sharif, S Rani, and R Myrzakulov.  Analysis of $f(r,t)$ gravity models through energy conditions.  The European Physical Journal Plus, 128:1–11, 2013.
  * [32] Flavio Gimenes Alvarenga, Mahouton Jonas Stephane Houndjo, Adjimon Vincent Monwanou, and Jean B Chabi Orou.  Testing some $f(r,t)$ gravity models from energy conditions.  Journal of Modern Physics, 4:130–139, 2013.
  * [33] J Santos, JS Alcaniz, MJ Reboucas, and FC Carvalho.  Energy conditions in $f(r)$ gravity.  Physical Review D, 76(8):083513, 2007.
  * [34] Janilo Santos, Marcelo J Rebouças, and Jailson S Alcaniz.  Energy conditions constraints on a class of $f(r)$-gravity.  International Journal of Modern Physics D, 19(08n10):1315–1321, 2010.
  * [35] Crislane S Santos, Janilo Santos, Salvatore Capozziello, and Jailson S Alcaniz.  Strong energy condition and the repulsive character of $f(r)$ gravity.  General Relativity and Gravitation, 49:1–14, 2017.
  * [36] Matt Visser.  Jerk, snap and the cosmological equation of state.  Classical and Quantum Gravity, 21(11):2603, 2004.
  * [37] Matt Visser.  Cosmography: Cosmology without the einstein equations.  General Relativity and Gravitation, 37:1541–1548, 2005.
  * [38] I. S. Farias and P. H. R. S. Moraes.  Using cosmographic energy conditions to constrain $f(R,T)$ gravity models.  European Physical Journal Plus, 138(5):469, May 2023.
  * [39] Dillon Brout, Dan Scolnic, Brodie Popovic, Adam G Riess, Anthony Carr, Joe Zuntz, Rick Kessler, Tamara M Davis, Samuel Hinton, David Jones, et al.  The pantheon+ analysis: cosmological constraints.  The Astrophysical Journal, 938(2):110, 2022.
  * [40] Salvatore Capozziello, Rocco D’Agostino, and Orlando Luongo.  Extended gravity cosmography.  International Journal of Modern Physics D, 28(10):1930016, 2019.