# SPOT! Revisiting Video-Language Models for Event Understanding

Gengyuan Zhang 1,2 Jinhe Bi 1111Preprint version. Jindong Gu 3 Yanyu Chen 1
Volker Tresp 1,2
1 LMU Munich, Munich, Germany
2 Munich Center for Machine Learning, Munich, Germany
3 University of Oxford, Oxford, United Kingdom
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Equal Contribution.

###### Abstract

Understanding videos is an important research topic for multimodal learning.
Leveraging large-scale datasets of web-crawled video-text pairs as weak
supervision has become a pre-training paradigm for learning joint
representations and showcased remarkable potential in video understanding
tasks. However, videos can be multi-event and multi-grained, while these
video-text pairs usually contain only broad-level video captions. This raises
a question: with such weak supervision, can video representation in video-
language models gain the ability to distinguish even factual discrepancies in
textual description and understand fine-grained events? To address this, we
introduce SPOT Prober, to benchmark existing video-language models’s
capacities of distinguishing event-level discrepancies as an indicator of
models’ event understanding ability. Our approach involves extracting events
as tuples (¡Subject, Predicate, Object, Attribute, Timestamps¿) from videos
and generating false event tuples by manipulating tuple components
systematically. We reevaluate the existing video-language models with these
positive and negative captions and find they fail to distinguish most of the
manipulated events. Based on our findings, we propose to plug in these
manipulated event captions as hard negative samples and find them effective in
enhancing models for event understanding.

Figure 1: Can Video-Language Models understand events with even only subtle
discrepancies? To explore this, we propose SPOT prober, using extracted event
SPOT tuples, ¡Subject, Predicate, Object, Attribute, Timestamp¿ from the video
scene graph and manipulate event tuples with different manipulation patterns,
like swapping different entities’ attributes or swapping predicates of
different timestamps. Based on positive and negative captions generated from
these tuples, we aim to reevaluate video-language models’ performance
sensitivity to any manipulation on event tuples as an indicator of model event
understanding abilities.

##  1 Introduction

Understanding videos is a critical task in vision-language multi-modal
learning. However, comprehending complex object relationships and dynamic
attribute changes in videos is challenging. For example, a short one-minute
video in Fig. 1 depicts the relationships and states of multiple characters in
a movie and the significant changes in their emotions over time. Even subtle
discrepancies in the video content, such as interchanging characters’
positions or altering the chronological sequence of characters’ state of mind,
can introduce errors in comprehending events. For instance, descriptions like
”Forrest Gump on the deck works with Lieutenant Taylor” should be accurate,
while ”Lieutenant Taylor on the deck works with Forrest Gump” might appear
similar but is actually false.

Recently, video-language models have shown significant progress in video
understanding tasks. By utilizing web-crawled video-text pairs as weak
supervision for video-language learning, extensive human annotations and
curation are circumvented. Moreover, Video-language models trained by video-
text pairs further show exceptional capacities for video understanding tasks
[28], making it a popular paradigm for video representation learning.

Even though weak textual supervision showcases generalizable abilities in
video understanding at a broad level, we are still concerned regarding whether
the video-language models can comprehend such subtle but critical
distinctions, as exemplified earlier.

This raises a question: In which granularity and in what aspects can video
representations of video-language models distinguish subtle discrepancies
within videos?

To explore this question, we introduce SPOT Prober to benchmark existing
video-language models’s capacities of distinguishing event-level discrepancies
for indicating models’ event understanding ability. By leveraging structured
descriptions of video events in the form of SPOT tuples, ”¡Subject, Subject
attribute, Predicate, Object, Object attribute, Timestamp¿.”, we attempt to
extract factual SPOT tuples from video scene graph and generate “foiled”
tuples that contradict facts by manipulating tuple components.

For instance, in Fig. 1, we generate two untruthful descriptive tuples of the
video by swapping the subject and object’s position attribute or altering the
same entity’s emotional attributes. To seamlessly bridge the gap between
tuple-formatted descriptions and natural language inputs, we utilize a Large
Language Model (LLM) as a decorator, transforming SPOT tuples into coherent
textual captions.

By generating positive and negative captions from SPOT event tuples, we
attempt to reevaluate video-language models on a Video-Text Retrieval
benchmark by comparing the model performance on positive and negative
captions. In this way, we can determine the sensitivity of video-language
models to specific manipulations and reveal if the model can distinguish and
understand event discrepancies.

Inspired by [22, 21], we also explore using false captions derived from
manipulated SPOT tuples as hard negative samples as post-training to improve
video-language models. Our experimental investigations suggest that augmenting
with additional negative captions can enhance video-language model
representations, particularly evident in downstream tasks.

To summarize, our contributions include:

  * •

we propose SPOT Prober to reevaluate if video-language models can distinguish
subtle but critical video discrepancy; by extracting SPOT tuples from the
video scene graph and generating truthful(positive) and manipulated(negative)
captions, we aim to probe the model sensitivity against different types of
manipulation;

  * •

we benchmark 5 existing video-language models with SPOT Prober on the common
Video-Text Retrieval task;

  * •

we give in-depth analysis of video-language models’ event understanding
abilities to distinguish subtle event discrepancies;

  * •

we explore using manipulated captions as hard negative samples for post-
training, despite improved performance on downstream tasks that demand
understanding more structured knowledge of video, the overheads of data
annotation and curation are still a hurdle.

(a) ActivityNet $Recall@1$

(b) Moviegraph $Recall@1$

(c) ActivityNet $Recall@5$

(d) Moviegraph $Recall@5$

Figure 2: We plot all models’ Relative Performance Gap $\Delta p$ of
$Recall@1$ and $Recall@5$ on two datasets. We denote different manipulation
types with different colors and models with different shapes. A large positive
number indicates the model’s sensitivity to the manipulation type and shows
the model can more distinguish such manipulation and comprehend the
corresponding event.

##  2 Related Works

Video-Language Model By learning a joint cross-modal representation, video-
language models bridge the gap between video representation and natural
language. Works have shown that with weak supervision of video-text pairs [29,
2], video-language models can learn good joint representations for downstream
video understanding tasks like Video-Text Retrieval [18, 13, 2], Video
Captioning [23] and Video Question Answering [26, 14, 13]. Recently, video
foundation models trained by large-scale video-text data, models like
InternVideo [28] even manifest excellent generalization performance on various
downstream tasks.

Understanding Vision-Language Model To better evaluate model capacities in
vision-language representation learning, probing methods have been an
important research topic. SVO probe [10] designs a benchmark centered on
verbs, which assesses subject, verb, and object triplets within the annotated
image–sentence pairs in English. This dataset is a valuable resource for
evaluating language and visual understanding models. [32] designs a benchmark
to reveal VLM’s poor understanding of compositional information, and a
composition-aware hard negative mining could improve the performance. [24]
introduces a novel benchmark to evaluate the ability of vision and language
models to conduct visio-linguistic compositional reasoning, and most SOTA
models can not outperform chance. [15] considers image representation event
structure information semantic meaning on image-text models. [20] generates
complex negative samples to evaluate VLM’s systematicity and productivity and
demonstrates that model performance consistently diminishes as novel
compositions dominate the retrieval set. [33] and [5] evaluate pre-trained
VLM’s fine-grained understanding abilities of nouns/verbs/attributes. [9]
introduces a question decomposition engine and designs compositional
consistency metrics to reveal that models struggle to reason correctly through
most compositions.

For videos, dynamic scenes and multiple events make video representation more
complicated and hard to probe. [21] investigates if video-language models can
tell the nuance difference between verbs. [1] probes if a video-language model
is sensitive to the temporal ordering. However, these two pioneering works
stay on a broad level.

To enhance video-language models’ capacities of understanding abilities like
verb understanding and compositional understanding, [6] uses generated
counterfactual samples in images and texts for the VQA task. [27] uses a text
encoder to synthesize complex negative pieces in feature space for ITM loss.
These methods shows that by injecting negative samples, vision-language models
can improve its representation capacities.

Video scene graphs in Video-Language dataset Video scene graphs represent
videos’ dynamic and multi-event nature in a graph format. However, annotating
video scene graphs is time and labor-intensive work. Benefit from the
contributions of the community, [30, 31, 25] provide dense and timestamped
scene graph-level annotations with rich types of predicates and attributes on
top of video data, which enables us to extract SPOT tuples and construct our
benchmark.

Type |  | Positive | Negative
---|---|---|---
temporal contact |  |  | A person with white skin is kneeling on a yellow yard
---
before eventually reaching out to touch the brown mulch
| A person with white skin is kneeling on the yellow yard,
---
while touching the brown mulch
temporal action |  |  | A silver metal knife is sliding,
---
and after a while, a white oven is opened
A silver metal knife is sliding as a white oven is being opened
temporal attribute |  |  A yellow bike after a while turns black |  A bike which is black, after a while its color is yellow
neighborhood attribute |  |  The color of the smoke ring is white, and the pipe is brown |  The smoke ring is brown in color, while the pipe is white in color
counterfactual contact |  |  A white chalk is drawn on the white street |  The white chalk is being erased from the white street
counterfactual action |  |  A green cloth punchbag is hanging |  A green cloth-made punchbag is swinging
counterfactual attribute |  |  A black bike is parked |  A shiny bike is parked
counterfactual spatial relationship |  |  A boy contestant is on a white hill. |  A boy contestant is standing beside the white hill
Table 1: We showcase the positive and negative captions generated from
extracted and manipulated event SPOT tuples on the ActivityNet dataset. We
highlight the truthful information in cyan and manipulated information in
magenta.

##  3 SPOT Prober: Benchmarking VLMs

###  3.1 Preliminary

Video scene graphs are a structured representation of video content. Videos
can be formulated as a concise form of video scene graphs:
$\mathcal{G}:=\mathcal{S}\times\mathcal{A}\times\mathcal{P}\times\mathcal{O}\times\mathcal{A}\times\mathcal{T}$.

Each tuple constitutes an event $\mathcal{E}$ in the video:

$\mathcal{E}=\cup_{t^{i}}(s,sa,p,o,oa,t^{i})$ (1)

###  3.2 Manipulating SPOT Tuples

By manipulating the event tuples, we can generate description foils that
differ from events in a non-trivial way. We define three manipulation methods:
temporal, neighborhood, and counterfactual manipulations. According to
manipulation targets, we can further define two manipulation patterns:
predicate manipulation and attribute manipulation.

Temporal Manipulation is strategically devised to interchange temporal
elements in a relative context. This highlights whether the video-language
models can sense the temporal ordering of two events, which is a key aspect of
videos.

For two events ¡Subject1, Subject attribute1, Predicate1, Object1, Timestamp1¿
and ¡Subject2, Subject attribute2, Predicate2, Object2, Timestamp2¿ in the
scene graph, the temporal manipulation
$\mathcal{M}_{\text{temporal}}^{\text{predicate}}$ and
$\mathcal{M}_{\text{temporal}}^{\text{attribute}}$ are defined as

$\displaystyle\mathcal{M}_{\text{temporal}}^{\text{predicate}}$
$\displaystyle=Swap\left((s^{1},p^{1},o^{1},t^{1})\cup(s^{2},p^{2},o^{2},t^{2})\right)$
(2)
$\displaystyle=\left((s^{1},p^{1},o^{1},t^{2})\cup(s^{2},p^{2},o^{2},t^{1})\right)$

$\displaystyle\mathcal{M}_{\text{temporal}}^{\text{attribute}}$
$\displaystyle=Swap\left((s^{1},sa^{1},t^{1})\cup(s^{1},sa^{2},t^{2})\right)$
(3) $\displaystyle=\left((s^{1},sa^{2},t^{1})\cup(s^{1},sa^{1},t^{2})\right)$

Neighborhood Manipulation is crafted to interchange attributes among distinct
neighboring entities in the scene graph. This highlights the ability to
connect predicates or attributes with entities.

For one event ¡Subject1, Subject attribute1, Predicate1, Object1, Object
attribute1¿ in the scene graph, the neighborhood manipulation
$\mathcal{M}_{\text{neighborhood}}^{\text{attribute}}$ is defined as

$\displaystyle\mathcal{M}_{\text{neighborhood}}^{\text{attribute}}$
$\displaystyle=Swap\left((s^{1},sa^{1},o^{1},oa^{1})\right)$ (4)
$\displaystyle=\left((s^{1},oa^{1},o^{1},sa^{1})\right)$

Counterfactual Manipulation is directed towards selecting a counterfactual
concept for the particular element within a tuple.

For a single event ¡Subject1, Subject attribute1, Predicate1, Object1¿ in the
scene graph, the counterfactual manipulation
$\mathcal{M}_{\text{counterfactual}}^{\text{predicate}}$ and
$\mathcal{M}_{\text{counterfactual}}^{\text{attribute}}$ are defined as

$\displaystyle\mathcal{M}_{\text{counterfactual}}^{\text{predicate}}$
$\displaystyle=Counter\left(s^{1},sa^{1},p^{1},o^{1}\right)$ (5)
$\displaystyle=\left(s^{1},sa^{1},p^{1}_{\text{-}},o^{1}\right)$

$\displaystyle\mathcal{M}_{\text{counterfactual}}^{\text{attribute}}$
$\displaystyle=Counter\left(s^{1},sa^{1},p^{1},o^{1}\right)$ (6)
$\displaystyle=\left((s^{1},sa^{1}_{\text{-}},p^{1},o^{1}\right))$

In this context, $sa^{1}_{\text{-}}$ and $p^{1}_{\text{-}}$ represent the
candidates selected as a counterfactual element.

Type |  | Positive | Negative
---|---|---|---
temporal interaction |  |  | Mikael Blomkvist who is male journalist asks Lisbeth Salander,
---
before Lisbeth Salander apologizes Mikael Blomkvist
| Mikael Blomkvist who is male journalist asks Lisbeth Salander,
---
after Lisbeth Salander apologizes Mikael Blomkvist
temporal action |  |  Workers cut branches off a tree, before Workers shred branches |  Workers cut branches off a tree, during Workers shred branches
temporal attribute |  |  Tommy is excited after a while Tommy becomes skeptical |  Tommy is skeptical after a while Tommy excited
neighborhood attribute |  |  |  A male Truck Driver is annoyed
---
and the Security Guard looks responsible and calm
|  A male Truck Driver looks responsible and calm
---
while the Security Guard looks annoyed.
counterfactual interaction |  |  Joey Donner talks to Bianca Stratford, Joey Donner, who is male |  Joey Donner flirts with Bianca Stratford, Joey Donner who is male
counterfactual action |  |  Greg Focker carries lawn chairs |  Greg Focker assembles lawn chairs
counterfactual attribute |  |  Cecilia is honest woman |  Cecilia is irresponsible woman
counterfactual relationship |  |  |  Lisbeth Salander is colleague of Mikael Blomkvist
---
|  Lisbeth Salander is the teacher of Mikael Blomkvist
---
Table 2: We showcase the positive and negative captions generated from
extracted and manipulated event SPOT tuples on the MovieGraphs dataset. We
highlight the truthful information in cyna and manipulated information in
magenta.

###  3.3 Data Preparation

Datasets We used two video datasets to establish our benchmark: ActivityNet
[4] and MovieGraphs [25]. To extract SPOT tuples, we utilize additional video
scene graph annotations in [31, 12] to augment the data sources. Datasets
statistics can be found in the Tab. 3.

Dataset | Domain |  | Video
---
Number
| Average
---
Duration
| Total
---
Caption
| Average
---
Caption Length
ActivityNet[4] | Activity | 11.5K | 180s | 16K | 35.11
MovieGraphs[25] | Movie | 7.6K | 44.28s | 7.6K | 8.1
Table 3: Statistics of two benchmarks. Original ActivityNet datasets provide the videos in ActivityNet, and the raw captions are provided in ActivityNet-Captions[12].  Manipulation Type | Number of Samples
---|---
temporal contact | 184
temporal action | 62
temporal attribute | 102
neighborhood attribute | 35
counterfactual contact | 1168
counterfactual action | 1008
counterfactual attribute | 818
counterfactual spatial relationship | 935
Table 4: Number of extracted SPOT tuples from ActivityNet in each manipulation category in our benchmark. Manipulation Type | Number of Samples
---|---
temporal interaction | 563
temporal action | 489
temporal attribute | 160
neighborhood attribute | 597
counterfactual relationship | 583
counterfactual interaction | 591
counterfactual action | 535
counterfactual attribute | 591
Table 5: Number of extracted SPOT tuples from MovieGraphs in each manipulation
category in our benchmark.

SPOT types Depending on the video scene graph annotation difference, we have
fine-grained types of predicates and attributes for each dataset.

By applying the above-defined manipulation methods on different tuple
components like Predicate and Attribute, and considering annotation
differences in datasets as shown in Tab. 6, we define 8 different manipulation
categories for ActivityNet and MovieGraphs.

Manipulation Details To control the consistency of the manipulated SPOT tuples
and original SPOT tuples, we only manipulate the predicates/attributes of the
same fine-grained type as describe in 4\- 5.

Dataset | Predicate Type | Attribute Type
---|---|---
ActivityNet |  | Occupation
Interaction | Gender
Action | Age
Relationship | Color
| Material
MovieGraphs | Interaction | Occupation
| Gender
Action | Age
| Ethnicity
Relationship | Emotion
| Appearance
Table 6: ActivityNet and MovieGraphs define different fine-grained types of
predicates and attributes.

LLM Decorator SPOT tuples are still far from natural descriptions in the video
language training corpus. Given a SPOT tuple, we employ GPT-3.5-turbo as a
controlled modifier to convert SPOT tuples into natural languages. We use a
prompt ”In this task, you are given a sentence, your job is to replace the
verb in the sentence with a verb which makes this sentence make sense, please
generate 10 sentences.” and a temperature coefficient to control the sentence
quality to avoid introducing misfacts. Several examples are delineated in Tab.
1\- 2.

###  3.4 Benchmark Metrics

To quantify the video-language models’ sensitivity to event discrepancies, we
reevaluate the Video-Text retrieval task on the original group of positive
captions and the control group of negative captions. We believe Video
Retrieval tasks are a good measure of learning the similarity of video and
text representations based on similarity-based metrics. We report the Relative
Performance Gap $\Delta p$ as follows:

$\Delta p=\frac{p-p_{control}}{p}$ (7)

where $p$ is the model performance on the original benchmark and $p_{control}$
is the model performance on the control group with manipulated captions.

The motivation behind this is that if the model cannot distinguish the
difference between positive and negative descriptions, the performance on the
control set with negative samples would drop dramatically, and $\Delta p$ can
be considerable. Otherwise, a small or even negative $\Delta p$ indicates that
the model cannot distinguish the nuanced difference between positive and
negative samples or is distracted by negative captions.

For all experiments, we report $\Delta p$ on both Video-to-Text and Text-to-
Video $Recall@k$, a standard evaluation metric on Video-Text Retrieval.

###  3.5 Baseline Model Choice

We reevaluate 5 video-language models: Clip4clip [18], Singularity [13],
X-CLIP [19], ALPRO [14], and InternVideo [28]. The first four are fine-tuned
datasets, which we again fine-tune with original videos and captions in
ActivityNet and MovieGraphs to ensure the model achieves comparable
performance. InternVideo is a large-scale pre-trained video-language model,
which we evaluate in a zero-shot setting.

Model | Visual Encoder | Textual Encoder | Pre-Training Objectives | Temporal Modeling | Fine-tuned
---|---|---|---|---|---
Clip4clip(mean) | ViT-B/32 | BERT | VTC | Mean-Pooling | ✓
Clip4clip(transf) | ViT-B/32 | BERT | VTC | Temporal transformer | ✓
Singularity-temporal |  BEiTBASE | BERT | VTC+VTM+MLM | Temporal transformer (2-layer) | ✓
X-CLIP | ViT-B/32 | BERT | VTC | Temporal transformer | ✓
ALPRO | ViT-B/16 |  BERTBASE | VTC+VTM+MLM+PEM | Temporal transformer+Mean-Pooling | ✓
InternVideo | ViT-L/14 | BERT | VTC | Linear Projection | ✗
Table 7: Comparison of model architecture. We compare 5 selected models (6 variants) on different dimensions on: Visual Encoder, Textual Encoder, Pre-training Objectives, Temporal Modeling, and if the model is fine-tuned by us on the datasets. Model |  | Temporal | Neighborhood | Counterfactual
---|---|---|---|---
| contact | action | attribute | attribute | contact | action | attribute | spatial relationship
| $\Delta p_{R@1}$ | $\Delta p_{R@5}$ | $\Delta p_{R@1}$ | $\Delta p_{R@5}$ | $\Delta p_{R@1}$ | $\Delta p_{R@5}$ | $\Delta p_{R@1}$ | $\Delta p_{R@5}$ | $\Delta p_{R@1}$ | $\Delta p_{R@5}$ | $\Delta p_{R@1}$ | $\Delta p_{R@5}$ | $\Delta p_{R@1}$ | $\Delta p_{R@5}$ | $\Delta p_{R@1}$ | $\Delta p_{R@5}$
Clip4clip(mean) | T2V | 0.00 | 0.00 | -0.21 | -0.10 | 0.16 | 0.02 | 0.14 | -0.03 | 0.04 | 0.03 | 0.29 | 0.28 | 0.12 | 0.14 | 0.03 | 0.03
V2T | 0.00 | 0.00 | 0.03 | 0.04 | -0.07 | 0.02 | 0.17 | 0.03 | 0.08 | 0.01 | 0.23 | 0.20 | 0.09 | 0.06 | -0.02 | 0.03
Clip4clip(transf) | T2V | 0.02 | -0.01 | -0.13 | -0.02 | 0.11 | 0.64 | 0.00 | 0.06 | 0.06 | 0.02 | 0.27 | 0.28 | 0.18 | 0.15 | 0.02 | -0.01
V2T | -0.01 | -0.01 | 0.06 | 0.00 | -0.09 | 0.07 | 0.17 | 0.00 | 0.00 | 0.05 | 0.19 | 0.18 | -0.19 | -0.02 | -0.10 | -0.03
Singularity-temporal | T2V | -0.01 | 0.01 | 0.25 | 0.11 | -0.17 | 0.11 | 0.13 | -0.03 | 0.16 | 0.03 | 0.32 | 0.21 | 0.34 | 0.14 | 0.07 | 0.03
V2T | -0.04 | -0.07 | 0.05 | 0.05 | 0.10 | 0.00 | 0.05 | 0.04 | 0.14 | 0.05 | 0.23 | 0.23 | 0.05 | 0.17 | 0.08 | 0.03
X-CLIP | T2V | 0.03 | -0.01 | -0.06 | 0.07 | -0.14 | 0.02 | 0.09 | 0.00 | 0.08 | 0.08 | 0.37 | 0.25 | 0.27 | 0.18 | 0.01 | 0.04
V2T | 0.03 | 0.01 | 0.17 | 0.05 | 0.08 | -0.03 | 0.13 | -0.03 | 0.05 | 0.09 | 0.11 | 0.10 | 0.00 | 0.08 | 0.06 | 0.03
ALPRO | T2V | 0.09 | 0.04 | -0.17 | -0.05 | 0.03 | 0.13 | 0.12 | 0.07 | -0.01 | 0.03 | 0.16 | 0.26 | 0.26 | 0.25 | -0.01 | 0.06
V2T | 0.04 | 0.01 | -0.03 | 0.02 | 0.07 | -0.07 | 0.06 | 0.07 | 0.11 | 0.06 | 0.28 | 0.27 | 0.34 | 0.18 | 0.06 | -0.02
InternVideo | T2V | 0.04 | 0.04 | 0.13 | 0.11 | -0.12 | 0.09 | 0.08 | 0.00 | 0.21 | 0.09 | 0.26 | 0.26 | 0.49 | 0.25 | 0.18 | 0.60
V2T | 0.08 | 0.02 | 0.18 | 0.16 | 0.00 | -0.02 | 0.00 | 0.06 | 0.16 | 0.11 | -0.02 | 0.07 | 0.13 | 0.06 | 0.04 | 0.04
Table 8: We report the Relative Performance Gap $\Delta p$ on the Video-Text Retrieval task on ActivityNet. A large positive number indicates the model’s sensitivity to the manipulation and shows the model can more distinguish such manipulation and comprehend the event. Model |  | Temporal | Neighborhood | Counterfactual
---|---|---|---|---
| interaction | action | attribute | attribute | interaction | action | attribute | relationship
| $\Delta p_{R@1}$ | $\Delta p_{R@5}$ | $\Delta p_{R@1}$ | $\Delta p_{R@5}$ | $\Delta p_{R@1}$ | $\Delta p_{R@5}$ | $\Delta p_{R@1}$ | $\Delta p_{R@5}$ | $\Delta p_{R@1}$ | $\Delta p_{R@5}$ | $\Delta p_{R@1}$ | $\Delta p_{R@5}$ | $\Delta p_{R@1}$ | $\Delta p_{R@5}$ | $\Delta p_{R@1}$ | $\Delta p_{R@5}$
Clip4clip(mean) | T2V | 0.05 | -0.01 | 0.00 | 0.01 | -0.10 | 0.01 | 0.03 | 0.01 | 0.17 | 0.03 | 0.17 | 0.07 | -0.07 | 0.04 | 0.09 | 0.02
V2T | 0.02 | 0.01 | 0.00 | -0.01 | -0.05 | 0.02 | -0.04 | 0.00 | 0.07 | 0.05 | 0.10 | 0.06 | 0.17 | 0.02 | 0.09 | 0.02
Clip4clip(transf) | T2V | 0.03 | -0.01 | 0.03 | 0.00 | 0.05 | 0.06 | 0.03 | 0.00 | 0.07 | 0.04 | 0.09 | 0.07 | 0.07 | 0.01 | 0.11 | -0.03
V2T | -0.04 | -0.02 | 0.03 | -0.03 | 0.00 | 0.08 | 0.00 | 0.00 | 0.12 | 0.06 | 0.09 | 0.04 | -0.06 | 0.03 | -0.02 | 0.00
Singularity-temporal | T2V | 0.10 | -0.01 | -0.18 | 0.03 | 0.18 | -0.14 | 0.25 | -0.02 | 0.08 | -0.06 | 0.00 | 0.07 | 0.00 | 0.04 | 0.00 | 0.09
V2T | 0.00 | -0.04 | 0.15 | -0.01 | 0.05 | -0.04 | 0.08 | 0.03 | 0.20 | 0.02 | 0.06 | 0.06 | 0.12 | 0.13 | 0.00 | 0.09
X-CLIP | T2V | 0.02 | 0.01 | 0.05 | 0.00 | -0.23 | 0.03 | 0.02 | 0.04 | 0.18 | 0.03 | 0.11 | 0.05 | 0.02 | 0.05 | -0.05 | 0.04
V2T | -0.03 | 0.00 | -0.02 | 0.01 | -0.21 | 0.00 | -0.02 | 0.04 | 0.10 | 0.05 | 0.12 | 0.10 | 0.07 | 0.04 | -0.02 | 0.03
ALPRO | T2V | 0.11 | -0.02 | 0.03 | 0.02 | 0.04 | -0.07 | -0.01 | -0.03 | 0.21 | 0.04 | 0.07 | 0.12 | 0.12 | 0.08 | 0.07 | 0.04
V2T | 0.11 | -0.02 | 0.07 | 0.01 | -0.18 | 0.02 | -0.06 | -0.04 | 0.15 | 0.08 | 0.22 | 0.14 | 0.08 | 0.13 | 0.03 | 0.06
InternVideo | T2V | -0.10 | 0.02 | -0.02 | 0.01 | 0.00 | 0.05 | -0.08 | 0.02 | 0.16 | 0.02 | 0.13 | 0.03 | 0.12 | 0.03 | 0.02 | 0.00
V2T | 0.17 | 0.02 | 0.04 | -0.02 | 0.18 | -0.04 | 0.00 | 0.00 | 0.50 | 0.18 | -0.08 | 0.04 | 0.19 | 0.11 | 0.10 | 0.03
Table 9: We report the Relative Performance Gap $\Delta p$ on the Video-Text
Retrieval task on MovieGraphs. A large positive number indicates the model’s
sensitivity to the manipulation and shows the model can more distinguish such
manipulation and comprehend the event.

Clip4clip[18] is built upon the CLIP framework and explores different temporal
modeling modules to address video-text retrieval. Our experiments employ two
distinct similarity calculators: a parameter-free type, specifically mean
pooling, and a sequential type based on the Transformer Encoder. For brevity,
we denote them as Clip4clip(mean) and Clip4clip(transf), respectively.

Singularity[13] exploits the single-frame bias of video and adopts an
extremely sparse sampling method on videos. We use the Singularity-temporal
variant described in [13] with a 2-layer transformer-based temporal encoder.

X-CLIP [19] stands out as a multi-grained contrastive model for video-text
retrieval. It aims to aggregate fine-grained and cross-grained similarity
matrices to achieve instance-level similarity.

ALPRO [14] functions on sparsely-sampled video frames, achieving enhanced
cross-modal alignment without relying on explicit object detectors.

InternVideo [28] serves as a comprehensive video foundation model. It adopts a
transformer-based video and text encoder trained on contrastive learning loss
and masked vision loss. We use the official implementation and pre-trained
model of InternVideo with a CLIP-based Vision Transformer as the video
encoder.

A systematic model comparison is listed in Tab. 7.

###  3.6 Findings

We report the relative performance gap $\Delta p$ in Tab. 8\- 9. To illustrate
the numbers in a clearer way, we also plot all models’ $\Delta p$ on a scatter
plot as shown in Fig. 2.

Performance on Retrieval Benchmark In general, we observe that the majority of
video-language models exhibit minimal or even negative $\Delta p$ across
various manipulation types. The results suggest that video-language models
struggle to differentiate or may even be affected by manipulated captions,
thereby restricting their proficiency in comprehending events. Among all the
models, ALPRO manifests comparable performance. We speculate that with entity-
level alignment, ALPRO can understand fine-grained event better.

Temporal manipulations are the hardest. As demonstrated in Fig. 2, it is
evident that video-language models lack sensitivity to various temporal
manipulations (near the origin point). This indicates their limited ability to
comprehend events with temporal discrepancies, the chronological order of
events, and even the nuanced changes in attributes.

Conterfactual manipulation is more distinguishable As shown in Fig. 2, models
exhibit a significantly higher sensitivity to counterfactual manipulations on
predicates and attributes. This observation suggests that video-language
models possess a considerable understanding of discrepancies that diverge from
the video content.

Even foundation models cannot distinguish We also observe that large-scale
pre-trained models like InternVideo [28] show low sensitivity to manipulations
and cannot distinguish the nuance between positive and negative. It shows that
large-scale datasets can benefit learning video representations on a broad
level but cannot discriminate more fine-grained events.

Understanding Spatial Relationship in Events is an interesting dimension
probing task. We find that on ActivityNet, most models show no sensitivity to
spatial relationship manipulation.

Understanding Abstract We notice that dataset annotations involve many
abstract concepts in event descriptions like emotion/personality attributes.
These attributes undoubtedly are much harder to comprehend and demand video-
language models of visual semantic reasoning abilities. As indicated in Fig.
2,

Out-of-context Event Understanding We also notice that scene graph annotations
involve complicated out-of-context concepts like name references like ”Forrest
Gump” and relationships like ”the sister of,” which cannot be deduced from the
video content directly but demands the video-language model of commonsense
knowledge or more comprehensive background information.

We include a detailed case study in the Supplementary Material for qualitative
studies.

In-video manipulations are harder We have demonstrated that existing video-
language models cannot distinguish subtle discrepancies, particularly in
relation to temporal and neighborhood manipulation within videos. These types
of manipulation are undeniably deceptive. Furthermore, we hypothesize that the
presence of noisy data of this nature in pre-training datasets may also
contribute to this issue.

##  4 Learning manipulated events as Hard Negative Samples

In this section, we explore the feasibility of leveraging negative captions of
manipulated SPOT tuples as hard negative training, as suggested in [21, 22,
11] and assess the model on a fine-grained Video Question Answering task to
see if hard negative samples can effectively help video-language models with
event understanding.

###  4.1 Experiments

We utilize a two-stage pipeline to post-train the video-language model with
hard negative samples. (1) we post-train the video and text encoders using
video-text contrastive learning with hard negative samples; (2) we freeze the
encoders and incorporate task-specific decoders for downstream tasks.

Prototype Model We selected a two-stream structure as our base model, which
consists of a ViT-based video encoder and a BERT-based text encoder. This
structure is widely utilized in the following works: Clip4clip[18] and
Singularity[13]. The encoders that are enhanced with post-training using hard
negative samples are kept frozen while an additional transformer-based decoder
is trained for the QA settings.

Hard-negative Post-training Loss Video-text Contrastive learning has
expeditiously emerged as the predominant methodology for multimodal alignment.
Drawing inspiration from the work [21, 22, 1], we employ the hard negative
noise contrastive multimodal alignment loss in this context to perform
reweighting of negative samples according to their hardness. Applied to any
given in-batch $X=\\{(v^{i},t^{i})\\}_{i=1}^{\mathcal{N}_{\text{in}}}$ with
hard negative sample $\\{t^{k}\\}_{k=1}^{\mathcal{N}^{i}_{\text{gen}}}$
comprising feature-encoded image-text pairs, our CL loss enhanced with
generated hard negative samples is defined as:

$\begin{aligned}
\mathcal{L}_{CL}=&-\log\frac{\exp(v^{i}\cdot{t^{i}}/\tau)}{{\exp(v^{i}\cdot{t^{i}}/\tau)}+\sum^{\mathcal{N}_{\text{in}}}_{j\neq
i}{\exp(v^{i}\cdot{t^{j}}/\tau)\omega^{v\rightarrow
t}_{i,j}}+\sum^{\mathcal{N}^{i}_{\text{gen}}}_{k=1}{\exp(v^{i}\cdot{t^{k}}/\tau)\omega^{v\rightarrow
t}_{i,k}}}\\\
&-\log\frac{\exp(v^{i}\cdot{t^{i}}/\tau)}{{\exp(v^{i}\cdot{t^{i}}/\tau)}+\sum^{\mathcal{N}_{\text{in}}}_{j\neq
i}{\exp(v^{j}\cdot{t^{i}}/\tau)\omega^{t\rightarrow v}_{j,i}}}\end{aligned}$

(8)

Here $\tau$ represents a temperature parameter, $\mathcal{N}_{\text{in}}$
denotes the set of samples in the batch, and $\mathcal{N}^{i}_{\text{gen}}$ is
the set of generated negative samples based on $t^{i}$. And weight functions
$\omega^{v\rightarrow t}_{i,j}$ and $\omega^{t\rightarrow v}_{j,i}$ are
defined as:

$\begin{aligned} &\omega^{v\rightarrow
t}_{i,j}=\frac{(\mathcal{N}_{\text{in}}+\mathcal{N}^{i}_{\text{gen}}-1)\exp\left(\beta
v^{i}\cdot t^{j}/\tau\right)}{\sum^{\mathcal{N}_{\text{in}}}_{m\neq
i}\exp\left(v^{i}\cdot t^{m}/\tau\right)},\\\ &\omega^{t\rightarrow
v}_{j,i}=\frac{(\mathcal{N}_{\text{in}}-1)\exp\left(\beta v^{j}\cdot
t^{i}/\tau\right)}{\sum^{\mathcal{N}_{\text{in}}}_{m\neq
i}\exp\left(v^{m}\cdot t^{i}/\tau\right)}\end{aligned}$

(9)

Implementation Details Inspired by [18, 13, 14], we use three learning
objectives: ITC, ITM, and MLM loss. We only adopt generated hard negative on
ITC loss. We initialize the vision encoder with the BEiTBASE [3] model, pre-
trained on ImageNet-21K [7]. Meanwhile, the text encoder is initialized with
the first 9 layers of BERTBASE [8]. Additionally, a 2-layer temporal encoder
is incorporated, and we extract four frames for each video.

Training Details We adopt a two-stage fine-tuning: data-specific post-training
and task-specific fine-tuning. Our optimization strategy spans 10 epochs,
utilizing the AdamW [17] optimizer with an initial learning rate of 1e-4. In
the first epoch, a learning rate warm-up is applied, followed by cosine decay
[16] to 1e-6 for the remainder of the training. To expedite the training
process, we employ mixed precision training. For hard negative post-training,
we use a total batch size of 64, and the model is trained on 2 NVIDIA A100
GPUs. For the Question-Answering (QA) task, we keep both encoders from the
post-training frozen and only train the text decoder.

###  4.2 Evaluation

To better inspect the model’s event understanding ability after hard negative
training, we evaluate the model on a Compositional Video QA benchmark, AnetQA
[31], which features answering fine-grained and temporal questions.

As shown in Tab. 10, injecting manipulated SPOT tuples as hard negative
samples is beneficial on Compositional Video QA benchmark. This indicates that
leveraging manipulated captions as augmented negative samples can help video-
language models with event understanding by learning subtle discrepancies in
the video.

###  4.3 Discussion

|  Accuracy ($\%$)
---|---
| temporal | counterfactual | overall
CL loss w/o HN-Samples | 29.18 | 3.80 | 21.47
CL loss w/ HN-Samples | 33.63 | 5.55 | 22.34
Table 10: The results on the test subset of AnetQA, with and without hard
negative, are based on a frozen post-trained Visual and Text encoder, and a
trained QA decoder on top. It has been shown that leveraging negative samples
from SPOT tuples can improve the performance on fine-grained categories of
video questions.

Limitations Despite the performance improvement with hard negative sampling,
the considerable overhead in scene graph-level video data annotation and
curation persists. Given the data scale in pre-training video-language models,
the pivotal issue of efficiently providing clean and diverse negative data for
video-language pre-training remains an open question.

Future Direction Considering the impractical hard negative mining from video
scene graph annotations, and the observation we state that video-language
models are more likely to be deceived by temporal and neighborhood
manipulation that happens in video, we find it interesting to discover whether
we can use temporal masked video prediction as a weak-supervised learning
object to guide the video-language models to understand multi-grained and
temporal event in videos.

##  5 Conclusion

In this work, we present the SPOT prober, which is designed to assess the
event-level discrepancy detection capabilities of existing video-language
models, serving as an indicator of their event understanding ability.

Our methodology involves extracting events from videos as tuples (¡Subject,
Predicate, Object, Attribute, Timestamps¿) and systematically manipulating
tuple components to generate false event tuples. By evaluating the existing
video-language models using both positive and negative captions generated from
these tuples, we discover that they struggle to distinguish the majority of
manipulated events like temporally manipulated events; even large-scale pre-
trained video-language models cannot survive. Based on these findings, we
propose incorporating these manipulated event captions as hard negative
samples, which effectively enhance models for event understanding.

Although it is still impractical to use hard negative samples generated from
crowdsourced scene graph annotations, as it requires intensive annotation
overhead, we believe this study sheds light on the weaknesses of video-
language models in event understanding.

## References

  * Bagad et al. [2023] Piyush Bagad, Makarand Tapaswi, and Cees GM Snoek.  Test of time: Instilling video-language models with a sense of time.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2503–2516, 2023.
  * Bain et al. [2021] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In _IEEE International Conference on Computer Vision_ , 2021.
  * Bao et al. [2022] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  BEit: BERT pre-training of image transformers.  In _International Conference on Learning Representations_ , 2022.
  * Caba Heilbron et al. [2015] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In _Proceedings of the ieee conference on computer vision and pattern recognition_ , pages 961–970, 2015.
  * Cascante-Bonilla et al. [2023] Paola Cascante-Bonilla, Khaled Shehada, James Seale Smith, Sivan Doveh, Donghyun Kim, Rameswar Panda, Gul Varol, Aude Oliva, Vicente Ordonez, Rogerio Feris, et al.  Going beyond nouns with vision & language models using synthetic data.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 20155–20165, 2023.
  * Chen et al. [2020] Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang.  Counterfactual samples synthesizing for robust visual question answering.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10800–10809, 2020.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  2009 ieee conference on computer vision and pattern recognition.  2009\.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * Gandhi et al. [2022] Mona Gandhi, Mustafa Omer Gul, Eva Prakash, Madeleine Grunde-McLaughlin, Ranjay Krishna, and Maneesh Agrawala.  Measuring Compositional Consistency for Video Question Answering.  In _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5036–5045, New Orleans, LA, USA, 2022. IEEE.
  * Hendricks and Nematzadeh [2021] Lisa Anne Hendricks and Aida Nematzadeh.  Probing image-language transformers for verb understanding.  In _Findings 2021_ , pages 3635–3644, Online, 2021. Association for Computational Linguistics.
  * Huang et al. [2023] Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Tangjie Lv, Zhipeng Hu, and Wen Zhang.  Structure-CLIP: Enhance Multi-modal Language Representations with Structure Knowledge, 2023.
  * Krishna et al. [2017] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles.  Dense-captioning events in videos.  In _International Conference on Computer Vision (ICCV)_ , 2017.
  * Lei et al. [2022] Jie Lei, Tamara L. Berg, and Mohit Bansal.  Revealing single frame bias for video-and-language learning, 2022.
  * Li et al. [2022a] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven C.H. Hoi.  Align and prompt: Video-and-language pre-training with entity prompts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022a.
  * Li et al. [2022b] Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang.  CLIP-Event: Connecting Text and Images With Event Structures.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16420–16429, 2022b.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts, 2017.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_ , 2019.
  * Luo et al. [2022] Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li.  Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.  _Neurocomputing_ , 508:293–304, 2022.
  * Ma et al. [2022] Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji.  X-clip: End-to-end multi-grained contrastive learning for video-text retrieval.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 638–647, 2022.
  * Ma et al. [2023] Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna.  CREPE: Can Vision-Language Foundation Models Reason Compositionally?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10910–10921, 2023.
  * Momeni et al. [2023] Liliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman, and Cordelia Schmid.  Verbs in action: Improving verb understanding in video-language models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15579–15591, 2023.
  * Radenovic et al. [2023] Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan.  Filtering, distillation, and hard negatives for vision-language pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6967–6977, 2023.
  * Sun et al. [2019] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 7464–7473, 2019.
  * Thrush et al. [2022] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross.  Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5238–5248, 2022.
  * Vicol et al. [2018] Paul Vicol, Makarand Tapaswi, Lluis Castrejon, and Sanja Fidler.  Moviegraphs: Towards understanding human-centric situations from videos.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8581–8590, 2018.
  * Wang et al. [2023a] Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, et al.  All in one: Exploring unified video-language pre-training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6598–6608, 2023a.
  * Wang et al. [2023b] Weihan Wang, Zhen Yang, Bin Xu, Juanzi Li, and Yankui Sun.  Vilta: Enhancing vision-language pre-training through textual augmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3158–3169, 2023b.
  * Wang et al. [2022] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao.  Internvideo: General video foundation models via generative and discriminative learning.  _arXiv preprint arXiv:2212.03191_ , 2022.
  * Wang et al. [2023c] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, et al.  Internvid: A large-scale video-text dataset for multimodal understanding and generation.  _arXiv preprint arXiv:2307.06942_ , 2023c.
  * Wu et al. [2021] Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua Tenenbaum, and Chuang Gan.  Star: A benchmark for situated reasoning in real-world videos.  In _Thirty-fifth Conference on Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Yu et al. [2023] Zhou Yu, Lixiang Zheng, Zhou Zhao, Fei Wu, Jianping Fan, Kui Ren, and Jun Yu.  Anetqa: A large-scale benchmark for fine-grained compositional reasoning over untrimmed videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 23191–23200, 2023.
  * Yuksekgonul et al. [2022] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou.  When and why vision-language models behave like bags-of-words, and what to do about it?  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Zhao et al. [2022] Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin.  VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations, 2022.

Figure 3: Case study: Understanding Spatial Relationship in Events. This
example is sourced from the ActivityNet dataset. In this case, we manipulate
the Spatial Relationship to assess the video understanding of each model. We
emphasize truthful information in cyan and manipulated information in magenta.
Similarly, we annotate the incorrect predictions in magenta and the correct
predictions in cyan for clarity. The “percentage” denotes the relative
similarity score calculated from each model.

In the Supplementary Material, we extend our discussion in Sec.3.6 with a case
study in Sec. A.

##  Appendix A Case Study

To better reveal the Video-Language Models’ event understanding abilities, we
conduct a case study of different facets of special cases as we have discussed
in Sec.3.6.

Understanding Spatial Relationship in Events In Figure 3, we present an
example to demonstrate the models’ capability to understand spatial
relationships in events. Specifically, the subtle difference between “In the
…” and “… on top” jeopardizes half of the models and shows that they are not
sensitive enough to spatial information.

Understanding Abstract In Fig.4, we present an example to illustrate the
models’ capability to understand abstract concepts. Specifically, we examine
the emotion expressions ”Dylan is uncomfortable” and ”Dylan is shy,” which
present a challenge to the models as they require discerning subtle
distinctions between human emotions. These emotions are inherently abstract
attributes that necessitate a high level of semantic understanding.

Not all models can understand the subtle discrepancy between them. However, we
found that InternVideo as a foundation model, can predict correctly with a
higher confidence ($53.6\%vs46.4\%$).

Figure 4: Case study: Understanding Abstract. This example is sourced from the
MovieGraph dataset. In this context, we manipulate the emotional attribute of
the entity.

Out-of-context Event Understanding Not all relationships can be deduced by the
local visual cues. In Fig.5 and Fig.6, we present two examples to showcase how
the models can understand events that are out of context. By ”out of context,”
we mean events that require additional contextual information to be
distinguished. For instance, differentiating between a “best friend/brother”
relationship and a “colleague/friend” relationship poses a challenge for the
model because this distinction cannot be directly inferred from the video
data. Even though almost all models make correct predictions, we cannot assert
the manipulated captions are totally wrong without additional information.

This actually reveals a blind spot of current video-language models for video
understanding: they lack the ability to understand out-of-context knowledge.

Figure 5: Case study: Out-of-context Event Understanding. This example is also
drawn from the MovieGraph dataset. In this instance, we manipulate the
relationship between two entities. Even though almost all models make a
correct prediction, it cannot be excluded that ‘brother’ is also correct.
Figure 6: Case study: Another example of Out-of-context Event Understandings
from the MovieGraph dataset. Similarly, ‘friend’ and ‘brother’ cannot be
distinguished without more background information of the characters.


# On the existence of Newton series and the asymptotics of finite differences

Glenn Bruda

###### Abstract.

Suppose a complex function $f$ has a Lebesgue measurable inverse Laplace
transform. We show that the $n$th order forward and backward differences of
$f$ at $z_{0}\in\mathbb{C}$ tend to zero as $n\to\infty$ whenever $z_{0}$ lies
in the region of absolute convergence of $f$. Under the same hypothesis, we
show that the Newton series of $f$ centered at $z_{0}$ exists and converges in
the half-plane $\Re(z)>\Re(z_{0})$. Assuming instead that $f$ has a Lebesgue
measurable inverse Fourier transform, we show that the $n$th order forward,
backward, and central differences of $f$ at any $y\in\mathbb{R}$ are
$o(2^{n})$. Consequently, we show that the binomial sum $\sum_{k\geq
0}{n\choose k}f(k)$ is $o(2^{n})$.

##  1\. Introduction

For a complex function $f$, we define the forward, backward, and central
differences of $f$ at $z\in\mathbb{C}$, denoted $\Delta_{h}^{n}[f](z)$,
$\nabla_{h}^{n}[f](z)$, $\delta_{h}^{n}[f](z)$ respectively, as

$\displaystyle\Delta_{h}^{n}[f](z)$
$\displaystyle\vcentcolon=\sum_{k=0}^{n}{n\choose k}f(z+kh)(-1)^{n-k},$
$\displaystyle\nabla_{h}^{n}[f](z)$
$\displaystyle\vcentcolon=\sum_{k=0}^{n}{n\choose k}f(z-kh)(-1)^{k},$
$\displaystyle\delta_{h}^{n}[f](z)$
$\displaystyle\vcentcolon=\sum_{k=0}^{n}{n\choose
k}f\left(z+\left(\frac{n}{2}-k\right)h\right)(-1)^{k},$

where $h\in\mathbb{R}^{+}$ and $n\in\mathbb{N}\vcentcolon=\\{0,1,2,\cdots\\}$.
By convention, if the subscript $h$ of a finite difference is omitted, $h$ is
assumed to be $1$.

In general, the limiting behavior of finite differences can be difficult to
determine. In [flajoletsedgewick], Flajolet and Sedgewick used the Nørlund-
Rice integral111See [flajoletsedgewick, Lemma 1]. to determine the asymptotics
of many finite differences relevant to combinatorics and computer science.
Being of interest to such fields, most of these differences do not converge to
zero. For some of these differences, this can be surprising, particularly when
the function differenced converges to zero as its argument tends to infinity.
Furthermore, the finite differences of many functions, even those that are
unbounded such as non-constant polynomials, do converge to zero. In this
article, we clarify the seemingly unpredictable limiting behavior of forward
and backward differences by analyzing a Laplace transform integral formula. In
particular, we prove the following result:

######  Proposition 1.

Assuming it exists, suppose the inverse Laplace transform of $f$ is a linear
combination of Lebesgue measurable functions and elements of the set

$\left\\{\delta^{(m)}(x-a):m\in\mathbb{N},a\in\mathbb{R}^{\geq 0}\right\\},$

where $\delta^{(m)}$ is the $m$th distributional derivative of the Dirac delta
function. Let $z\in\mathbb{C}$ lie in the region of absolute convergence of
$f$ and let $h\in\mathbb{R}^{+}$. Then

$\displaystyle\lim_{n\to\infty}\sum_{k=0}^{n}{n\choose k}f(z+kh)(-1)^{n-k}=0.$

Newton series222Also known as _Newton’s forward difference formula_. See page
6 for definition., often described as a “finite analog of a Taylor series”
[wolfram], can be tricky to prove the existence of. Though Newton series exist
for a variety of analytic functions, there are curious exceptions (e.g., the
canonical $\sin\pi z$). Using the aforementioned Laplace transform integral
formula, we guarantee the existence of Newton series for certain functions:

######  Proposition 2.

Assuming it exists, suppose the inverse Laplace transform of $f$ is a linear
combination of Lebesgue measurable functions and elements of the set

$\left\\{\delta^{(m)}(x-a):m\in\mathbb{N},a\in[0,\log 2)\right\\}.$

Let $z_{0}\in\mathbb{C}$ lie in the region of absolute convergence of $f$.
Then the Newton series of $f$ centered at $z_{0}$ exists and converges for all
$z\in\mathbb{C}:\Re(z)>\Re(z_{0})$.

This result induces the corollary that, for an appropriately chosen
$z_{0}\in\mathbb{C}$, any rational function $R$ with real coefficients admits
a Newton series centered at $z_{0}$ which is convergent in the half-plane
$\Re(z)>\Re(z_{0})$. In particular, we can take $z_{0}$ to be any complex
number such that its real part is strictly larger than the real parts of every
pole of $R$ in the right half-plane. If $R$ has no poles in the right half-
plane, then we must take $\Re(z_{0})>0$.

By making the fairly strong assumption that $f$ has an inverse Laplace
transform (alongside other conditions), we are able to show that the forward
and backward differences of $f$ converge to zero. If we instead assume that
$f$ has a Lebesgue measurable inverse Fourier transform (certain translations
of the Dirac delta function and its derivatives are permitted), we find the
$n$th order forward, backward, and central differences of $f$ at any
$y\in\mathbb{R}$ to be $o(2^{n})$. A corollary to this gives a similar result
for general $n$-term binomial sums:

######  Proposition 3.

Assuming it exists, suppose the inverse Fourier transform of $f$ is a linear
combination of Lebesgue measurable functions and elements of the set

$\\{\delta^{(m)}(x-a):m\in\mathbb{N},a\in\mathbb{R}\backslash\mathbb{Z}\\}.$

Then

$\displaystyle\sum_{k=0}^{n}{n\choose k}f(k)=o(2^{n}).$

Following this introduction, the bulk of this article is split into two parts:
an analysis of finite differences using a Laplace transform integral formula
and an analysis of finite differences using Fourier transform integral
formulas. The entirety of the former is dedicated to proving Proposition 1 and
Proposition 2 and their corollaries, while the latter’s primary focus is
proving Proposition 3.

##  2\. Analysis of a Laplace transform integral formula

######  Definition 1 (Laplace regions of conditional and absolute
convergence).

Suppose $f$ has an inverse Laplace transform. Then the Laplace region of
conditional convergence of $f$, denoted $\mathcal{C}_{L}(f)$, is the set

$\displaystyle\mathcal{C}_{L}(f)\vcentcolon=\left\\{z\in\mathbb{C}:\int_{0}^{\infty}e^{-zt}\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}^{-1}f(t)dt<\infty\right\\}.$

The Laplace region of absolute convergence of $f$, denoted
$\mathcal{A}_{L}(f)$, is the set

$\displaystyle\mathcal{A}_{L}(f)\vcentcolon=\left\\{z\in\mathbb{C}:\int_{0}^{\infty}\left|e^{-zt}\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}^{-1}f(t)\right|dt<\infty\right\\}.$

######  Lemma 1.

Suppose $f$ has an inverse Laplace transform. Let $n\in\mathbb{N}$,
$h\in\mathbb{R}^{+}$, and $z\in\mathcal{C}_{L}(f)$. Then

$\displaystyle\Delta_{h}^{n}[f](z)=\int_{0}^{\infty}e^{-zt}(e^{-ht}-1)^{n}\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}^{-1}f(t)dt.$

###### Proof.

Let $n\in\mathbb{N}$ and $h\in\mathbb{R}^{+}$. Let $f\in\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
<EMAIL_ADDRESS>Let $z\in\mathcal{C}_{L}(f)$. Denoting $g=\mbox{\hskip 6.25002pt\hskip
1.40623pt{1 0 -.45 1}\hbox
<EMAIL_ADDRESS>for brevity, note $\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
<EMAIL_ADDRESS>So we have that

$\displaystyle\Delta_{h}^{n}[\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45
1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}g](z)$
$\displaystyle=\sum_{k=0}^{n}{n\choose k}(-1)^{n-k}\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}g(z+kh)$
$\displaystyle=\sum_{k=0}^{n}{n\choose
k}(-1)^{n-k}\int_{0}^{\infty}e^{-(z+kh)t}g(t)dt$
$\displaystyle=\int_{0}^{\infty}e^{-zt}g(t)\sum_{k=0}^{n}{n\choose
k}(-1)^{n-k}e^{-htk}dt$
$\displaystyle=\int_{0}^{\infty}e^{-zt}(e^{-ht}-1)^{n}g(t)dt.$

Thus,

$\displaystyle\Delta_{h}^{n}[f](z)=\int_{0}^{\infty}e^{-zt}(e^{-ht}-1)^{n}\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}^{-1}f(t)dt,$

as desired. ∎

One of the two applications of Lemma 1 arises in proving the following
theorem, wherein we show when $\lim_{n\to\infty}\Delta^{n}_{h}[f](z)=0$. To do
this, we assume that $z\in\mathcal{A}_{L}(f)$; by [analyticinROAC, Subsection
2.7.3], this implies that $f$ is analytic at $z$. Since finite differences are
a way of approximating derivatives, it is not surprising that $n$th order
forward differences converge when we make assumptions on $f$ that imply
analyticity. However, analyticity is not enough, as shown in
[flajoletsedgewick, Example 2].

######  Definition 2.

For $m\in\mathbb{N}$, we write $\delta^{(m)}$ to denote the $m$th
distributional derivative of the Dirac delta function.

######  Theorem 1.

Assuming it exists, suppose that $\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0
-.45 1}\hbox
<EMAIL_ADDRESS>is a linear combination of Lebesgue measurable functions and elements of the
set $\left\\{\delta^{(m)}(x-a):m\in\mathbb{N},a\in\mathbb{R}^{\geq
0}\right\\}$. Let $z\in\mathcal{A}_{L}(f)$ and $h\in\mathbb{R}^{+}$. Then

$\displaystyle\lim_{n\to\infty}\Delta_{h}^{n}[f](z)=0.$

###### Proof.

Let $z\in\mathcal{A}_{L}(f)$. Let $h\in\mathbb{R}^{+}$. Suppose that
$\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
<EMAIL_ADDRESS>is a linear combination of Lebesgue measurable functions and elements of the
set $\left\\{\delta^{(m)}(x-a):m\in\mathbb{N},a\in\mathbb{R}^{\geq
0}\right\\}$; that is,

$\displaystyle\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}^{-1}f(t)=\sum_{k=0}^{u}a_{k}g_{k}(t)+\sum_{k=0}^{j}b_{k}\delta^{(k)}(t-c_{k}),$

where $u,j\in\mathbb{N}$, each $g_{k}$ is a Lebesgue measurable function,
$(a_{k}),(b_{k})$ are sequences from $\mathbb{C}$, and $(c_{k})$ is a sequence
from $\mathbb{R}^{\geq 0}$. As a finite sum of Lebesgue measurable functions
is Lebesgue measurable, without loss of generality we have that

$\displaystyle\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}^{-1}f(t)=m(t)+\sum_{k=0}^{j}b_{k}\delta^{(k)}(t-c_{k})$

for some Lebesgue measurable function $m$. We know that $\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
<EMAIL_ADDRESS>exists at $z$ since $m$ is a difference of functions with a Laplace transform
existing at $z$. So

$\displaystyle f(z)=\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m(z)+\sum_{k=0}^{j}b_{k}e^{-c_{k}z}z^{k}.$

Let $p(z)=\sum_{k=0}^{j}b_{k}e^{-c_{k}z}z^{k}$. Then

$\displaystyle\Delta_{h}^{n}[f](z)=\Delta_{h}^{n}[\mbox{\hskip 6.25002pt\hskip
1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m](z)+\Delta_{h}^{n}[p](z).$

Let $N\in\\{0,1,\cdots,j\\}$. We now argue that
$\Delta^{n}_{h}[e^{-c_{N}z}z^{N}](z)$ converges to zero. We have that

$\displaystyle\Delta^{n}_{h}[e^{-c_{N}z}z^{N}](z)$ $\displaystyle=\sum_{k\geq
0}{n\choose k}(-1)^{n-k}e^{-c_{N}(z+kh)}(z+kh)^{N}$
$\displaystyle=e^{-c_{N}z}\sum_{b\geq 0}{N\choose b}z^{N-b}\sum_{k\geq
0}{n\choose k}(-1)^{n-k}e^{-c_{N}kh}(kh)^{b}.$

Note

$\displaystyle\sum_{k\geq 0}{n\choose
k}(-1)^{n-k}e^{-tkh}(kh)^{b}=(-1)^{b}\frac{d^{b}}{dt^{b}}\left(e^{-th}-1\right)^{n}=O(n^{b}(e^{-th}-1)^{n}).$

Thus by taking $t=c_{N}$,

$\displaystyle\Delta^{n}_{h}[e^{-c_{N}z}z^{N}](z)$
$\displaystyle=e^{-c_{N}z}\sum_{b\geq 0}{N\choose
b}z^{N-b}O(n^{b}(e^{-c_{N}h}-1)^{n})$
$\displaystyle=O\left((e^{-c_{N}h}-1)^{n}\right)\sum_{b\geq 0}{N\choose
b}z^{N-b}O(n^{b})$
$\displaystyle=O\left((e^{-c_{N}h}-1)^{n}\right)O\left(n^{N}\right)=O\left(n^{N}(e^{-c_{N}h}-1)^{n}\right).$

As $|e^{-c_{N}h}-1|<1$, it follows that
$\lim_{n\to\infty}\Delta^{n}_{h}[e^{-c_{N}z}z^{N}](z)=0$. As $p$ is a finite
sum of terms which each converge to zero as $n\to\infty$, we have that
$\lim_{n\to\infty}\Delta^{n}_{h}[p](z)=0$.

We now argue that $\Delta^{n}_{h}[\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0
-.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m](z)$
converges to zero. As $z\in\mathcal{C}_{L}(\mbox{\hskip 6.25002pt\hskip
1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m)$, by
Lemma 1 we have that

$\displaystyle\Delta_{h}^{n}[\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45
1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m](z)=\int_{0}^{\infty}e^{-zt}(e^{-ht}-1)^{n}m(t)dt.$

Note $|e^{-ht}-1|<1$ for all all $t>-h^{-1}\log 2$, so $(e^{-ht}-1)^{n}$
converges pointwise to $0$ for all $t\geq 0$. Let
$j_{n}(t)\vcentcolon=(e^{-ht}-1)^{n}e^{-zt}m(t)$. We have that

$\displaystyle|j_{n}(t)|\leq|e^{-zt}m(t)|$

for all $t\in\mathbb{R}^{+}$ and $n\in\mathbb{N}$. Note that since $m$ is
Lebesgue measurable, $e^{-zt}m(t)$ and each $j_{n}$ are Lebesgue measurable.
As $z\in\mathcal{A}_{L}(f)$ and $z\in\mathbb{C}=\mathcal{A}_{L}(p)$, we have
that $z\in\mathcal{A}_{L}(\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45
1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m)$. So
$e^{-zt}m(t)$ is absolutely integrable on $\mathbb{R}^{+}$. As $j_{n}(t)$
converges pointwise to $0$ for all $t\geq 0$, by the dominated convergence
theorem we have that

$\displaystyle\lim_{n\to\infty}\Delta_{h}^{n}[\mbox{\hskip 6.25002pt\hskip
1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m](z)=\lim_{n\to\infty}\int_{0}^{\infty}j_{n}(t)dt=0.$

Thus,

$\displaystyle\lim_{n\to\infty}\Delta_{h}^{n}[f](z)=\lim_{n\to\infty}\Delta_{h}^{n}[\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m](z)+\lim_{n\to\infty}\Delta_{h}^{n}[p](z)=0,$

as desired. ∎

Note that the assumption that $z\in\mathcal{A}_{L}(f)$ is necessary;
$z\in\mathcal{C}_{L}(f)$ is not enough to guarantee convergence to zero.
Indeed, consider the $n$th order forward differences of
$f(z)=(1+z^{2})^{-1/2}$. By [laplacebessel, Appendix B], we have that
$\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}^{-1}f(t)=J_{0}(t)$,
where $J_{0}$ is the zeroth order Bessel function of the first kind, and
$0\in\mathcal{C}_{L}(f)$ since $J_{0}$ is integrable on $\mathbb{R}^{+}$ per
[NIST:DLMF, Equation 10.22.41]. However, [flajoletsedgewick, pg.119] showed
that for some $c_{1},c_{2}\in\mathbb{R}$,

$\displaystyle(-1)^{n}\Delta^{n}[f](0)=\sum_{k\geq 0}{n\choose
k}\frac{(-1)^{k}}{\sqrt{k^{2}+1}}=1+c_{1}\sqrt{\log n}\cos(\log
n+c_{2})+O((\log n)^{-1/2}),$

which clearly does not converge to zero.

######  Corollary 1.

Assuming it exists, suppose that $\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0
-.45 1}\hbox
<EMAIL_ADDRESS>is a linear combination of Lebesgue measurable functions and elements of the
set $\left\\{\delta^{(m)}(x-a):a\in\mathbb{R}^{\geq
0},m\in\mathbb{N}\right\\}$. Let $z\in\mathbb{C}$ such that
$-z\in\mathcal{A}_{L}(g)$. Let $h\in\mathbb{R}^{+}$. Let $f(t)=g(-t)$. Then

$\displaystyle\lim_{n\to\infty}\nabla_{h}^{n}[f](z)=0.$

###### Proof.

This result follows immediately from Theorem 1 by noting that

$\displaystyle\nabla^{n}_{h}[f](z)=(-1)^{n}\Delta^{n}_{h}[g](-z).$

∎

######  Definition 3 (Newton series).

For a suitable function $f$, the Newton series of $f$ centered at
$z_{0}\in\mathbb{C}$ is given by

$\displaystyle f(z)=\sum_{k\geq 0}{z-z_{0}\choose k}\Delta^{k}[f](z_{0}).$

The second application of Lemma 1 allows us to guarantee the existence and
convergence of certain Newton series, which can be a delicate task otherwise.

######  Theorem 2.

Assuming it exists, suppose that $\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0
-.45 1}\hbox
<EMAIL_ADDRESS>is a linear combination of Lebesgue measurable functions and elements of the
set $\left\\{\delta^{(m)}(x-a):m\in\mathbb{N},a\in[0,\log 2)\right\\}$. Let
$z_{0}\in\mathcal{A}_{L}(f)$. Then the Newton series of $f$ centered at
$z_{0}$ exists and converges in the half-plane $\Re(z)>\Re(z_{0})$.

###### Proof.

Let $z_{0}\in\mathcal{A}_{L}(f)$. Let $z\in\mathbb{C}:\Re(z)>\Re(z_{0})$.
Suppose that $\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
<EMAIL_ADDRESS>is a linear combination of Lebesgue measurable functions and elements of the
set
$\left\\{\delta^{(m)}(x-a):m\in\mathbb{N},a\in[0,\log 2)\right\\}$. So,
without loss of generality, we have that

$\displaystyle\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}^{-1}f(t)=m(t)+\sum_{k=0}^{j}a_{k}\delta^{(k)}(t-b_{k}),$

where $j\in\mathbb{N}$, $(a_{k})\subseteq\mathbb{C}$, $(b_{k})\subseteq[0,\log
2)$, and $m$ is some Lebesgue measurable function. We know that $\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
<EMAIL_ADDRESS>exists at $z_{0}$ since $m$ is a difference of functions with a Laplace
transform existing at $z_{0}$. So

$\displaystyle f(z_{0})=\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45
1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m(z_{0})+\sum_{k=0}^{j}a_{k}e^{-b_{k}z_{0}}z_{0}^{k}.$

Let $p(z)=\sum_{k=0}^{j}a_{k}e^{-b_{k}z}z^{k}$. Then

$\displaystyle\Delta^{k}[f](z_{0})=\Delta^{k}[\mbox{\hskip 6.25002pt\hskip
1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m](z_{0})+\Delta^{k}[p](z_{0}).$

So

$\displaystyle\sum_{k\geq 0}{z-z_{0}\choose k}\Delta^{k}[f](z_{0})=\sum_{k\geq
0}{z-z_{0}\choose k}\Delta^{k}[\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45
1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m](z_{0})+\sum_{k\geq
0}{z-z_{0}\choose k}\Delta^{k}[p](z_{0}).$

Let $g(z)=p(z+z_{0})$. Note that $g$ is entire and is of exponential type less
than $\log 2$ since each $|b_{k}|<\log 2$. Thus by [entireandexptype, pg.127],
$g(z)=\sum_{k\geq 0}{z\choose k}\Delta^{k}[g](0)=\sum_{k\geq 0}{z\choose
k}\Delta^{k}[p](z_{0})$ for all $z\in\mathbb{C}$. So
$p(z)=g(z-z_{0})=\sum_{k\geq 0}{z-z_{0}\choose k}\Delta^{k}[p](z_{0})$. Thus,

$\displaystyle\sum_{k\geq 0}{z-z_{0}\choose
k}\Delta^{k}[f](z_{0})=p(z)+\sum_{k\geq 0}{z-z_{0}\choose
k}\Delta^{k}[\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m](z_{0}).$

We now argue that $\sum_{k\geq 0}{z-z_{0}\choose k}\Delta^{k}[\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m](z_{0})=\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m(z)$.
Since $z_{0}\in\mathcal{C}_{L}(\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45
1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m)$, we
have by Lemma 1 that

$\displaystyle\sum_{k\geq 0}{z-z_{0}\choose k}\Delta^{k}[\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m](z_{0})=\sum_{k\geq
0}{z-z_{0}\choose k}\int_{0}^{\infty}(e^{-t}-1)^{k}e^{-z_{0}t}m(t)dt.$

We now justify the interchange of the sum and integral. As $m$ is Lebesgue
measurable, it is sufficient to show that

$\displaystyle\int_{0}^{\infty}\left|e^{-z_{0}t}m(t)\right|\sum_{k\geq
0}\left|{z-z_{0}\choose k}(e^{-t}-1)^{k}\right|dt<\infty.$

For all $t\geq 0$, $|e^{-t}-1|<1$; thus,

$\displaystyle\sum_{k\geq 0}\left|{z-z_{0}\choose
k}(e^{-t}-1)^{k}\right|\leq\sum_{k\geq 0}\left|{z-z_{0}\choose
k}\right|=\vcentcolon T(z,z_{0}).$

Thus,

$\displaystyle\int_{0}^{\infty}\left|e^{-z_{0}t}m(t)\right|\sum_{k\geq
0}\left|{z-z_{0}\choose
k}(e^{-t}-1)^{k}\right|dt\leq\int_{0}^{\infty}T(z,z_{0})\left|e^{-z_{0}t}m(t)\right|dt.$

The series $T(z,z_{0})$ is known to converge whenever $\Re(z)>\Re(z_{0})$; see
[binomialseries]. As $z_{0}\in\mathcal{A}_{L}(f)$ and
$z_{0}\in\mathbb{C}=\mathcal{A}_{L}(p)$, we have that
$z_{0}\in\mathcal{A}_{L}(\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45
1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m)$.
So, as $T(z,z_{0})$ is constant with respect to $t$,

$\displaystyle\int_{0}^{\infty}T(z,z_{0})\left|e^{-z_{0}t}m(t)\right|dt<\infty.$

So

$\displaystyle\int_{0}^{\infty}\left|e^{-z_{0}t}m(t)\right|\sum_{k\geq
0}\left|{z-z_{0}\choose k}(e^{-t}-1)^{k}\right|dt<\infty.$

Thus, we can interchange the sum and integral; consequently,

$\displaystyle\sum_{k\geq 0}{z-z_{0}\choose
k}\int_{0}^{\infty}(e^{-t}-1)^{k}e^{-z_{0}t}m(t)dt$
$\displaystyle=\int_{0}^{\infty}e^{-z_{0}t}m(t)\sum_{k\geq 0}{z-z_{0}\choose
k}(e^{-t}-1)^{k}dt$
$\displaystyle=\int_{0}^{\infty}e^{-z_{0}t}m(t)\left((e^{-t}-1)+1\right)^{z-z_{0}}dt$
$\displaystyle=\int_{0}^{\infty}e^{-zt}m(t)dt.$

Since $z_{0}\in\mathcal{A}_{L}(\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45
1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m)\subseteq\mathcal{C}_{L}(\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m)$ and
$\Re(z)>\Re(z_{0})$,

$\displaystyle\int_{0}^{\infty}e^{-zt}m(t)dt=\mbox{\hskip 6.25002pt\hskip
1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m(z).$

So

$\displaystyle\sum_{k\geq 0}{z-z_{0}\choose k}\Delta^{k}[\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m](z_{0})=\mbox{\hskip
6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m(z).$

Thus,

$\displaystyle\sum_{k\geq 0}{z-z_{0}\choose
k}\Delta^{k}[f](z_{0})=p(z)+\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45
1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}m(z)=f(z),$

as desired. ∎

######  Corollary 2.

Let $R$ be a rational function with real coefficients. Let $S$ be the set of
poles of $R$. Let $z_{0}\in\mathbb{C}$ such that
$\Re(z_{0})>\max\\{0,\Re(\sigma):\sigma\in S\\}$. Then the Newton series of
$R$ centered at $z_{0}$ exists and converges in the half-plane
$\Re(z)>\Re(z_{0})$.

###### Proof.

Let $R=P/Q$ be a rational function with real coefficients. Let $S$ be the set
of poles of $R$. Let $z_{0}\in\mathbb{C}$ such that
$\Re(z_{0})>\max\\{0,\Re(\sigma):\sigma\in S\\}$. Let us decompose $R$ as

$\displaystyle R(z)=p(z)+\sum_{k}\frac{P_{k}(z)}{Q_{k}(z)},$

where $p$ is a polynomial, and each $P_{k}/Q_{k}$ is a proper333A rational
function $p/q$ is said to be _proper_ if $\deg(p)<\deg(q)$. rational function.
By this decomposition, it is sufficient to show that the Newton series of $p$
and each $P_{k}/Q_{k}$ exist and converge for all
$z\in\mathbb{C}:\Re(z)>\Re(z_{0})$.

As $p$ is a polynomial, $p(z)=\sum_{k}a_{k}z^{k}$ for some sequence $(a_{k})$.
So $\mbox{\hskip 6.25002pt\hskip 1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}^{-1}p(t)=\sum_{k}a_{k}\delta^{(k)}(t)$.
As $z_{0}\in\mathbb{C}=\mathcal{A}_{L}(p)$, it follows by Theorem 2 that the
Newton series of $p$ centered at $z_{0}$ exists and converges for all
$z\in\mathbb{C}:\Re(z)>\Re(z_{0})$.444In fact, a Newton series of any
polynomial centered anywhere exists and converges in all of $\mathbb{C}$. This
follows from [entireandexptype, pg.127].

As each $P_{k}/Q_{k}$ is a proper rational function with real coefficients, by
[rationalinvlaplace, Theorem 4.1] each $\mbox{\hskip 6.25002pt\hskip
1.40623pt{1 0 -.45 1}\hbox
to0.0pt{\hss\leavevmode\hbox{\set@color$\mathscr{L}$}}\hskip-1.40623pt}^{-1}(P_{k}/Q_{k})(t)$
is a linear combination of the set

$\displaystyle\left\\{t^{k_{1}}e^{k_{2}t}\sin(k_{3}t),t^{j_{1}}e^{j_{2}t}\cos(j_{3}t)\bigg{|}k_{1},j_{1}\in\mathbb{N},k_{2},j_{2}\leq\max\\{0,\Re(\sigma):\sigma\in
S\\},k_{3},j_{3}\in\mathbb{R}\right\\}.$

Note that all of these functions are Lebesgue measurable. Importantly, this
set allows each $k_{i}$ and $j_{i}$ to be zero, yielding the other terms
(e.g., polynomials and exponentials) listed in [rationalinvlaplace, Theorem
4.1]. Taking
$k_{1},j_{1}\in\mathbb{N},k_{2},j_{2}\leq\max\\{0,\Re(\sigma):\sigma\in S\\}$,
and $k_{3},j_{3}\in\mathbb{R}$ as seen in the above set, we see that since
$k_{2}-\Re(z_{0})<0$,

$\displaystyle\int_{0}^{\infty}|t^{k_{1}}e^{k_{2}t}\sin(k_{3}t)e^{-z_{0}t}|dt\leq\int_{0}^{\infty}t^{k_{1}}e^{\left(k_{2}-\Re(z_{0})\right)t}dt<\infty.$

The case with cosine is identical; thus, by linearity,
$z_{0}\in\mathcal{A}_{L}(P_{k}/Q_{k})$ for each $k$. Thus by Theorem 2, the
Newton series of each $P_{k}/Q_{k}$ centered at $z_{0}$ exists and converges
for all $z\in\mathbb{C}:\Re(z)>\Re(z_{0})$. Thus, the Newton series of $R$
centered at $z_{0}$ exists and converges for all
$z\in\mathbb{C}:\Re(z)>\Re(z_{0})$. ∎

The region of convergence claimed in Corollary 2 may be extended if $R$ has a
linear denominator; see [entireandexptype, Theorem 1].

##  3\. Analysis of Fourier transform integral formulas

We now use Fourier transform integral formulas to yield results about the
asymptotic behavior of finite differences (and eventually general binomial
sums).

######  Definition 4.

In this article, we use the following convention for the definition of the
Fourier transform of a function $f$ with suitable conditions:

$\displaystyle\mathcal{F}f(z)=\int_{\mathbb{R}}f(x)e^{-2\pi izx}dx.$

######  Lemma 2.

Suppose $f$ has an inverse Fourier transform. Let $n\in\mathbb{N}$,
$h\in\mathbb{R}^{+}$, and $z\in\mathbb{C}$. Then

$\displaystyle\delta^{n}_{h}[f](z)$ $\displaystyle=\int_{\mathbb{R}}(1-e^{2\pi
ihx})^{n}e^{-i\pi hxn}e^{-2\pi izx}\mathcal{F}^{-1}f(x)dx,$
$\displaystyle\Delta^{n}_{h}[f](z)$ $\displaystyle=\int_{\mathbb{R}}(e^{-2\pi
ihx}-1)^{n}e^{-2\pi izx}\mathcal{F}^{-1}f(x)dx.$

###### Proof.

Let $n\in\mathbb{N}$ and $h\in\mathbb{R}^{+}$. Let $f\in\mathcal{F}$. Let
$g=\mathcal{F}^{-1}f$. Let $z\in\mathbb{C}$. Then we have that

$\displaystyle\delta_{h}^{n}[\mathcal{F}g](z)$
$\displaystyle=\sum_{k=0}^{n}{n\choose
k}\mathcal{F}g\left(z+\left(\frac{n}{2}-k\right)h\right)(-1)^{k}$
$\displaystyle=\sum_{k=0}^{n}{n\choose
k}(-1)^{k}\int_{\mathbb{R}}g(x)\exp\left({-2\pi
i\left(z+\left(\frac{n}{2}-k\right)h\right)x}\right)dx$
$\displaystyle=\int_{\mathbb{R}}g(x)e^{-i\pi hxn}e^{-2\pi
izx}\sum_{k=0}^{n}{n\choose k}(-1)^{k}e^{2\pi ihxk}dx$
$\displaystyle=\int_{\mathbb{R}}g(x)e^{-i\pi hxn}e^{-2\pi izx}(1-e^{2\pi
ihx})^{n}dx.$

Thus,

$\displaystyle\delta^{n}_{h}[f](z)=\int_{\mathbb{R}}(1-e^{2\pi
ihx})^{n}e^{-i\pi hxn}e^{-2\pi izx}\mathcal{F}^{-1}f(x)dx.$

We also have that

$\displaystyle\Delta_{h}^{n}[\mathcal{F}g](z)$
$\displaystyle=\sum_{k=0}^{n}{n\choose
k}\mathcal{F}g\left(z+kh\right)(-1)^{n-k}$
$\displaystyle=\sum_{k=0}^{n}{n\choose
k}(-1)^{n-k}\int_{\mathbb{R}}g(x)\exp\left({-2\pi
i\left(z+kh\right)x}\right)dx$ $\displaystyle=\int_{\mathbb{R}}g(x)e^{-2\pi
izx}\sum_{k=0}^{n}{n\choose k}(-1)^{n-k}e^{-2\pi ihxk}dx$
$\displaystyle=\int_{\mathbb{R}}g(x)e^{-2\pi izx}(e^{-2\pi ihx}-1)^{n}dx.$

Thus,

$\displaystyle\Delta^{n}_{h}[f](z)=\int_{\mathbb{R}}(e^{-2\pi
ihx}-1)^{n}e^{-2\pi izx}\mathcal{F}^{-1}f(x)dx,$

as desired. ∎

######  Definition 5.

Let $a,b\in\mathbb{C}$. We define

$\displaystyle a+b\mathbb{Z}\vcentcolon=\\{a+bn:n\in\mathbb{Z}\\}.$

An application of Lemma 2 gives the following theorem:

######  Theorem 3.

Let $h\in\mathbb{R}^{+}$. Assuming it exists, suppose that $\mathcal{F}^{-1}f$
is a linear combination of Lebesgue measurable functions and elements of the
set

$\\{\delta^{(m)}(x-a):m\in\mathbb{N},a\in\mathbb{R}\backslash(h^{-1}/2+h^{-1}\mathbb{Z})\\}.$

Let $y\in\mathbb{R}$. Then

$\displaystyle\Delta_{h}^{n}[f](y)=o(2^{n}),$
$\displaystyle\nabla_{h}^{n}[f](y)=o(2^{n}),$
$\displaystyle\delta_{h}^{n}[f](y)=o(2^{n}).$

###### Proof.

Let $y\in\mathbb{R}$. Let $h\in\mathbb{R}^{+}$. Suppose that
$\mathcal{F}^{-1}f$ is a linear combination of Lebesgue measurable functions
and elements of the set

$\\{\delta^{(m)}(x-a):m\in\mathbb{N},a\in\mathbb{R}\backslash(h^{-1}/2+h^{-1}\mathbb{Z})\\}.$

So, without loss of generality, we have that

$\displaystyle\mathcal{F}^{-1}f(x)=m(x)+\sum_{k=0}^{j}a_{k}\delta^{(k)}(x-c_{k}),$

where $j\in\mathbb{N}$, $(a_{k})\subseteq\mathbb{C}$,
$(c_{k})\subseteq\mathbb{R}\backslash(h^{-1}/2+h^{-1}\mathbb{Z})$, and $m$ is
some Lebesgue measurable function. So

$\displaystyle f(y)=\mathcal{F}m(y)+\sum_{k=0}^{j}a_{k}e^{-2\pi ic_{k}y}(2\pi
iy)^{k}.$

Let $p(y)=\sum_{k=0}^{j}a_{k}e^{-2\pi ic_{k}y}(2\pi iy)^{k}$. First, we
address the claim for central differences. Let $N\in\\{0,1,\cdots,j\\}$. We
now argue that $\delta^{n}_{h}[e^{-2\pi ic_{N}y}(2\pi iy)^{N}](y)$ is
$o(2^{n})$. We have that

$\displaystyle\delta^{n}_{h}[e^{-2\pi ic_{N}y}(2\pi iy)^{N}](y)=\sum_{k\geq
0}{n\choose k}(-1)^{k}e^{-2\pi ic_{N}(y+(n/2-k)h)}(2\pi i(y+(n/2-k)h))^{N}$
$\displaystyle=e^{-2\pi ic_{N}(y+nh/2)}\sum_{b\geq 0}{N\choose b}(2\pi
i(y+nh/2))^{N-b}\sum_{k\geq 0}{n\choose k}(-1)^{k}e^{2\pi ic_{N}kh}(-2\pi
ikh)^{b}.$

Note

$\displaystyle\sum_{k\geq 0}{n\choose k}(-1)^{k}e^{2\pi ixkh}(-2\pi
ikh)^{b}=(-1)^{b}\frac{d^{b}}{dx^{b}}\left(1-e^{2\pi
ixh}\right)^{n}=O(n^{b}(e^{2\pi ixh}-1)^{n}).$

Thus by taking $x=c_{N}$,

$\displaystyle\delta^{n}_{h}[e^{-2\pi ic_{N}y}(2\pi iy)^{N}](y)=e^{-2\pi
ic_{N}(y+nh/2)}\sum_{b\geq 0}{N\choose b}(2\pi i(y+nh/2))^{N-b}O(n^{b}(e^{2\pi
ic_{N}h}-1)^{n})$ $\displaystyle=e^{-2\pi ic_{N}(y+nh/2)}O((e^{2\pi
ic_{N}h}-1)^{n})\sum_{b\geq 0}{N\choose b}(2\pi i(y+nh/2))^{N-b}O(n^{b})$
$\displaystyle=e^{-\pi ic_{N}nh}O((e^{2\pi ic_{N}h}-1)^{n})O(n^{N})=e^{-\pi
ic_{N}nh}O(n^{N}(e^{2\pi ic_{N}h}-1)^{n}).$

As $c_{N}\not\in h^{-1}/2+h^{-1}\mathbb{Z}$, it follows that $|e^{2\pi
ic_{N}h}-1|<2$; thus, $\Delta^{n}_{h}[e^{-2\pi ic_{N}y}(2\pi
iy)^{N}](y)=o(2^{n})$. So $\delta^{n}_{h}[p](y)=o(2^{n})$.

We now argue that $\delta^{n}_{h}[\mathcal{F}m](y)=o(2^{n}).$ By Lemma 2, we
have that

$\displaystyle\delta^{n}_{h}[\mathcal{F}m](y)$
$\displaystyle=\int_{\mathbb{R}}(1-e^{2\pi ihx})^{n}e^{-i\pi hxn}e^{-2\pi
iyx}m(x)dx$ $\displaystyle=\int_{\mathbb{R}}(e^{-i\pi hx}-e^{i\pi
hx})^{n}e^{-2\pi iyx}m(x)dx$ $\displaystyle=\int_{\mathbb{R}}\left(-2i\sin(\pi
hx)\right)^{n}e^{-2\pi iyx}m(x)dx$
$\displaystyle=(-2i)^{n}\int_{\mathbb{R}}\sin^{n}(\pi hx)e^{-2\pi iyx}m(x)dx.$

Note

$\displaystyle|\sin(\pi
hx)|\in\begin{cases}\\{1\\}&x=\frac{2k+1}{2h}~{}\text{for
some~{}}k\in\mathbb{Z}\\\ [0,1)&\text{otherwise.}\\\ \end{cases}$

So $\left|\sin^{n}(\pi hx)\right|$ converges pointwise to

$\displaystyle s(x)\vcentcolon=\begin{cases}1&x=\frac{2k+1}{2h}~{}\text{for
some~{}}k\in\mathbb{Z}\\\ 0&\text{otherwise.}\\\ \end{cases}$

Let $j_{n}(x)\vcentcolon=\left|\sin^{n}(\pi hx)e^{-2\pi iyx}m(x)\right|$. We
have that

$\displaystyle|j_{n}(x)|\leq\left|e^{-2\pi iyx}m(x)\right|=\left|m(x)\right|$

for all $x\in\mathbb{R}$ and $n\in\mathbb{N}$. Note that since $m$ has Fourier
transform $\mathcal{F}m$, $m$ is absolutely integrable. By assumption, $m$ is
Lebesgue measurable, so each $j_{n}$ is Lebesgue measurable. As $j_{n}(x)$
converges pointwise to $s(x)\left|e^{-2\pi
iyx}m(x)\right|=s(x)\left|m(x)\right|$, by the dominated convergence theorem
we have that

$\displaystyle\lim_{n\to\infty}\int_{\mathbb{R}}j_{n}(x)dx=\int_{\mathbb{R}}s(x)\left|m(x)\right|dx.$

As $s=0$ almost everywhere,

$\displaystyle\int_{\mathbb{R}}s(x)\left|m(x)\right|dx=0.$

Thus,

$\displaystyle\lim_{n\to\infty}\left|\frac{\delta^{n}_{h}[\mathcal{F}m](y)}{2^{n}}\right|=\lim_{n\to\infty}\left|\int_{\mathbb{R}}\sin^{n}(\pi
hx)e^{-2\pi
iyx}m(x)dx\right|\leq\lim_{n\to\infty}\int_{\mathbb{R}}j_{n}(x)dx=0.$

Therefore

$\displaystyle\lim_{n\to\infty}\frac{\delta^{n}_{h}[\mathcal{F}m](y)}{2^{n}}=0.$

Thus,

$\displaystyle\delta_{h}^{n}[\mathcal{F}m](y)=o(2^{n}).$

Thus,

$\displaystyle\delta_{h}^{n}[f](y)=\delta_{h}^{n}[\mathcal{F}m](y)+\delta_{h}^{n}[p](y)=o(2^{n})+o(2^{n})=o(2^{n}).$

Now, we address the claim for forward differences. We now argue that
$\Delta^{n}_{h}[e^{-2\pi ic_{N}y}(2\pi iy)^{N}](y)$ is $o(2^{n})$. We have
that

$\displaystyle\Delta^{n}_{h}[e^{-2\pi ic_{N}y}(2\pi iy)^{N}](y)=\sum_{k\geq
0}{n\choose k}(-1)^{n-k}e^{-2\pi ic_{N}(y+kh)}(2\pi i(y+kh))^{N}$
$\displaystyle=e^{-2\pi ic_{N}y}\sum_{b\geq 0}{N\choose b}(2\pi
iy)^{N-b}\sum_{k\geq 0}{n\choose k}(-1)^{n-k}e^{-2\pi ic_{N}kh}(2\pi
ikh)^{b}.$

Note

$\displaystyle\sum_{k\geq 0}{n\choose k}(-1)^{n-k}e^{-2\pi ixkh}(2\pi
ikh)^{b}=(-1)^{b}\frac{d^{b}}{dx^{b}}\left(e^{-2\pi
ixh}-1\right)^{n}=O(n^{b}(e^{-2\pi ixh}-1)^{n}).$

Thus by taking $x=c_{N}$,

$\displaystyle\Delta^{n}_{h}[e^{-2\pi ic_{N}y}(2\pi iy)^{N}](y)$
$\displaystyle=e^{-2\pi ic_{N}y}\sum_{b\geq 0}{N\choose b}(2\pi
iy)^{N-b}O(n^{b}(e^{-2\pi ic_{N}h}-1)^{n})$ $\displaystyle=O((e^{-2\pi
ic_{N}h}-1)^{n})\sum_{b\geq 0}{N\choose b}(2\pi iy)^{N-b}O(n^{b})$
$\displaystyle=O((e^{-2\pi ic_{N}h}-1)^{n})O(n^{N})=O(n^{N}(e^{-2\pi
ic_{N}h}-1)^{n}).$

As $c_{N}\not\in h^{-1}/2+h^{-1}\mathbb{Z}$, it follows that $|e^{-2\pi
ic_{N}h}-1|<2$; thus, $\Delta^{n}_{h}[e^{-2\pi ic_{N}y}(2\pi
iy)^{N}](y)=o(2^{n})$. As $p$ is a finite sum of terms which are $o(2^{n})$,
we have that $\Delta^{n}_{h}[p](y)=o(2^{n})$.

We now argue that $\Delta^{n}_{h}[\mathcal{F}m](y)=o(2^{n})$. By Lemma 2, we
have that

$\displaystyle\lim_{n\to\infty}\left|\frac{\Delta^{n}_{h}[\mathcal{F}m](y)}{2^{n}}\right|$
$\displaystyle=\lim_{n\to\infty}\left|\int_{\mathbb{R}}\left(\frac{e^{-2\pi
ihx}-1}{2}\right)^{n}e^{-2\pi iyx}m(x)dx\right|$
$\displaystyle\leq\lim_{n\to\infty}\int_{\mathbb{R}}\left|\left(\frac{e^{-2\pi
ihx}-1}{2}\right)^{n}e^{-2\pi iyx}m(x)\right|dx.$

Let

$\ell_{n}(x)\vcentcolon=\left(\frac{e^{-2\pi ihx}-1}{2}\right)^{n}.$

Note for each $n\in\mathbb{N}$, $\ell_{n}(\mathbb{R})$ is contained in the
closed unit disk, intersecting the unit circle only at $z=-1$. Specifically,
for all $n\in\mathbb{N}$,

$\displaystyle\ell^{-1}_{n}\left(\\{-1\\}\right)=\left\\{x\in\mathbb{R}:x=\frac{2k+1}{2h}~{}\text{for
some~{}}k\in\mathbb{Z}\right\\}.$

So $|\ell_{n}(x)|$ converges pointwise to

$\displaystyle s(x)\vcentcolon=\begin{cases}1&x=\frac{2k+1}{2h}~{}\text{for
some~{}}k\in\mathbb{Z}\\\ 0&\text{otherwise.}\\\ \end{cases}$

The remaining argument follows identically to the argument given that
$\delta^{n}_{h}[\mathcal{F}m](y)=o(2^{n})$. So

$\displaystyle\lim_{n\to\infty}\left|\frac{\Delta^{n}_{h}[\mathcal{F}m](y)}{2^{n}}\right|\leq\lim_{n\to\infty}\int_{\mathbb{R}}\left|\left(\frac{e^{-2\pi
ihx}-1}{2}\right)^{n}e^{-2\pi iyx}m(x)\right|dx=0.$

Therefore

$\displaystyle\lim_{n\to\infty}\frac{\Delta^{n}_{h}[\mathcal{F}m](y)}{2^{n}}=0.$

Thus,

$\displaystyle\Delta_{h}^{n}[\mathcal{F}m](y)=o(2^{n}).$

Thus,

$\displaystyle\Delta_{h}^{n}[f](y)=\Delta_{h}^{n}[\mathcal{F}m](y)+\Delta_{h}^{n}[p](y)=o(2^{n})+o(2^{n})=o(2^{n}).$

As we have proven the claim for forward differences, the claim for backward
differences follows by the relationship

$\displaystyle\nabla_{h}^{n}[f](y)=(-1)^{n}\Delta^{n}_{h}[g](-y),$

where $g(y)=f(-y)$. So

$\displaystyle\nabla_{h}^{n}[f](y)=o(2^{n}),$

completing the proof. ∎

An application of Theorem 3 yields the following result:

######  Theorem 4.

Let $h\in\mathbb{R}^{+}$. Assuming it exists, suppose $\mathcal{F}^{-1}f$ is a
linear combination of Lebesgue measurable functions and elements of the set

$\\{\delta^{(m)}(x-a):m\in\mathbb{N},a\in\mathbb{R}\backslash(h^{-1}\mathbb{Z})\\}.$

Let $y\in\mathbb{R}$. Then

$\displaystyle\sum_{k=0}^{n}{n\choose k}f(y+kh)=o(2^{n}),$
$\displaystyle\sum_{k=0}^{n}{n\choose k}f(y-kh)=o(2^{n}),$
$\displaystyle\sum_{k=0}^{n}{n\choose
k}f\left(y+\left(\frac{n}{2}-k\right)h\right)=o(2^{n}).$

###### Proof.

Let $y\in\mathbb{R}$. Let $h\in\mathbb{R}^{+}$. Suppose that
$\mathcal{F}^{-1}f$ is a linear combination of Lebesgue measurable functions
and elements of the set

$\\{\delta^{(m)}(x-a):m\in\mathbb{N},a\in\mathbb{R}\backslash(h^{-1}\mathbb{Z})\\}.$

Let $\alpha(y)=e^{i\pi y/h}f(y)$. Note the relationships

$\displaystyle\Delta^{n}_{h}[\alpha](y)=(-1)^{n}e^{i\pi
y/h}\sum_{k=0}^{n}{n\choose k}f(y+kh),$
$\displaystyle\nabla^{n}_{h}[\alpha](y)=e^{i\pi y/h}\sum_{k=0}^{n}{n\choose
k}f(y-kh),$ $\displaystyle\delta^{n}_{h}[\alpha](y)=e^{i\pi y/h}e^{i\pi
n/2}\sum_{k=0}^{n}{n\choose k}f\left(y+\left(\frac{n}{2}-k\right)h\right).$

We have that $\mathcal{F}^{-1}\alpha(x)=\mathcal{F}^{-1}f(x+h^{-1}/2)$. As
translations of Lebesgue measurable functions are Lebesgue measurable,
$\mathcal{F}^{-1}\alpha$ is a linear combination of Lebesgue measurable
functions and elements of the set
$\\{\delta^{(m)}(x-a):m\in\mathbb{N},a\in\mathbb{R}\backslash(h^{-1}/2+h^{-1}\mathbb{Z})\\}$.
Thus by Theorem 3,

$\displaystyle\Delta_{h}^{n}[\alpha](y)=o(2^{n}),$
$\displaystyle\nabla_{h}^{n}[\alpha](y)=o(2^{n}),$
$\displaystyle\delta_{h}^{n}[\alpha](y)=o(2^{n}).$

Thus,

$\displaystyle\sum_{k=0}^{n}{n\choose k}f(y+kh)=o(2^{n}),$
$\displaystyle\sum_{k=0}^{n}{n\choose k}f(y-kh)=o(2^{n}),$
$\displaystyle\sum_{k=0}^{n}{n\choose
k}f\left(y+\left(\frac{n}{2}-k\right)h\right)=o(2^{n}),$

as desired. ∎

Note that the conclusion of the above theorem cannot be improved to
$o(\alpha^{n})$ for $\alpha<2$. Indeed, take $f(y)=(y+1)^{-1}$. The inverse
Fourier transform of $f$ exists and is Lebesgue measurable; however,

$\displaystyle\sum_{k\geq 0}{n\choose k}\frac{1}{k+1}=\frac{2^{n+1}-1}{n+1},$

which is not $o(\alpha^{n})$ if $\alpha<2$. This identity may be verified by
observing that

$\displaystyle\sum_{k\geq 0}{n\choose k}\frac{x^{k+1}}{k+1}=\sum_{k\geq
0}{n\choose
k}\int_{0}^{x}t^{k}dt=\int_{0}^{x}(t+1)^{n}dt=\frac{(x+1)^{n+1}-1}{n+1},$

where we take $x=1$.

Unfortunately, we cannot allow all translations of the Dirac delta function
and its derivatives in the inverse Fourier transform of $f$ for Theorem 4.
This is easy to see: take $f(y)=1$, wherein $\mathcal{F}^{-1}f(x)=\delta(x)$
and

$\displaystyle\sum_{k\geq 0}{n\choose k}f(k)=\sum_{k\geq 0}{n\choose
k}=2^{n}\neq o(2^{n}).$

Theorem 6 fails to apply to the above sum since $0\in\mathbb{Z}$.

##  4\. Acknowledgements

Thanks to Sai Sivakumar for directing me to results relevant to the topics of
this article and for his excellent mentorship. Thanks to Kenneth DeMason for a
thorough review of the paper.

## References


* Le Tiec  _et al._ [2013] A. Le Tiec, A. Buonanno, A. H. Mroué, H. P. Pfeiffer, D. A. Hemberger, G. Lovelace, L. E. Kidder, M. A. Scheel, B. Szilágyi, N. W. Taylor, and S. A. Teukolsky, Periastron Advance in Spinning Black Hole Binaries: Gravitational Self-Force from Numerical Relativity, Phys. Rev. D 88, 124027 (2013), arXiv:1309.0541 [gr-qc] .