# A Simple Attention-Based Mechanism for Bimodal Emotion Classification

Mazen Elabd, Sardar Jaf
University of Sunderland
Sunderland
{mazen.elabd<EMAIL_ADDRESS>

###### Abstract

Big data contain rich information for machine learning algorithms to utilize
when learning important features during classification tasks. Human beings
express their emotion using certain words, speech (tone, pitch, speed) or
facial expression. Artificial Intelligence approach to emotion classification
are largely based on learning from textual information. However, public
datasets containing text and speech data provide sufficient resources to train
machine learning algorithms for the tack of emotion classification. In this
paper, we present novel bimodal deep learning-based architectures enhanced
with attention mechanism trained and tested on text and speech data for
emotion classification. We report details of different deep learning based
architectures and show the performance of each architecture including rigorous
error analyses. Our finding suggests that deep learning based architectures
trained on different types of data (text and speech) outperform architectures
trained only on text or speech. Our proposed attention-based bimodal
architecture outperforms several state-of-the-art systems in emotion
classification.

##  1 Introduction

Recognizing human emotions automatically from text is a challenging task.
Textual data lacks emotional cues, such as speech tone, pitch, vocal
expression, facial expression, etc., that are helpful in determining the
emotion of a person accurately. Therefore, approaches to automated emotion
recognition that rely only on text are inherently limited. Recent attempts at
emotion recognition are directed at using other types of information such as
audio, image, and/or video as well as text in order to enrich the information
needed to accurately recognize/classify human emotions.

With advances in machine learning applications for various tasks (such as
image processing, computer vision, and natural language processing), it is
possible to design multimodal emotion recognition/classification systems by
training machine learning algorithms on labeled datasets containing different
types of data modalities such as text, images, audio, and/or video. Multimodal
emotion recognition/classification involves training one or more machine
learning algorithms that are best suited for learning from different types of
data. For example, we could train algorithm $x$ on text data, $y$ on audio
data, and $z$ on visual data and then combine their learning capabilities from
these different types of data to perform the classification task. In this
paper, we propose a novel design of deep learning architecture that uses text
and audio information for the task of emotion classification. We make the
following contributions:

  * •

We propose a novel deep-learning multimodal architecture to extract and
utilize important features from different types of data (text and audio) to
classify emotions into one of several emotion classes.

  * •

We propose a new state-of-the-art multimodal emotion classifier.

  * •

We propose mid-level data fusion methods for extracting rich features from
different unimodal architectures.

The rest of the paper is organized as follows: Section 2 outlines key related
work to our proposed system. We describe our methodology in section 3, in
section 4 we show the performance of the proposed system and compare it
against several published works. In section 5 we analyse the system’s
performance through several confusion matrices. We conclude the paper in
section 6.

##  2 Related work

The advances in deep learning methods for audio and text processing have
motivated researchers to develop different approaches to emotion
classification. Generally, these approaches involve training deep learning
algorithms on audio and text data, and incorporating a fusion mechanism to
combine both modalities. Early feature extraction methods from text, such as
word2vec, involve learning information from text in relation to word features.
Further advancements in feature extraction from text include bidirectional
long short-term memory networks (Bi-LSTM), gated recurrent unit (GRU), and
transformers. Furthermore, rich language models containing multi-lingual and
contextual information around words were developed such as Bidirectional
Encoder Representations from Transformers (BERT) [1], Robustly optimized BERT
(RoBERTa) [2], and GPT [3], etc.

Methods to combine several data modalities (e.g., different types of data such
as text, audio, and images) have been effective in emotion classification [4].
Multimodal transformers that allow the concatenation of the feature
representation from different types of data have replaced previous methods [5]
[6], with further advances through multi-view sequential learning models [7]
and dynamic fusion graph [8]

Dutta et al [9] proposed a hierarchical cross-attention model approach using
recurrent and co-attention neural networks models by training them on text and
audio data. Their first stage involved training utterance-level embedding
extractor from the input data (text and audio), which trains their model to
classify individual utterances without accounting for the inter-utterance
conversational context. The second stage of training involves feeding the
utterance-level embedding from the first stage to a bidirectional gated
recurrent unit (Bi-GRU) to introduce inter-utterance context to the model.
This stage enhances the model with conversational context information. The
last stage consists of the fusion of the embedding from different modalities
using self-attention and cross-attention mechanisms. These attention
mechanisms allow for capturing relationships and dependencies within input
sequences (sentences or speech utterances).

[10] proposed a hierarchical transformer-based model (HiTrans) consisting of
transformer-based content and a speaker-sensitive model for emotion
classification. Their method uses two hierarchical transformers: a BERT model
is used as the low-level transformer for generating local utterance
representation, and a high-level transformer that takes the output of the low-
level transformer as input to make the model sensitive to the global context
of the conversation. They integrate a “pairwise utterance speaker
verification” (PUSV) method to detect whether two utterances belong to the
same speaker.

[11] proposed a Dialogue Graph Convolutional Network (DialogueGCN) model
utilizing a graph neural network based approach to emotion classification by
leveraging self and inter-speaker dependency of the interlocutors to model
conversational context for emotion classification. One of the main benefits of
such a method is addressing context propagation issues present in the current
RNN-based methods.

[12] proposed a learning-based system for emotion classification using
multiple input modalities that combined information from text, facial cues,
and speech. Their system seems to pay more attention to reliable cues while
suppressing less helpful cues on a per-sample basis by using Canonical
Correlation Analysis, which differentiates between effective and ineffective
modalities. The major strength of their proposed system is its robustness to
sensor noise in any of the individual modalities.

##  3 Proposed Method

We propose a supervised machine learning approach for the task of emotion
classification. We use a public dataset containing different types of data
(text, speech, and video) for the training and evaluation of a multimodal
emotion classifier. The benefit of using different types of data is to extract
different levels of details and cues for identifying emotions. Different data
types contain different features/information for training machine learning
algorithms. Due to limited access to computational resources, we only use text
and speech data types in this study. We use different deep learning based
feature extraction methods (embedding) for learning patterns from text and
speech data. For text, we use BERT embedding [1] and for speech, we use Audio
Spectogram Transformer (AST) embedding [13].

Figure 1: Baseline multimodal classification architecture.

###  3.1 System Design

Our system design is based on fine-tuning attention-based models as well as
the reliance on attention-based feature fusion. We believe that two main
problems should be addressed when building any multimodal architecture: i) the
data representation for each modality in the system (i.e., data
representation), and ii) the data concatenation from different modalities
(i.e., data fusion).

Our proposed multimodal system for emotion classification relies on the fusion
of extracted features from the data through two unimodal architectures: BERT-
based architecture for text-based emotion classification, and AST-based
architecture for speech-based emotion classification. The last hidden layer
from both BERT and AST models is known to have the most contextualised
representation of the input sequence. Thus, we specifically use the output of
the last hidden layer of fine-tuned versions of BERT and AST to represent the
text and audio modalities, respectively. In our attention-based architecture,
we adjust the size of the output of the last hidden layer from the text model
through a padding layer to match the size of the output of the last hidden
layer for the audio model. Similar to late-fusion-based multimodal systems,
relying on fine-tuning pre-trained transformer-based models gives the
capability of utilising powerful and highly efficient models to represent the
multimodal data with a relatively small dataset size and limited hardware
resources in the training process.

To address the feature fusion problem, we rely on a multi-head cross-attention
layer as a concatenation technique. If we relied on a method such as averaging
or voting, the multimodal system would not benefit from the interactions that
could occur between different modalities.

Instead of merging both modalities at a classifier level, as done in late
feature fusion techniques, we merge them at a data representation layer, which
is similar to early fusion techniques, to be able to capture the interactions
between audio and text. Our proposed system leverages both early and late
fusion techniques. We use powerful pre-trained unimodal models while merging
the data before the classifier layer using a simple concatenation layer (as
shown in Fig. 1 for the baseline system) and cross-attention fusion technique,
as presented in Fig. 2. We believe that cross-attention fusion is capable of
capturing the interactions between different modalities. It may also be able
to make sense of confusing and contradicting data.

Figure 2: Multi-head attention-based multimodal architecture.

To perform the cross-attention fusion, we feed the output from the last hidden
layer of the audio model as the key tensor to the multi-head attention layer
while feeding the padded output from the last hidden layer of the text model
as the query and the value tensors of the multi-head attention layer, as shown
in Fig. 2. The multi-head cross-attention layer is followed by a multi-head
self-attention layer which we added in the pursuit of enriching the combined
audio and text representation. The combined feature vector is fed as the
query, key, and value for the multi-head self-attention layer, as in Fig. 2.
After combining the features from the different data modalities, the combined
feature vector undergoes flattening along the first dimension. Then, the
combined feature vector is passed to a linear layer with a 10% dropout, which
is followed by the classification layer, as in Fig. 1 and 2.

####  3.1.1 Selecting the unimodal models

Since the presented architectures in this work rely on unimodal systems to
create feature representations for each modality, the selection of the
unimodal system is critical to optimise our system’s performance. We believe
that the various unimodal models utilized within a multimodal classification
architecture should rely on similar feature representation algorithms to reach
a homogeneous feature vector when combining them in the system.

We choose attention-based unimodal approaches to build the feature vector for
each modality since they are proven to achieve state-of-the-art results in
many benchmarks and most importantly to maintain homogeneous feature
representations between modalities. Attention-based models are popular in
various unimodal classification domains, such as text, speech, video, and
image, which enable our multimodal systems to easily scale to include another
modality. The selected unimodal systems are mainly based on transformer-
encoder and only rely on attention to keep the models as homogeneous as
possible. In addition, the selected systems are simple and fast to fine-tune.

The last hidden layer from each of the transformers (BERT and AST) contains
the most comprehensive and contextualized representation of the input
sequence; therefore, it is logical to use it as the feature vector for each
modality.

####  3.1.2 Baseline system

For our baseline model, as depicted in Fig. 1, the information derived from
the last hidden layer of BERT and AST is fed into a simple concatenation layer
as the fusion layer. This was followed by a flattening layer, then a linear
layer with dropout (10% dropout rate) as the classification layer.

####  3.1.3 Attention-based system

For this system, we rely on feeding the feature vector from BERT and AST
models to a multi-head cross-attention layer, as in Fig. 2, which acts as a
fusion layer that is capable of capturing the interactions between modalities
and developing a comprehensive understanding of the multimodal data. The
multi-head cross-attention layer is followed by a multi-head self-attention
layer to enrich the combined feature vector before classification. Next, a
flattening layer and a linear layer with a 10% dropout rate serve as the
classification layer.

###  3.2 dataset

We use Multimodal EmotionLines Dataset (MELD) [14] for the evaluation of the
proposed systems. The dataset contains more than over 13000 utterances derived
from multi-party conversations. it is divided into three parts: training,
validation, and testing. The training set consists of 9,988 utterances, the
validation set 1,108 utterances, and the test set 2,610 utterances. We use
these partitions unchanged for training, validation, and testing.

###  3.3 Hyperparameters

The MELD dataset suffers from imbalance class distribution which impacted our
decision to rely on AdamW as an optimizer, the weighted cross-entropy as the
loss function since the dataset suffer from imbalance class distribution
problem, and the weighted F1 score as our main evaluation metric. We
integrated early stopping in all of our experiments to stop the training once
the weighted F1 score started to decrease.

Some hyperparameters were only shared across the multimodal experiments such
as using a 10% dropout rate in the last linear layer. Also, both multi-head
attention layers used in this work were configured to consist of 128 attention
heads111We carried out experiments on fewer than 128 attention heads, 64 heads
and 32 heads, but the obtained results were slightly worse than using 128
attention heads. It could be an indication that a higher number of attention
heads than 128 introduces a model that is more capable of capturing the
interactions between different modalities.. We performed limited experiments
to reduce the number of attention heads, although the weighted F1 score was
negatively impacted.

Despite the fact that we managed to achieve state-of-the-art performance, the
selected hyperparameters still might not be perfect since we were only aiming
to get an acceptable learning curve. Also, some hyperparameters, such as
learning rate and batch size in table Table 1, were configured due to hardware
limitations as we only depended on a 12GB NVIDIA GeForce GPU to carry out the
experiments in this work.

Table 1: The learning rate and the batch size used in each experiment Model | Learning Rate | Batch Size
---|---|---
BERT | 5e-6 | 2
AST | 5e-6 | 2
Baseline | 5e-8 | 2
Attention-based | 5e-8 | 2

####  3.3.1 evaluation metric

Some of the most robust and widely used performance measures for
classification tasks are: (i) recall, (ii) precision, and (iii) F1 score. The
recall metric measures the proportion of actual positives that the model is
able to classify. Precision, on the other hand, measures the proportion of
predicted positives that are correctly identified. Individually, these metrics
do not offer a complete view of model performance. The F1 score computes the
harmonic mean of the recall and precision offering a robust evaluation. Since
the data are imbalanced, we use a weighted F1 score to account for each
class’s contribution based on its support, which considers the number of
actual occurrences of the class in the dataset. Moreover, we compute and
present the macro average and weighted average performances of the models to
show the overall system performance. We present the macro average to assess
precision, recall, and F1 score across individual classes, treating all
classes equally. We also use the weighted average performance to account for
the importance/weight of each category when calculating the overall system
performance.

Table 2: Performance of text and audio unimodal models by emotion class.

| text | speech
---|---|---
Categories | precision | recall | F1 score | precision | recall | F1 score
Anger | 0.497 | 0.487 | 0.492 | 0.387 | 0.249 | 0.303
Disgust | 0 | 0 | 0 | 0 | 0 | 0
Fear | 0.5 | 0.02 | 0.038 | 0 | 0 | 0
Joy | 0.602 | 0.627 | 0.615 | 0.228 | 0.192 | 0.208
Neutral | 0.741 | 0.847 | 0.790 | 0.577 | 0.757 | 0.655
Sadness | 0.376 | 0.337 | 0.355 | 0.175 | 0.154 | 0.164
Surprise | 0.654 | 0.537 | 0.590 | 0.314 | 0.231 | 0.266
Macro Avg. | 0.481 | 0.408 | 0.412 | 0.240 | 0.226 | 0.228
Weighted Avg. | 0.623 | 0.654 | 0.633 | 0.411 | 0.464 | 0.429

##  4 Result and Discussion

The proposed deep learning systems consist of appropriate deep learning
algorithms for each data type. They are trained and tested on two types of
data: text and speech data.

We present several performance aspects of the proposed systems for recognizing
different types of emotions. We use several standard evaluation metrics for
each category: precision, recall and F1 score, macro average and weighted
average F1 score. These performance measures allow us to identify the model’s
performance at the category level. In the following subsections, we outline
the performance of the systems when trained and tested for emotion
classification.

###  4.1 Model evaluation result at category level

####  4.1.1 Text unimodal performance

Table 2 shows the performance of the text-based unimodal system in classifying
emotions when trained and tested on text data. The second, third, and fourth
columns show the precision, recall and F1 score for each type of emotion. The
text-based unimodal model failed to classify the “disgust” emotion. Although
it scores a recall of 50% when classifying the “fear” category, it has a very
poor precision of 0.02 (2%), resulting in a very poor F1 score of 0.038. The
text-based unimodal model classifies the “neutral” category more accurately
than other categories, achieving an F1 score of 0.790. This is followed by the
“joy” category (with an F1 score of 0.615) and the “surprise” category (with
an F1 score of 0.590). The performance in classifying “anger” and “sadness”
categories are F1 scores of 0.492 and 0.355, respectively.

####  4.1.2 Speech unimodal performance

The speech-based unimodal system performance is relatively similar to that of
the text-based unimodal system. It performs better in classifying the
“neutral” emotion than any other type of emotion and performs very poorly in
classifying “disgust” and “fear” emotions. As presented in Table 2, the
speech-based unimodal system fails to classify “disgust” and “fear” categories
while achieving an F1 score of over 0.655 for the “neutral” category. Similar
to the text-based unimodal system, following the “neutral” category, the
speech unimodal system achieves an F1 score of 0.303, 0.266, and 0.208 for the
“anger”, “surprise”, and “joy” categories, respectively.

Table 3: Performance of the baseline multimodal model by emotion class.  Categories | precision | recall | F1 score
---|---|---|---
Anger | 0.577 | 0.435 | 0.496
Disgust | 0.5 | 0.012 | 0.029
Fear | 0.154 | 0.08 | 0.105
Joy | 0.568 | 0.657 | 0.609
Neutral | 0.793 | 0.766 | 0.779
Sadness | 0.333 | 0.375 | 0.353
Surprise | 0.483 | 0.705 | 0.573
Macro Avg. | 0.487 | 0.433 | 0.421
Weighted Avg. | 0.640 | 0.635 | 0.627
Table 4: Performance of the attention-based multimodal model by emotion class. Categories | precision | recall | F1 score
---|---|---|---
Anger | 0.516 | 0.672 | 0.584
Disgust | 0 | 0 | 0
Fear | 0 | 0 | 0
Joy | 0.675 | 0.709 | 0.692
Neutral | 0.858 | 0.792 | 0.824
Sadness | 0.478 | 0.418 | 0.446
Surprise | 0.616 | 0.868 | 0.721
Macro Avg. | 0.449 | 0.494 | 0.467
Weighted Avg. | 0.689 | 0.706 | 0.693

####  4.1.3 Baseline multimodal performance

We evaluated the proposed multimodal baseline system to assess its performance
at classifying different types of emotions. Table 3 presents the performance
of the baseline system. The system produced the highest F1 score in
classifying the “neutral” emotion (0.779), followed by “joy” (0.609). The
system produced the lowest F1 scores in classifying the ’ disgust” and ‘fear”
categories with 0.029 and 0.105, respectively. The overall F1 score across all
the emotion categories is 0.627. The baseline system has a reasonable overall
weighted average of precision and recall across all emotion categories,
producing scores of 0.640 and 0.635, respectively.

####  4.1.4 Attention-based multimodal performance

Table 4 presents the performance of the attention-based system. The system
produced the highest weighted F1 score, recall, and precision compared to
other models in this work, with scores of 0.693, 0.706, and 0.689,
respectively. It also produced the highest macro average F1 score, with 0.467.
Although it was unable to classify any instances from the least represented
classes “fear” and “disgust”, the system produced the highest F1 score in
classifying the “neutral” emotion (0.824), followed by “surprise” (0.721), and
then “joy” (0.692).

Table 5: Comparative result. Weighted F1 score (in %) Models | Speech | Text | Speech+Text
---|---|---|---
Majumder et al [15] | 41.80 | 57.00 | 60.30
Ho et al [16] | 45.30 | 59.00 | 60.50
Lian et al [17] | 38.20 | 58.30 | 60.50
Baijun et al [18] | 32.10 | 61.20 | 64.00
Dutta et al [9] | 50.10 | 65.60 | 65.80
Proposed baseline model | 42.90 | 63.3 | 62.7
Proposed attention-based model | 42.90 | 63.3 | 69.7

###  4.2 System performance comparison with previous works

Table 5 presents the weighted F1 score performance of our proposed systems
(baseline and attention-based) and several previously published works on
unimodal and multimodal emotion classification. Our proposed unimodal systems
lagged behind some previously published systems, while our proposed multimodal
systems have performed better than most of the other systems.

Our proposed attention-based system significantly outperformed state-of-the-
art systems by achieving a 69.7% F1 score, nearly 5% higher than the reported
state-of-the-art system proposed by [9]. This is despite the low performance
of our unimodal systems compared to the unimodal systems proposed by [9]. This
indicates the strength of our novel method of combining the last hidden layers
of attention-based text and speech unimodal models using a multi-head cross-
attention layer followed by a multi-head self-attention layer for fusing text
and audio modalities.

###  4.3 Multimodal performance discussion

Our proposed speech-based unimodal emotion classifier underperforms compared
to those classifiers proposed by Dutta et al [9] and Ho et al [16]. Our text-
based unimodal model is only outperformed by the model proposed by Dutta et al
[9]. However, our multimodal classifier outperforms the state-of-the-art. We
anticipate that the superior performance of our multimodal system results from
our homogenous attention-based model which includes an attention-based fusion
method that merges the learned information from attention-based unimodal
models, which are fine-tuned versions of BERT and AST. In comparison to
several other published works, our speech-based unimodal performance is on par
with many other systems and better than others in some cases. Our text-based
unimodal model performs better than almost all other published works, as shown
in Table 5. Furthermore, our multimodal attention-based system outperforms all
other published works, making it the state-of-the-art multimodal emotion
classifier.

Figure 3: text unimodal model

##  5 Error analyses: confusion matrix

Machine learning applications for supervised classification tasks require a
set of labeled data that represents the classes. Labeled datasets often
contain errors. One of the main issues is label errors (annotation errors),
where some data samples are labeled with incorrect emotions. Annotation errors
impact machine learning algorithms’ performance because the algorithms learn
from the errors present in the annotated dataset. Confusion matrix graphs are
helpful for highlighting classification errors by identifying misclassified
classes and the classes they are confused with. We present several confusion
matrices to outline the classification errors of our proposed unimodal and
multimodal systems in classifying different types of emotions.

###  5.1 Emotion misclassification errors

As presented in Fig. 3, our text unimodal system miscalssifies“anger” class as
“netural” 26.1%. The class“disgust” is misclassified with “neutral” 39.7%,
“fear” is misclassified with“neutral” 40%, and topped by “sadness” with 41.8%.
Contrary to the “neutral” class that has the largest data samples in the
dataset, the text-based unimodal does not confuse any class (except “joy”
confused with “fear”, though negligibly) with the “disgust” and “fear”
classes, which have the fewest data samples in the dataset.

Figure 4: Speech unimodal model Figure 5: Attention-based multimodal model

The speech-based unimodal seems to miss-classify most of the classes largely
with the “neutral”, as shown in Fig. 4. The miss-classification error rate
between “neutral” and all other class of emotions range from 44.8%, where
“surprise” is miss-classified as “netural”, to 60.0%, where “sadness” is miss-
classified as “neutral”. The “neutral” class appears to be the least miss-
classified class. It has 75.7% accuracy. It’s highest miss-classified rate of
9.7% is with “joy” followed by “sadness” (5.7%) and “surprised” (4.6%).

The cross-attention multimodal system, as presented in Fig. 5, confused
“disgust” class as “anger” (58.8% error rate), “neutral” (11.8% error rate),
“sadness” (17.6% error rate), and “surprised 8.8% error rate). The cross-
attention multimodal has increased the accuracy of classifying “anger” emotion
by reducing the error rate of miss-classification of this type of emotion with
“neutral” to 9.9% compared with the error rate recorded in the unimodal
systems. However, “anger” emotion is miss-classified as “joy” 8.7%, as
“sadness” 2.3% and as “surprise” 11.1%. “Sandess” emotion class has been
confused by the cross-attention multimodal system largely as “neutral” with an
error rate of 27.4%, followed by 17.3% error rate when miss-classified as
“anger”, 4.8% as “joy” and 8.7% as “surprise”. Other emotion classes (“joy”
and “surprise”) are mostly miss-classified as “neutral” and“neutral” class is
largely confused as “joy” (6.4% error rate). It appears our proposed multi-
head cross-attention system performed best at classifying “surprise” emotion
with 86.8% accuracy.

##  6 conclusion and future work

Recent advances in automated data classification of emotion involve the
application of machine learning algorithms to automate the process of
classifying certain types of emotions (e.g., anger, happy, surprised, etc.).
The focus of this study was to design and evaluate novel deep learning
architectures that learn from different types of data (text and speech) to
classify different kinds of emotions. We applied deep learning based language
models, such as BERT and Audio Spectrogram Transformer. We proposed novel deep
learning system to fuse embedding data using attention mechanism for
extracting important features from different types of data to classify
emotion. We have measured the performance of each architecture using different
performance matrices. We present rigorous error analyzes of the proposed
system classification performance, where certain emotion classes are miss-
classified. Our finding suggests that deep learning architectures trained only
on text or speech data could underperform architectures that are trained on a
fusion of data (e.g., text and speech). Thus, proving that multimodal systems
could perform better than unimodal systems for emotion classification. Our
future work is to extend our multimodal system to evaluate it on sentiment
analyses, and to fuse other data types when training it (e.g., fusing video
data with text and speech). We believe this is feasible because the public
dataset (MELD) contains videos and our current architecture is designed to be
simple and thus allowing us to easily extend it to include information learn
by current embedding algorithms from videos. Also, MELD data contains
information for sentiment analysis.

## References

  * [1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  CoRR, abs/1810.04805, 2018.
  * [2] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized BERT pretraining approach.  CoRR, abs/1907.11692, 2019.
  * [3] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * [4] Zheng Lian, Bin Liu, and Jianhua Tao.  Smin: Semi-supervised multi-modal interaction network for conversational emotion recognition.  IEEE Transactions on Affective Computing, pages 1–1, 2022.
  * [5] Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov.  Multimodal transformer for unaligned multimodal language sequences, 2019.
  * [6] Soumya Dutta and Sriram Ganapathy.  Multimodal transformer with learnable frontend and self attention for emotion recognition.  In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6917–6921, 2022.
  * [7] Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency.  Memory fusion network for multi-view sequential learning.  CoRR, abs/1802.00927, 2018.
  * [8] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency.  Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236–2246, Melbourne, Australia, July 2018. Association for Computational Linguistics.
  * [9] Soumya Dutta and Sriram Ganapathy.  Hcam – hierarchical cross attention model for multi-modal emotion recognition, 2023.
  * [10] Jingye Li, Donghong Ji, Fei Li, Meishan Zhang, and Yijiang Liu.  HiTrans: A transformer-based context- and speaker-sensitive model for emotion detection in conversations.  In Proceedings of the 28th International Conference on Computational Linguistics, pages 4190–4200, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.
  * [11] Deepanway Ghosal, Navonil Majumder, Soujanya Poria, Niyati Chhaya, and Alexander F. Gelbukh.  Dialoguegcn: A graph convolutional neural network for emotion recognition in conversation.  CoRR, abs/1908.11540, 2019.
  * [12] Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha.  M3er: Multiplicative multimodal emotion recognition using facial, textual, and speech cues.  In AAAI Conference on Artificial Intelligence, 2019.
  * [13] Yuan Gong, Yu-An Chung, and James R. Glass.  AST: audio spectrogram transformer.  CoRR, abs/2104.01778, 2021.
  * [14] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea.  MELD: A multimodal multi-party dataset for emotion recognition in conversations.  CoRR, abs/1810.02508, 2018.
  * [15] Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander F. Gelbukh, and Erik Cambria.  Dialoguernn: An attentive RNN for emotion detection in conversations.  CoRR, abs/1811.00405, 2018.
  * [16] Ngoc-Huynh Ho, Hyung-Jeong Yang, Soo-Hyung Kim, and Gueesang Lee.  Multimodal approach of speech emotion recognition using multi-level multi-head fusion attention-based recurrent neural network.  IEEE Access, 8:61672–61686, 2020.
  * [17] Zheng Lian, Bin Liu, and Jianhua Tao.  Ctnet: Conversational transformer network for emotion recognition.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:985–1000, 2021.
  * [18] Baijun Xie, Mariia Sidulova, and Chung Hyuk Park.  Robust multimodal emotion recognition from conversation with transformer-based crossmodality fusion.  Sensors, 21(14), 2021.


Ultrafast electro-optic Time-Frequency Fractional Fourier Imaging at the Single-Photon Level

Michał Lipka,1,* and Michał Parniak1

The Fractional Fourier Transform (FRT) corresponds to an arbitrary-angle rotation in the phase space, e.g. the time-frequency (TF) space, and generalizes the fundamentally important Fourier Transform. FRT applications range from classical signal processing (e.g. time-correlated noise optimal filtering) to emerging quantum technologies (e.g. super-resolution TF imaging) which rely on or benefit from coherent low-noise TF operations. Here a versatile low-noise single-photon-compatible implementation of the FRT is presented. Optical TF FRT can be synthesized as a series of a spectral disperser, a time-lens, and another spectral disperser. Relying on the state-of-the-art electro-optic modulators (EOM) for the time-lens, our method avoids added noise inherent to the alternatives based on non-linear interactions (such as wave-mixing, cross-phase modulation, or parametric processes). Precise control of the EOM-driving radio-frequency signal enables fast all-electronic control of the FRT angle. In the experiment, we demonstrate FRT angles of up to 1.63 for pairs of coherent temporally separated 11.5-wide pulses in the near-infrared (800). We observe a good agreement between the simulated and measured output spectra in the bright-light and single-photon-level regimes, and for a range of pulse separations (20 to 26.67). Furthermore, a tradeoff is established between the maximal FRT angle and bandwidth, with the current setup accommodating up to 248 of bandwidth.
With the ongoing progress in EOM on-chip integration, we envisage excellent scalability and vast applications in all-optical TF processing both in the classical and quantum regimes.

§ INTRODUCTION

The time-frequency (TF) domain plays a fundamental role from classical photonic communication, extensively relying on technologies like wavelength-division-multiplexing, to emerging quantum technologies [1, 2, 3, 4, 5, 6, 7, 8], for which complete TF frameworks have been proposed [9]. The latter in particular include super-resolution imaging [10, 11], mode sorting [12], continuous-variables protocols [13, 14], bandwidth shaping [15, 16], spectroscopy [5], multi-mode quantum repeaters [7], and TF imaging systems serving as waveform compressors [17] or optical oscilloscopes [18], also in one-shot configurations [19].
The need for diverse ultrafast TF characterization methods grows with the increasing prevalence of femtosecond-laser-based techniques in biology, chemistry, and spectroscopy [20], communications and medicine [21], material science [22], quantum information and atomic and molecular physics [23].
Among these, TF protocols in the quantum regime most often require precisely controlled time-frequency transformations that are sufficiently low-noise for the single-photon-level light.

The Fractional Fourier Transform (FRT) corresponding to an arbitrary rotation in the TF space constitutes an important operation vastly prevalent in classical signal processing e.g. for optimal filtering of time-correlated noise [24], encryption [25] or chirp-based encoding schemes [26]. All-optical FRT processing avoids the digitization of an optical signal, often detrimental in terms of the system bandwidth and noise figures. Furthermore, electro-optic FRT has been theoretically proposed as a pre-distortion method mitigating temporal pulse broadening [27] and as a central element of secure chaos-based communication [28].

Here we experimentally present a scalable, all-electronically-controlled implementation of a low-noise single-photon-compatible FRT based on a grating pulse stretcher followed by an electro-optic modulator (EOM) acting as a temporal lens. The implementation is aimed at the high-bandwidth regime of faint single-photon-level picosecond pulses where an ability to coherently transfer the elusive temporal structure to the much more easily accessible spectral degree of freedom is greatly valued.

For optical TF processing, the FRT constitutes a fundamental, primitive operation - a coherent rotation of the TF coordinates. As such its applications are wide.
In the quantum regime, one example is the TF implementations of super-resolution protocols based on mode-sorting. Indeed a constellation of $N$ parallel FRT operations with the FRT angles separated by $2\pi/N$ and followed by a passive linear optical network can implement an $N$ mode sorter [29].
In the context of ultrafast pulse characterization, FRT combined with a projective spectral measurement provides a tomographic cross-section of the TF space, while combined with spectral interferometry, it allows for the retrieval of otherwise elusive cubic spectral phase coefficient [30]. Porting the ideas from conventional signal processing techniques also holds a promise to provide new applications, in particular related to filtration of time-correlated noise directly at the level of an optical signal. Such processing directly in the optical domain is particuarly important in the photon-starved regime.

Optical FRT has been previously demonstrated with a single dispersive element – a linearly chirped Bragg diffraction grating (LCFBG) implementing a TF analog of the Fresnel diffraction [31]. However, this conceptually and practically simple implementation remains constrained by poor frequency resolution, limited bandwidth, and lack of further tunability once the LCFBG is manufactured. An interesting alternative has been demonstrated in ref. [32] which describes a feedback-loop-based single-element FRT for optically-carried RF waveforms. The bandwidth is however limited by the loop response time to tens of MHz. A controllable TF FRT has been also implemented in a quantum memory, aimed at sub-GHz-bandwidth light [33]. For ultrafast high-bandwidth pulses an adjustable, however, technically demanding implementation of TF FRT includes a chain of a dispersive element (pulse stretcher), quadratic temporal phase modulation (a time lens), and another dispersive element [34].
Hitherto demonstrations of ultrafast TF FRT relied on non-linear phenomena such as wave-mixing, cross-phase modulation, or parametric processes to implement the temporal lens, achieving large modulation depth, however at the cost of introducing inherent optical noise detrimental to the single-photon-level operation  [35, 36].

The EOM-based implementation remains low-noise and single-photon-compatible at the same time offering a robust and versatile control of the FRT angle via all-electronic adjustments of the driving signal.
While an ordinary Fourier transform – a special case of FRT has been demonstrated with an EOM-based time lens and a grating-based dispersive line [37], it has not been extended experimentally nor in the interpretation to the regime of arbitrary FRT angles.
The task of which requires precise control over the EOM driving waveform.
Finally, EO TF FRT promises excellent scalability and compactness if the diffraction-grating-based dispersive lines are exchanged for fiber-based Bragg diffraction gratings and the EOMs and linear optical elements are implemented on-chip, which is in the range of current technology.

The article is organized as follows: in section <ref> we introduce the Fractional Fourier Transform and its action in the time-frequency phase-space, also for a special case of an initial state of two temporarily separated pulses which is further studied experimentally. Section <ref> presents the experimental and simulation results alongside an overview of the setup. Details of the experiment, simulation, and calculations are presented in section <ref>. In particular, we describe therein the methodology of the design and calibration of the optical and RF setups and discuss the bounds on the FRT bandwidth. Finally, we draw conclusions in section <ref>.

Schematic depiction of a Fractional Fourier transform in the optical domain. A subsequent application of a quadratic phase in spectral $\Phi_2$, temporal $K$, and spectral $\Phi^{'}_2$ domains yields an arbitrary rotation of the initial state in the time-frequency space.

§ FRACTIONAL FOURIER TRANSFORM

Fractional Fourier Transform (FRT) is a natural extension of a Fourier transform – a transformation fundamental for (optical) signal processing [38, 39, 40, 41].
The definition of FRT as an integral transform (see e.g. ref. [39]) is not directly interpretable in terms of physical operations and as such we shall not repeat it here.

Instead, a more insightful way to define FRT is as a propagator of the quantum harmonic oscillator. The behavior of which can be simply described as a rotation of the state's quasi-probability distribution in its TF phase space. In this picture, FRT of degree $P$ acts as a rotation by an angle $\alpha=P\pi/2$ [39] with the special case of $P=1$ being the ordinary Fourier transform. To this end, let us describe the TF phase space with the Chronocyclic Wigner Function (CWF) defined for a well-behaved function $f(t)$ as [42]:

\begin{equation}
\mathcal{W}_f(\omega,t) =
\infint t'\ f\left(t + \frac{t'}{2}\right)f^{*}\left(t - \frac{t'}{2}\right)e^{i\omega t'}.
\end{equation}

Conveniently, this rotation can be accomplished using alternating temporal and spectral shearing operations, with two possible sequences: temporal-spectral-temporal or spectral-temporal-spectral operations [39].
We are concerned with the latter case which at the level of light's electric field corresponds to imposing a quadratic phase in spectral

\begin{equation}
\varphi_\omega(\omega)=\Phi_2(\omega-\omega_c)^2/2,
\label{eq:phiomega}
\end{equation}

\begin{equation}
\varphi_t(t)=K t^2/2,
\label{eq:phit}
\end{equation}

and again spectral domains ($G^{'}_\omega$, $\omega^{'}_c$), where $\omega_c$, $\omega^{'}_c$ are some central frequencies, the time frame is aligned to the temporal centroid of the pulse, $\Phi_2$ is the group delay dispersion (GDD), and $K$ is the time-lens chirp rate. The sequence and the CWF at different stages have been schematically depicted in Fig. <ref> for an initial state of two coherent, temporarily separated Gaussian pulses.
Notably, these operations are completely analogous to a series of a propagation, passing through a lens, and a propagation of a single transverse spatial dimension in the paraxial approximation [43, 44, 45].

In our notation, the FRT angle $\alpha$ is given by the following equations [39]

\begin{align}
& \Phi_2 = \mathcal{G} \tan{\frac{\alpha}{2}},
\label{eq:alphdefeqsGom}\\
& K = \mathcal{G}^{-1} \sin{\alpha},
\label{eq:alphdefeqsGt}
\end{align}

where $\mathcal{G}$ is a GDD scaling factor. Solving for $\alpha$ we get

\begin{equation}
%\alpha =\pm 2\arctan\left(\frac{\sqrt{\Phi_2 K}}{\sqrt{2-\Phi_2 K}}\right),
\alpha =\pm 2\arcsin\sqrt{\frac{\Phi_2 K}{2}},
\label{eq:frtangle}
\end{equation}

with the $+$ ($-$) sign for $0\leq\alpha<\pi$  ($-\pi<\alpha\leq 0$).
Notice that Eqs. (<ref>),(<ref>) hold if $K$ and $\Phi_2$ have the same sign, hence experimentally a positive dispersion must be matched with a positive time lens and vice versa.

Let us write the electric field of light pulse in the slowly varying envelope (SVE) approximation [46].

\begin{equation}
\mathcal{E}(t) = \mathcal{A}(t)\exp(i\omega_0 t) =
\frac{\exp(i\omega_0 t)}{\sqrt{2\pi}}\int \mathrm{d}\tilde{\omega} \:\tilde{\mathcal{A}}(\tilde{\omega})\exp{(i\tilde{\omega} t)},
\label{eq:et_svea}
\end{equation}

where $\tilde{\omega}=\omega-\omega_0$ and $\omega_0$ is the central optical frequency. We can further transform just the SVE $\mathcal{A}(t)$ while bearing eq. (<ref>) in mind.

We begin by imposing the spectral phase, as given by Eq. (<ref>).

$\tilde{\mathcal{A}}(\tilde{\omega})\rightarrow \tilde{\mathcal{A}}(\tilde{\omega})\times\exp\left[{i\varphi_\omega(\omega)}\right]$.

Observe that any misalignment $\omega_c-\omega_0\neq 0$ between the central optical frequency of the pulse $\omega_0$ and the central frequency of the first stretcher $\omega_c$ effectively amounts to an additional linear spectral phase

\begin{equation}
\frac{\Phi_2}{2}(\omega-\omega_c)^2 = \frac{\Phi_2}{2}\left[\tilde{\omega}^2 - 2\omega(\omega_c-\omega_0)\right] + \mathrm{const.}
\end{equation}

which in the time domain manifests as an additional delay. In practice, such a misalignment will be compensated for when the delay of the EOM RF driving signal is matched to the arrival time of the pulse. Under such an assumption, the SVE after the stretcher reads:

\begin{equation}
\mathcal{A}_1(t)=\frac{1}{\sqrt{2\pi}}\int \mathrm{d}\tilde{\omega} \:
\tilde{\mathcal{A}}(\tilde{\omega})
\exp{\left(i \frac{\Phi_2}{2}\tilde{\omega}^2\right)}
\exp{\left(i\tilde{\omega} t\right)}.
\end{equation}

Finally, EOM modulation imposes a temporal phase according to Eq. (<ref>) $\mathcal{A}_2(t)=\mathcal{A}_1(t)\times\exp{\left[i\varphi_t(t)\right]}$.
In the experiment, we will observe the pulses with a spectrometer, which up to its limited resolution measures:

\begin{align}
\tilde{\mathcal{A}}_2(\tilde{\omega})=&\frac{1}{\sqrt{2\pi}} \int \mathrm{d} t \: \mathcal{A}_1(t)
\exp{\left[i\varphi_t(t)\right]}
\exp{\left(-i\tilde{\omega} t\right)}.
\label{eq:a2om}
\end{align}

Hence, we can neglect the last stretcher which ideally introduces only a spectral phase component, not observable in the spectrally-resolved intensity measurement.

Note that we have included only the quadratic terms in the spectral and temporal phase modulations. Higher-order terms become particularly important with higher bandwidth (spectral) or longer pulses (temporal). We further discuss the regime in which our approximation remains valid. For a rigorous description of the higher-order terms in a similar task of spectral-temporal imaging see ref. [47].

§.§ Two pulses

Let us consider a special case of two identical pulses separated in time

\begin{equation}
\mathcal{A}(t) = \frac{1}{\sqrt{2}}\left[a(t-\frac{\delta t}{2})+e^{i\varphi}a(t+\frac{\delta t}{2})\right],
\end{equation}

where $a(t)$ describes the SVE of a single pulse, $\varphi$ their phase difference, and $\delta t$ the temporal separation.
Such an input state, analogous to a cat state in quantum optics, constitutes a simple and traceable yet informative probe of the FRT setup parameters.
Indeed the spectrum of the input state consists of a series of fringes, their density dependent on $\delta t$, while the spectra of the constituent pulses are identical.

\begin{equation}
|\tilde{\mathcal{A}}(\tilde{\omega})|^2=|\tilde{a}(\tilde{\omega})|^2\left[1+\cos{\left(\tilde{\omega} \delta t  + \varphi \right)}\right]
\label{eq:initspectrum}
\end{equation}

Rotation in the TF space spectrally separates both pulses, gradually removing the fringes.
Let us assume Gaussian pulses with spectral width $\sigma$:

\begin{equation}
a(t)=\left(\frac{\sigma^2}{\pi}\right)^{1/4}\exp{\left( -\frac{1}{2}t^2\sigma^2\right)}.
%\tilde{a}(\tilde{\omega})=\frac{1}{(\pi\sigma^2)^{\frac{1}{4}}}\exp{\left( -\frac{\tilde{\omega}^2}{2\sigma^2}\right)}.
\end{equation}

Evaluating the integrals of Eq. (<ref>) we get the following spectral intensity:

\begin{equation}
\frac{1}{\sqrt{\pi \xi \sigma^2 }} \times
\exp{\left(-\frac{\tilde{\omega} ^2}{\xi  \sigma ^2}-\frac{\delta t^2 K^2}{4 \xi  \sigma ^2}\right)}
\times
\left\lbrace\cosh
\left(\frac{\delta t \tilde{\omega}  K}{\xi  \sigma ^2}\right)+\cos \left[\frac{\delta t \tilde{\omega}
}{\xi }\left(1-K \Phi_2\right)+\varphi \right]\right\rbrace,
\end{equation}

\begin{equation}
\xi =\frac{K^2}{\sigma ^4}+\left(K \Phi_2-1\right){}^2.
\end{equation}

Indeed, putting $K=0$ ($\xi=1$) correctly retrieves the initial spectrum, as given by Eq. (<ref>).
Whereas for the FRT angle of $\alpha=\pi/2$ we get the ordinary Fourier transform $t \rightarrow \mathcal{G}\omega$. In this case, using Eqs. (<ref>), (<ref>), we have $K=\mathcal{G}^{-1}$, $\Phi_2=\mathcal{G}$, $\xi=1/(\sigma^4 \mathcal{G}^2)$ and the transformed spectrum reads:

\begin{equation}
\frac{\mathcal{G} \sigma}{\sqrt{\pi }}
\exp{\left(-\frac{\sigma^2 \delta t^2}{4}\right)}
\exp{\left(-\sigma ^2  \tilde{\omega} ^2 \mathcal{G}^2\right)}\times
\left[\cosh \left(\delta t
\tilde{\omega}  \mathcal{G} \sigma ^2\right)+\cos (\varphi )\right],
\end{equation}

which contains (after some algebraic transformations) two Gaussians spectrally separated (in angular frequency) by $\delta t/\mathcal{G}$. Notably, there are no spectral fringes, as the dependence on $\omega$ under the cosine term vanishes.

Simplified experimental setup. Ultrafast ca. 100 pulses from a Ti:Al$_2$O$_3$ laser are spectrally filtered to ca. 74 (4f filter). The pulses are split and recombined in a Mach-Zehnder-type interferometer with a controlled arm-length difference (Pulse Splitter). The quadratic spectral phase is imposed by a grating-based quadruple-pass stretcher while the quadratic temporal phase by an electro-optic modulator (EOM). The EOM-driving signal is first obtained by exciting a photodiode with a small part (ca. average 1, 80 repetition) of the pre-filtered pulse. The radio-frequency (RF) line shapes and amplifies the signal. Spectra are measured either with a standard or a single-photon-level spectrometer. PBS – polarizing beamsplitter, H(Q)WP – true-zero-order half- (quarter-) waveplate, VS – vertical-shift retroreflector, LN – low noise, S – standard, VA – variable attenuator, BPF – band-pass filter.

§ EXPERIMENT

(a)-(c) Experimental spectra for a pair of temporally separated pulses undergoing FRT, for different levels of chirp rate $K$, with a fixed $\Phi_2=\SI{15.5}{\pico\second\squared}$. Equivalent FRT angles $\alpha$ are indicated. Each column corresponds to a different temporal separation of pulses $\delta t$.
(d)-(f) Corresponding simulated spectra. Fidelities $\mathscr{F}$ between experimental and simulated maps are (d) $96.28\%$, (e) $96.27\%$, (f) $95.88\%$. Simulation parameters were $f_m=\SI{15}{\giga\hertz}$, $\Delta_t=\SI{11.6}{\pico\second}$, $\Phi_2=\SI{15.5}{\pico\second\squared}$ (see section <ref>).
(g)-(i) Equivalent experimental spectra, collected at a single-photon level with an average number of photons per frame of $\bar{n}=0.38$ ($2.4\times10^{-4}$ photons per pulse) and $50\times10^3$ frames per each $K$ setting. Note a different phase $\varphi$, visible in the fringes.

In the experiment, as depicted in Fig. <ref>, we begin with 100 pulses from Ti:Sapphire laser (SpectraPhysics MaiTai) at a 80 repetition rate and a central wavelength of 800. With a 4f grating pulse shaper we apply an amplitude rectangular mask in the spectral domain to carve ca. 74 FWHM pulses which are coupled to a polarization-maintaining (PM) fiber and sent to a combined free-space and fiber Mach-Zehnder interferometer with a regulated delay in one arm. The interferometer splits the pulse into two temporarily separated parts with otherwise identical modes. One of the interferometer outputs is sent via fiber to the main FRT setup.
The FRT is implemented as a series of a grating stretcher (Martinez [48] configuration with a single grating, quadruple-pass) and a fiber EOM. The output of the FRT is observed with a spectrometer. The efficiency of the FRT is ca. $1\%$ excluding detection.

While in principle the EOM would be followed by another stretcher, its quadratic spectral phase is not measurable with a spectrometer (see sec. <ref>).

Experimentally collected spectra with bright light, for a range of FRT angles $\alpha$, have been depicted in Fig. <ref>(a)-(c) together with a simulation (d)-(f), partially optimized for the highest fidelity $\mathscr{F}$. Equivalent single-photon-level measurements are depicted in Fig. <ref> (g)-(i).

Note a very good agreement between the experimental spectra and the simulation. Residual infidelity stems mostly from an artificial spectrum modulation, visible in the strong-light (a)-(c) measurements, albeit not in the single-photon level measurements (g)-(i), and most probably stemming from a polarization-degree interference in the spectrometer.

Even with a relatively low $\Phi_2$ which can be easily extended several times in the current setup, the range of FRT angles $\alpha$ goes beyond the point of ordinary Fourier transform $\alpha=\pi/2$.

§ METHODS

§.§ Simulation

A direct numerical calculation has been performed to compare against experimental data. We assumed Gaussian, Fourier-limited pulses with intensity temporal FWHM of $\Delta_t=\SI{11.6}{\pico\second}$. The stretcher is assumed to produce only quadratic spectral phase, while for the EO modulator, we model the phase as a cosine with a main frequency, as experimentally measured, of $f_m=\SI{15}{\giga\hertz}$ and the amplitude equivalent of $K$ under a series expansion of cosine to the quadratic degree.
The simulation procedure consists of generating the SVE of the pulses on a temporal grid and a subsequent series of fast Fourier transforms intertwined with multiplications (in the matching domain) by spectral or temporal phase profiles corresponding to the action of a stretcher or a time lens, respectively.

§.§ Fidelity
Fidelity $\mathscr{F}$ as a measure of correspondence between the simulated $\mathcal{I}^{(\mathrm{th})}(f,K)$ and experimental spectra $\mathcal{I}^{(\mathrm{exp})}(f,K)$ has been calculated as follows:

\begin{equation}
\mathscr{F} = \frac{\sum_{f,K}\sqrt{\mathcal{I}^{(\mathrm{exp})}(f,K)\mathcal{I}^{(\mathrm{th})}(f,K)}}{\sqrt{\sum_{f,K}\mathcal{I}^{(\mathrm{exp})}(f,K)}\sqrt{\sum_{f,K}\mathcal{I}^{(\mathrm{exp})}(f,K)}},
\end{equation}

where the summation goes over the discrete points of the measured and simulated spectra in frequency $f$ and chirp rate $K$ coordinates.

The simulation parameters were taken as independently measured in the experiment ($\Phi_2$, range of $K$, $f_m$) or were optimized over for the highest fidelity (relative phase between pulses $\varphi$, the temporal mismatch between the time lens center and the pulses, and pulses FWHM $\Delta_t$).

§.§ Second stretcher

A complete FRT setup includes a second stretcher after the temporal lens to correct the spectral phase of the state. Its inclusion is vital with temporal or spectro-temporal characterization methods, as exemplified in the last two panels of Fig. <ref>, and implemented for sub-GHz pulses in ref. [33]. However, with a spectral intensity measurement, it does not affect the final results, except for the reduced efficiency of the setup. Hence we have experimentally verified the equivalence of the collected spectra with and without the last stretcher for a few cases and further proceeded without for the results presented herein. Finally, we note that simultaneous spectral and temporal phase characterization in the 100 regime remains technically challenging, in particular with confidence and resolution sufficient to characterize the FRT operation.

§.§ Grating stretcher design

The stretcher is designed with a single grating, in a configuration akin presented in ref. [49]. As illustrated in Fig. <ref>, a complex of a diffraction grating (Newport 33067FL01-290R, 1800ln, 26.7 blaze angle) D-shaped input and output mirrors and a vertical-shift retro-reflector is placed on a translation stage. The stage movement direction, and input, diffracted, and output beams are kept parallel, allowing simple regulation of the GDD with the stage position. Input D-shape mirror and the grating angles can be adjusted together to set the frequency of the shortest optical path $\omega_c$ and the GDD scaling per unit length.

A first-order diffracted (in the horizontal plane) beam travels over distance $L\approx\SI{71.5}{\milli\meter}$ and then enters a unit-magnification telescope with a mirror placed in the Fourier plane of the imaging lens (focal length $f$). Importantly, the lens is vertically shifted to spatially separate the first returning beam. Vertical-shift retro-reflector sends the returning beam back through the grating-telescope setup. The vertical positions are chosen, so that the last returning beam (4th pass) can exit via an output D-shape mirror.

Following ref. [50] the total GDD of such a stretcher is given by:

\begin{equation}
\Phi_2 = \frac{ m^2 \lambda^3 L}{2 \pi c^2 d^2 \cos^2\theta_d},
\label{eq:gdd}
\end{equation}

where $m$ is the diffraction order, $c$ the speed of light, $d$ the grating groove period, and $\theta_d$ the diffraction angle for $\omega_c$.

The choice of the focal length $f$ does not directly affect the GDD; however, the quadruple-pass design requires highly off-axis passage through the lens. Hence, care should be taken to avoid spherical aberration, especially present with lower-$f$ lenses. Ideally, an aspheric lens should be used. Additionally, if $L\gg f$ a precise calibration of the telescope becomes crucial. Simple ray optics consideration shows that the displacement of the mirror from the lens Fourier plane $\delta x$ must be kept $\delta x \ll f^2/L$.
With a narrow-band $\approx\SI{100}{\giga\hertz}$ light chromatic aberrations are less concerning. However, with wide-band pulses achromatic lenses are required, together with sufficiently large grating dimensions to accommodate the spread of the returning beam.

Since with larger $f$ the optical path in a grating stretcher can get quite long (in our case with $f=\SI{200}{\milli\meter}$ it is ca. 2) a larger beam with long Rayleigh range is generally preferred.

Fundamentally GDD of a grating stretcher has a geometrical origin [48]. Since only one spatial dimension acts as a proxy for the spectral degree of freedom, the output beam, even if correctly devoid of spatio-spectral correlations, will develop astigmatism. This is particularly prevalent for larger GDD $\gg\SI{100}{\pico\second\squared}$ and requires optical correction for efficient fiber coupling e.g. a cylindrical-lens-based telescope.

For the two implemented grating stretchers the efficiencies are 8% (the one used in this work) and 13%. Quadruple-pass through the grating itself amounts to ca. $(70\%)^4 = 24\%$ efficiency which is in agreement with the grating specification. The remaining losses are due to output fiber coupling.

The stretcher GDD measurement via the stationary phase point method. (a)-(e) Data for a single position of the strecher stage (100) and for a delay of either (a),(c) $\delta t=\SI{0}{\pico\second}$ or (b),(d) $\delta t = \SI{17}{\pico\second}$. (a),(b) Subsequent normalized interference spectra sampled at ca. 100. Frequency offset zero is arbitrary. (c),(d) Interference spectra are background-subtracted and Fourier-domain-filtered. Finally, normalized variance over the repetitions produces the illustrated signal which maximum corresponds to the stationary phase point. (e) Signal centroids (stationary phase points) for a series of delays. The slope of the linear fit corresponds to the measured GDD. (f) GDD measurement for a series of translation stage positions. Error bars are unnoticeable (typically 1%-2%).

§.§ Strecher GDD calibration

Strecher GDD has been estimated via the stationary phase point (SPP) method [51] which is based on spectrally-resolved interferometry. The input pulse (spectrally wider than for the main FRT measurement) is divided into two parts with equal intensity. One passes through the stretcher while the other is temporarily shifted by a known small delay $\delta t$ (in addition to a major delay compensating the travel time through the stretcher). The two parts then interfere on a balanced beamsplitter which one output port is observed with a spectrometer.
The dominant spectral phase imposed by the stretcher is quadratic:

\begin{equation}
\varphi(\omega)=\frac{\Phi_2}{2}(\omega-\omega_c)^2,
\end{equation}

where $\omega_c$ is a frequency for which the optical path of the stretcher is the shortest.
In turn, a delay of $\delta t$ corresponds to a linear spectral phase of $\delta t \omega$. Taking the phase difference between the arms and converting the quadratic (in $\omega$) polynomial to the canonical form, the observed fringe pattern will then have a form of

\begin{equation}
\propto \left(1+\cos\left\lbrace\frac{\Phi_2}{2}[\omega-(\omega_c+\frac{\delta t}{\Phi_2})]^2\right\rbrace\right),
\end{equation}

where we neglected the constant phase term and assumed perfect interference visibility. Hence, the stationary phase point can be observed as the point of vanishing fringe frequency $\omega_\mathrm{SPP}=\omega_c+\delta t/\Phi_2$. Estimating $\omega_\mathrm{SPP}(\delta t)$ for a series of $\delta t$ measurements allows $\Phi_2^{-1}$ to be retrieved as the linear fit coefficient.
In our approach, for each $\delta t$ we collect many spectra with the global phase between the interferometer arms slowly fluctuating. This way, characteristic circular fringe patterns can be observed on a map of the spectrum versus repetition, as illustrated in Fig. <ref> (a),(b). To estimate the SPP point, the maps are low-pass filtered along the frequency dimension (convolved with a Gaussian kernel) and a variance is calculated across repetitions. The maximum of the variance corresponds to the point where the fringe frequency vanishes (highest response to the filter) i.e. the SPP.

Fig. <ref> (e) depicts a final result for a series of $\delta t$. The slope of a linear fit corresponds to the estimated GDD $\Phi_2$. Fig. <ref> (f) gives GDD estimates for different settings of the translation stage which effectively changes $L$ in eq. (<ref>). GDD error taken as a standard deviation of the linear fit slope, seems to be in the range of 1%-2%; however, already a higher variance can be observed by slightly altering the signal filtering parameters. Hence, realistically we estimate the true GDD error in the range of 5-10%, with higher errors for lower GDDs. In this regime, a more precise 2-dimensional SPP could in principle yield better estimates [52].

Optical measurement of the EOM driving waveform. (a) Spectra of a single pulse passing subsequently through the stretcher and the EOM for the attenuation of 5 and different RF delays $t_\mathrm{RF}$. (b) Centroid $\Delta f_0$ and full-width-half-maximum (FWHM) of Gaussians fitted to each collected spectrum. (c)-(e) Phase modulation of the EOM $\varphi_\mathrm{EOM}$ retrieved by integrating the centroid positions. (c) For a single attenuation of 5. (d) (solid curve) For a single RF delay of $t_\mathrm{RF}=\SI{60}{\pico\second}$. (d) (dashed curve) chirp rate $K$ from a parabola fit to the waveform maximum for a given attenuation level.

§.§ RF line and EO modulation

The radio-frequency (RF) signal driving the EO modulator (iXblue NIR-MPX800-LN-20) is produced with a photodiode (PD) (Hamamatsu Photonics G4176-03 + bias tee Mini-circuits ZX85-12G-S+ set to 10) excited with the origin femtosecond pulses, preamplified (low noise Mini-circuits ZX60-06183LN+ and Mini-circuits  ZX60-183-S+), filtered ($6-18$ band pass filter Mini-circuits ZBSS-12G-S+), amplitude-controlled with a programmable attenuator (6-bit, 0.5 LSB, Analog Devices ADRF5720), and finally, power-amplified with a 3 amplifier (Mini-circuits ZVE-3W-183+). The exact RF chain sequence is depicted in Fig. <ref>.

Delay matching between the optical pulse and the RF driving signal is twofold: via a motorized optical delay line for the PD driving pulse (Delay line) and via a ca. 1 long manual quadruple-pass delay line placed after the Pulse Splitter (not shown).

Conveniently, the RF driving pulse can be temporarily characterized by observing the spectral shift of a single pulse sent through the stretcher-EOM combination. The spectral shift $\Delta f_0$ is proportional to the temporal phase gradient

\begin{equation}
\partial_t\varphi_\mathrm{EOM}(t) = 2 \pi \Delta f_0.
\end{equation}

Hence, estimating the centroid  $\Delta f_0$ of measured spectra for a series of RF delays $t_\mathrm{RF}$ and integrating $\Delta f_0 (t_\mathrm{RF})$ over the delay $t_\mathrm{RF}$ we get the EOM-induced phase $\varphi_\mathrm{ROM}(t)$. To this end, such a measurement for the RF driving waveform has been depicted in Fig. <ref>.

The chirp rate $K$ can be directly retrieved by fitting a parabola to the waveform extrema.
Otherwise, a theoretical prediction, obtained by series-expanding a cosine, reads:

\begin{equation}
K = \frac{1}{2\pi^3}\frac{V_\pi}{f_m^2 V_\mathrm{pp}},
\end{equation}

where $V_\pi\;[\mathrm{V}]$ is the voltage on EOM for a $\pi$ phase shift, $V_\mathrm{pp}$ is the applied peak-to-peak voltage, and $f_m$ is the modulation frequency
In our case $V_\pi=\SI{4}{\volt}$, $V_\mathrm{pp}\approx\SI{19.5}{\volt}$, and $f_m\approx\SI{15}{\giga\hertz}$. Maximal $V_\mathrm{pp}\approx\SI{34.6}{\volt}$ limited by the 3 power amplifier, and RF power dissipation in the EOM.

Importantly, the ability to perform arbitrary-angle FRT without distortion relies on a precise RF amplitude control with a minimal introduced phased shift. In our implementation, it is warranted by a carefully selected variable attenuator. Even though its nominal relative phase shift is ca. 20 at 15 between minimal 0 and maximal 31.5 attenuation states (linearly increasing through intermediate states), we experimentally find the net phase shift much lower. It is visible in Fig. <ref> (e) where at $t_\mathrm{RF}\approx\SI{60}{\pico\second}$ the $\arg\max\varphi_\mathrm{EOM}$ stays constant regardless of attenuation. Conveniently, the attenuation level is all-digitally controlled with a Bluepill STM32F103C8T6 board directly selecting the attenuation state of the variable attenuator with a 6-bit range and a least-significant-bit (LSB) step of 0.5.

§.§ Pulse preparation and FRT bandwidth

(left axis) Bounds (shaded region) on the initial pulse bandwidth $\Delta_f$ for a range of spectral GDD values $\Phi_2$, assuming a temporal aperture of $D_t=\SI{24.3}{\pico\second}$. (right axis) FRT angle $\alpha$ for a chirp rate of $K=\SI{6.80e-2}{\per\square\pico\second}$ and a range of $\Phi_2$.

The bandwidth of EO FRT is mainly limited by the modulation frequency $f_m$ of the EOM which determines the temporal aperture of the time lens. In our case $f_m\approx\SI{15}{\giga\hertz}$.

Assuming $5\%$ accuracy of the parabolic approximation, we get a full-width temporal aperture of ca. $\mathcal{D}_t=0.34\times f_m^{-1}\approx\SI{24.3}{\pico\second}$.
Noticeably $\mathcal{D}_t$ applies after the first stretcher, hence the pulse duration is limited after accommodating for elongation due to $\Phi_2$. Let us assume a Fourier-limited Gaussian pulse with temporal intensity full-width-half-maximum $\Delta_t$.
The elongated pulse will have a width of [50]

\begin{equation}
\Delta^{\mathrm{post}}_t  = \Delta_t\sqrt{1+\left(4\log 2 \frac{\Phi_2}{\Delta^2_t}\right)^2}.
\end{equation}

We require $\Delta_f^\mathrm{post}\leq\mathcal{D}_t$. For Fourier-limited pulses $\Delta_t \Delta_\omega = 4 \log 2$, hence we get the following bounds for the initial pulse bandwidth in terms of the spectrum (intensity) full-width-half-maximum

\begin{align}
& \Delta_f^{(-)}  \leq \Delta_f \leq \Delta_f^{(+)},\\% &&\\
& \Delta_f^{(\pm)} = \frac{\mathcal{D}_t}{2\pi\sqrt{2} \Phi_2} \sqrt{1\pm \sqrt{1- \left(\frac{8 \Phi_2 \log 2}{\mathcal{D}_t^2}\right)^2}}.%  &&
\end{align}

Note the limit of $\Delta_f^{(-)}$ for small $\Phi_2$:

\begin{equation}
\lim_{\Phi_2\to0}\Delta_f^{(-)} = \frac{2\log 2}{ \pi \mathcal{D}_t},
\end{equation}

which just states that the initial pulse must fit in the temporal aperture. Similarly, for the range of bandwidths, we get asymptotically:

\begin{equation}
\Delta_f^{(+)}-\Delta_f^{(-)}\sim \frac{1}{2\pi}\frac{\mathcal{D}_t}{\Phi_2}
\end{equation}

Conversely, for large $\Phi_2$ such that
$\xi\coloneqq 1- \left(8 \Phi_2 \log 2/\mathcal{D}_t^2\right)^2\xrightarrow{}0$
we get:

\begin{equation}
\Delta_f^{(+)}-\Delta_f^{(-)}\xrightarrow{}\frac{2 \sqrt{2}  \log (2)}{\pi }\times\frac{\sqrt{\xi}}{d}.
\end{equation}

The bounds on the initial pulse bandwidth have been depicted in Fig. <ref> together with the FRT angle, as given by Eq. (<ref>), for a fixed chirp rate $K=\SI{6.80e-2}{\per\square\pico\second}$.
For instance, taking
$\Phi_2 = \SI{15.5}{\pico\second\squared}$, we get $\SI{18.2}{\giga\hertz}\leq\Delta_f\leq\SI{248.9}{\giga\hertz}$.

Lower $\Phi_2$ presents a wider bandwidth, yet for a given angle of FRT would require a higher $K$, which in turn for a fixed $f_m$ needs higher power on EOM or a smaller $V_\pi$.
Nevertheless, within the limitations of the current setup, $\Phi_2$ can be increased several times (just by increasing the relatively small distance $L$), trading off the maximal bandwidth $\Delta f$ for increased FRT angle $\alpha$.

In the experiment, we use a folded 4f setup, illustrated in Fig. <ref>, to carve pulses with ca. $\Delta_f\approx\SI{74}{\giga\hertz}$. The setup involves a first-order diffraction on a grating, far-field imaged onto a rectangular 1-dimensional aperture with regulated width. Immediately after the aperture a mirror is placed facilitating backpropagation through the setup. The vertical offset of the imaging lens spatially separates the returning beam.

§.§ Spectrometers

In the strong-light regime for the spectrally-resolved intensity detection, we employ a custom-made grating spectrometer in a second-order double-pass configuration, described in detail in ref. [53].

Whereas for the single-photon-level measurements, we use a similar grating spectrometer (1200ln, second-order, double-pass) with a custom single-photon camera, described in detail in refs [6, 5], as the detector. A collimated beam with ca. 10 diameter impinges onto the diffraction grating with the incidence angle $\theta_i$ of ca. 67 and a diffraction angle $\theta_d$ close to 90. A vertical-shift retro-reflector routes the $m=2$ diffracted order back onto the grating, vertically-shifted by ca. 10. A diffracted beam from the second pass is separated via a D-shaped mirror, followed by a $f=\SI{400}{\milli\meter}$ lens and a single-photon camera placed in its focal range. The single-photon camera is based on a fast CMOS sensor (Luxima LUX2100) and custom FPGA electronics (Xilinx Zynq-7020, Z-turn board), with an image intensifier (II, Hamamatsu V7090D71G262) in the Chevron configuration, for single-photon sensitivity. The camera collects $50\times400$ pixel frames at $\SI{2e4}{}$ frames per second, with II gating time of $\SI{20}{\micro\second}$ . A camera pixel corresponds to a frequency step of 1.67, calibrated independently with a series of interferometric fringe-density measurements for a pair of temporarily separated pulses with varying, known separation.
The theoretical frequency resolution $\delta f$ of the single-photon spectrometer is limited by the diffraction grating and reads [54]:

\begin{equation}
\delta f = 2\times \frac{c}{W |\sin{\theta_i}+\sin{\theta_d}|} \leq \frac{c}{W},
\end{equation}

where $\theta_i$, $\theta_d$ are taken positive on the same side of the grating normal, $W\approx\SI{19.2}{\milli\meter}$ is the diameter of the grating area covered by the beam (including elongation due to the angle of incidence and diffraction), $c$ is the speed of light, and the factor of $2$ comes from a double-pass configuration.
In our case $\delta f \approx \SI{8}{\giga\hertz}$.

The measured efficiency of the single-photon spectrometer was $\approx4\times10^{-5}$.

§ CONCLUSION

We have demonstrated a scalable, single-photon-level-compatible method for performing an arbitrary-angle Fractional Fourier Transform in the Time-Frequency domain. Based on an electro-optic time lens, our FRT implementation avoids the optical noise inherent to the solutions based on non-linear processes, hence remains compatible with the single-photon-level quantum light.
A tailored RF line with a carefully chosen variable attenuator provides an all-electronic, fast, and precise control over the FRT angle. Furthermore, this FRT implementation is based on optical devices with miniaturized on-chip or fiber-based equivalents, promising excellent scalability.

The single-photon-level compatibility, all-electronic control, and the potential for miniaturization make the FRT a suitable building block for TF-domain quantum optical protocols with prominent applications in mode-sorting, super-resolution imaging, or synthesis of arbitrary unitaries.
Finally, it may find applications not only in the quantum domain, in the most advanced devices such as quantum time-frequency processor [55, 56], but also in engineering more complex and capable solutions for classical light control for the purpose of various mode-division multiplexing protocols [57].

Fundacja na rzecz Nauki Polskiej (MAB/2018/4 “Quantum Optical Technologies”); European Regional Development Fund; Narodowe Centrum Nauki (2021/41/N/ST2/02926);
The “Quantum Optical Technologies” project is
carried out within the International Research Agendas programme of the
Foundation for Polish Science co-financed by the European Union under the
European Regional Development Fund.
This research was funded in whole  or in part by National Science Centre, Poland 2021/41/N/ST2/02926.
ML was supported by the Foundation for Polish Science (FNP) via the START scholarship.
We would like to thank K. Banaszek and W. Wasilewski for the generous support.

The authors declare no conflicts of interest.

Data availability
Data for figures <ref>, <ref> and <ref>  has been deposited at [58] (Harvard Dataverse).

[1]
B. Li, B. Yuan, C. Chen, X. Xiang, R. Quan, R. Dong, S. Zhang, and R.-B. Jin,
Spectrally resolved two-photon interference in a modified
hong–ou–mandel interferometer, Opt. Laser
Technol. 159, 109039 (2023).

[2]
Y. Zhang, D. England, A. Nomerotski, and B. Sussman, High speed
imaging of spectral-temporal correlations in hong-ou-mandel interference,
Opt. Express 29, 28217–28227 (2021).

[3]
F. Albarelli, E. Bisketzi, A. Khan, and A. Datta, Fundamental limits
of pulsed quantum light spectroscopy: Dipole moment estimation,
Phys. Rev. A 107, 062601 (2023).

[4]
Y. He, Z. Liu, C. Ott, A. N. Pfeiffer, S. Sun, M. B. Gaarde, T. Pfeifer, and
B. Hu, Resonant perfect absorption yielded by zero-area pulses,
Phys. Rev. Lett. 129, 273201 (2022).

[5]
M. Lipka and M. Parniak, Single-photon hologram of a zero-area pulse,
Phys. Rev. Lett. 127, 163601 (2021).

[6]
M. Lipka and M. Parniak, Fast imaging of multimode
transverse–spectral correlations for twin photons,
Opt. Lett. 46, 3009–3012 (2021).

[7]
T.-S. Yang, Z.-Q. Zhou, Y.-L. Hua, X. Liu, Z.-F. Li, P.-Y. Li, Y. Ma, C. Liu,
P.-J. Liang, X. Li, Y.-X. Xiao, J. Hu, C.-F. Li, and G.-C. Guo,
Multiplexed storage and real-time manipulation based on a multiple
degree-of-freedom quantum memory, Nat. Commun.
9, 3407 (2018).

[8]
H.-H. Lu, N. B. Lingaraju, D. E. Leaird, A. M. Weiner, and J. M. Lukens,
High-dimensional discrete fourier transform gates with a quantum
frequency processor, Opt. Express 30,
10126–10134 (2022).

[9]
B. Brecht, D. V. Reddy, C. Silberhorn, and M. G. Raymer, Photon
temporal modes: A complete framework for quantum information science,
Phys. Rev. X 5, 041017 (2015).

[10]
J. M. Donohue, V. Ansari, J. ŘŘehá čček, Z. Hradil, B. Stoklasa,
M. Paúr, L. L. Sánchez-Soto, and C. Silberhorn, Quantum-limited
time-frequency estimation through mode-selective photon measurement,
Phys. Rev. Lett. 121, 090501 (2018).

[11]
M. Shah and L. Fan, Frequency superresolution with spectrotemporal
shaping of photons, Phys. Rev. Appl. 15,
034071 (2021).

[12]
C. Joshi, B. M. Sparkes, A. Farsi, T. Gerrits, V. Verma, S. Ramelow, S. W. Nam,
and A. L. Gaeta, Picosecond-resolution single-photon time lens for
temporal mode quantum processing, Optica
9, 364–373 (2022).

[13]
N. Fabre, G. Maltese, F. Appas, S. Felicetti, A. Ketterer, A. Keller,
T. Coudreau, F. Baboux, M. I. Amanti, S. Ducci, and P. Milman,
Generation of a time-frequency grid state with integrated biphoton
frequency combs, Phys. Rev. A 102, 012607

[14]
N. Fabre, A. Keller, and P. Milman, Time and frequency as quantum
continuous variables, Phys. Rev. A 105,
052429 (2022).

[15]
M. Karpiński, M. Jachura, L. J. Wright, and B. J. Smith,
Bandwidth manipulation of quantum light by an electro-optic time
lens, Nat. Photonics 11, 53–57 (2017).

[16]
F. Sośnicki, M. Mikołajczyk, A. Golestani, and M. Karpiński,
Interface between picosecond and nanosecond quantum light pulses,
Nat. Photonics  (2023).

[17]
M. A. Foster, R. Salem, Y. Okawachi, A. C. Turner-Foster, M. Lipson, and A. L.
Gaeta, Ultrafast waveform compression using a time-domain
telescope, Nat. Photonics 3, 581–585

[18]
M. A. Foster, R. Salem, D. F. Geraghty, A. C. Turner-Foster, M. Lipson, and
A. L. Gaeta, Silicon-chip-based ultrafast optical oscilloscope,
Nature 456, 81–84 (2008).

[19]
V. J. Hernandez, C. V. Bennett, B. D. Moran, A. D. Drobshoff, D. Chang,
C. Langrock, M. M. Fejer, and M. Ibsen, 104 mhz rate single-shot
recording with subpicosecond resolution using temporal imaging,
Opt. Express 21, 196–203 (2013).

[20]
P. Kukura, D. W. McCamant, and R. A. Mathies, Femtosecond stimulated
raman spectroscopy, Annu. Rev. Phys. Chem.
58, 461–488 (2007).

[21]
M. E. Fermann, A. Galvanauskas, and G. Sucha, Ultrafast lasers:
technology and applications, vol. 80 (CRC Press, 2002).

[22]
K. Sugioka and Y. Cheng, Ultrafast lasers—reliable tools for
advanced materials processing, Light Sci. Appl.
3, e149 (2014).

[23]
R. de Vivie-Riedle and U. Troppmann, Femtosecond lasers for quantum
information technology, Chem. Rev. 107,
5082–5100 (2007).

[24]
A. Kutay, H. Ozaktas, O. Ankan, and L. Onural, Optimal filtering in
fractional fourier domains, IEEE Trans. Signal
Process. 45, 1129–1143 (1997).

[25]
B. Hennelly and J. T. Sheridan, Optical image encryption by random
shifting in fractional fourier domains, Opt. Lett.
28, 269–271 (2003).

[26]
X. Ouyang and J. Zhao, Orthogonal chirp division multiplexing,
IEEE Trans. Commun.. 64, 3946–3957 (2016).

[27]
Q. Han, W. Li, and M. Yang, An optical waveform pre-distortion method
based on time domain fractional fourier transformation,
Optics Communications 284, 660–664 (2011).

[28]
M. Cheng, L. Deng, H. Li, and D. Liu, Enhanced secure strategy for
electro-optic chaotic systems with delayed dynamics by using fractional
fourier transformation, Opt. Express 22,
5241–5251 (2014).

[29]
Y. Zhou, M. Mirhosseini, D. Fu, J. Zhao, S. M. Hashemi Rafsanjani, A. E.
Willner, and R. W. Boyd, Sorting photons by radial quantum number,
Phys. Rev. Lett. 119, 263602 (2017).

[30]
M. Brunel, S. Coetmellec, M. Lelek, and F. Louradour, Fractional-order
fourier analysis for ultrashort pulse characterization,
J. Opt. Soc. Am. A 24, 1641–1646 (2007).

[31]
C. Cuadrado-Laborde, A. Carrascosa, A. Díez, J. L. Cruz, and M. V. Andres,
Photonic fractional fourier transformer with a single dispersive
device, Opt. Express 21, 8558–8563

[32]
C. Schnébelin and H. G. de Chatellus, Agile photonic fractional
fourier transformation of optical and rf signals,
Optica 4, 907–910 (2017).

[33]
B. Niewelt, M. Jastrz ębski, S. Kurzyna,
J. Nowosielski, W. Wasilewski, M. Mazelanik, and M. Parniak,
Experimental implementation of the optical fractional fourier
transform in the time-frequency domain, Phys. Rev.
Lett. 130, 240801 (2023).

[34]
A. W. Lohmann and D. Mendlovic, Fractional fourier transform: photonic
implementation, Appl. Opt. 33, 7661–7664

[35]
R. Salem, M. A. Foster, A. C. Turner, D. F. Geraghty, M. Lipson, and A. L.
Gaeta, Optical time lens based on four-wave mixing on a silicon
chip, Opt. Lett. 33, 1047–1049 (2008).

[36]
J. Zhou, C. Guo, and R. Wang, Fractional fourier transform based on a
temporal gradient-index lens, Opt. Continuum
1, 1418–1424 (2022).

[37]
M. T. Kauffman, W. C. Banyai, A. A. Godil, and D. M. Bloom,
Time‐to‐frequency converter for measuring picosecond optical
pulses, Appl. Phys. Lett. 64, 270–272

[38]
V. Namias, The Fractional Order Fourier Transform and its Application
to Quantum Mechanics, IMA J. Appl. Math.
25, 241–265 (1980).

[39]
A. W. Lohmann, Image rotation, wigner rotation, and the fractional
fourier transform, J. Opt. Soc. Am. A 10,
2181–2186 (1993).

[40]
H. M. Ozaktas and D. Mendlovic, Fractional fourier optics,
J. Opt. Soc. Am. A 12, 743–751 (1995).

[41]
E. Sejdić, I. Djurović, and L. Stanković, Fractional
fourier transform as a signal processing tool: An overview of recent
developments, Signal Process. 91,
1351–1369 (2011). Fourier Related Transforms for Non-Stationary Signals.

[42]
J. Paye, The chronocyclic representation of ultrashort light pulses,
IEEE J. Quantum Electron. 28, 2262–2273

[43]
R. Salem, M. A. Foster, and A. L. Gaeta, Application of
space–time duality to ultrahigh-speed optical signal processing,
Adv. Opt. Photon. 5, 274–317 (2013).

[44]
V. Torres-Company, J. Lancis, and P. Andrés, Chapter 1 - space-time
analogies in optics,  (Elsevier, 2011), pp. 1–80.

[45]
B. Kolner, Space-time duality and the theory of temporal imaging,
IEEE Journal of Quantum Electronics 30,
1951–1963 (1994).

[46]
G. Agrawal, Chapter 2 - pulse propagation in fibers, in
Nonlinear Fiber Optics,  (Academic Press, Boston, 2013), pp. 27–56,
5th ed.

[47]
C. Bennett and B. Kolner, Aberrations in temporal imaging,
IEEE J. Quantum Electron. 37, 20–32

[48]
O. Martinez, J. Gordon, and R. Fork, Negative group-velocity
dispersion using refraction, JOSA A 1,
1003–1006 (1984).

[49]
M. Lai, S. T. Lai, and C. Swinger, Single-grating laser pulse
stretcher and compressor, Appl. Opt. 33,
6985–6987 (1994).

[50]
A. M. Weiner, Dispersion and Dispersion Compensation (John Wiley &
Sons, Ltd, 2009), chap. 4, pp. 147–197.

[51]
C. Sainz, P. Jourdian, R. Escalona, and J. Calatroni, Real time
interferometric measurements of dispersion curves,
Opt. Commun. 110, 381–390 (1994).

[52]
A. Kovács, K. Osvay, G. Kurdi, M. Görbe, J. Klebniczki, and Z. Bor,
Dispersion control of a pulse stretcher–compressor system with
two-dimensional spectral interferometry, Appl. Phys.
B 80, 165–170 (2005).

[53]
S. Kurzyna, M. Jastrzębski, N. Fabre, W. Wasilewski, M. Lipka, and M. Parniak,
Variable electro-optic shearing interferometry for ultrafast
single-photon-level pulse characterization, Opt.
Express 30, 39826–39839 (2022).

[54]
E. G. Loewen and E. Popov, Diffraction gratings and applications (Marcel
Dekker, Inc., New York, 1997).

[55]
H.-H. Lu, J. M. Lukens, N. A. Peters, O. D. Odele, D. E. Leaird, A. M. Weiner,
and P. Lougovski, Electro-optic frequency beam splitters and
tritters for high-fidelity photonic quantum information processing,
Phys. Rev. Lett. 120, 030502 (2018).

[56]
A. J. Pizzimenti, J. M. Lukens, H.-H. Lu, N. A. Peters, S. Guha, and C. N.
Gagatsos, Non-gaussian photonic state engineering with the quantum
frequency processor, Phys. Rev. A 104,
062437 (2021).

[57]
J. Schröder, M. A. F. Roelens, L. B. Du, A. J. Lowery, S. Frisken, and
B. J. Eggleton, An optical fpga: Reconfigurable simultaneous
multi-output spectral pulse-shaping for linear optical processing,
Opt. Express 21, 690–697 (2013).

[58]
M. Lipka and M. Parniak, Data for: Ultrafast Electrooptic
Time-Frequency Fractional Fourier Imaging at the Single-Photon Level,
Harvard Dataverse, <https://doi.org/10.7910/DVN/OCIPMC> (2023).