11institutetext: Aselsan Research, Ankara, Turkey
11email<EMAIL_ADDRESS>
# XCAT - Lightweight Quantized Single Image Super-Resolution Using
Heterogeneous Group Convolutions and Cross Concatenation

Mustafa Ayazoglu  Bahri Batuhan Bilecen

###### Abstract

We propose a lightweight, single-image super-resolution mobile device network
named XCAT, and introduce Heterogeneous Group Convolution Blocks with Cross
Concatenations (HXBlock). The heterogeneous split of the input channels to the
group convolution blocks reduces the number of operations, and cross
concatenation allows for information flow between the intermediate input
tensors of cascaded HXBlocks. Cross concatenations inside HXBlocks can also
avoid using more expensive operations like 1x1 convolutions. To further
prevent expensive tensor copy operations, XCAT utilizes non-trainable
convolution kernels to apply upsampling operations. Designed with integer
quantization in mind, XCAT also utilizes several techniques in training, like
intensity-based data augmentation. Integer quantized XCAT operates in real-
time on Mali-G71 MP2 GPU with 320ms, and on Synaptics Dolphin NPU with 30ms
(NCHW) and 8.8ms (NHWC), suitable for real-time applications.

###### Keywords:

Single image super-resolution, quantization, group convolutions, mobile AI

##  1 Introduction

|
---
|
---
|
---
|
---

(a) HR

(b) Portion
of HR

(c) XCAT
(320ms)

(d) ESPCN
[32](363ms)

(e) ABPN
[8](600ms)

(f) FSRCNN
[7](485ms)

(g) XLSR
[2](370ms)

Figure 1: Comparative results of UINT8 quantized models on DIV2K (Val image:
890). (d) and (f) yield both visually and numerically worse results than the
rest. Visual results of (c), (e), and (g) are indistinguishable to the naked
eye; however, XCAT runs the fastest

Super-resolution (SR) is an extensively studied computer vision problem that
aims to generate higher resolution (HR) image(s) given lower resolution (LR)
image(s). In single image super-resolution (SISR), a single image; and in
multi-image super-resolution (MISR), multiple images are utilized to generate
a single HR image. In either case, image super-resolution is an ill-posed
problem, since there is no unique solution. This ill-posed problem has been
attempted to be solved via classical methods [9] and deep-learning based
methods [32][6]; however, many new methods based on deep-learning are still
being developed, most of which purely focus on data fidelity.

However, for the SR method to be practically applicable, the runtime is as
important as the method’s PSNR performance. Due to its practical importance,
recent literature studies on SR focus on deployability, runtime, quantization,
and efficiency, as well as PSNR of the method [2][8][3][13][26]. Yet,
achieving real-time performance with satisfactory visual quality during the
quantization process further complicates the problem and careful network
design is needed.

Figure 2: Proposed HXBlock (left) vs. GBlock [2] (right). Dashed rectangles
represent tensors. Group convolutions inside HXBlock are heterogeneous
compared to GBlock’s. HXBlock uses cross concatenation (a) instead of depth
wise 1x1 convolutions (b), which provides information flow through different
convolutional kernels when HXBlocks are cascaded. For HXBlock, using (a)
results in a significant runtime performance increase in return of a small
PSNR drop compared to using (b)

In this study, we focus on the efficiency and mobile deployment, in the scope
of _Mobile AI & AIM 2022 Real-Time Image Super-Resolution Challenge_ [14]. Our
model, named XCAT, is a SISR network incorporating the proposed HXBlock and
modifying both new and existing techniques from the literature for providing a
quantization-aware, robust, real-time performance model, suitable for mobile
devices.

Our work makes the following contributions:

  1. 1.

HXBlocks, which are heterogeneous grouped convolutions with cross
concatenation layers for allowing information flow with almost no
computational cost through different convolutional kernels of the group.
Relevant studies done on HXBlocks have shown that they can be replaced with
traditional group convolutions with little sacrifice from PSNR, but a
significant gain in run time performance.

  2. 2.

A method for nearest neighborhood up sampling method with fixed 2D
convolutional kernels to replace expensive tensor copy operations on mobile
devices, which makes the model robust to quantization.

  3. 3.

An efficient, mobile device friendly, single image super-resolution network
named XCAT.

##  2 Related Works

DNN-based single image super-resolution. First deep-learning-based SISR
algorithm was proposed by Dong et. al. as SRCNN [6]. Later, as a speed
improvement on SRCNN, FSRCNN [7] was developed; which introduced a
deconvolutional layer at the end of the network, replaced ReLU with a PReLU
activation layer, and reformulated SRCNN by adopting smaller filter sizes but
more mapping layers. Shi et. al. [32] introduced a novel, efficient sub-pixel
convolutional layer (also known as depth to space), which is actually widely
used in many fast SR networks right now. VDSR [19], EDSR [27], and WDSR [39]
continued the development of deep-learning-based SR by increasing the number
of parameters, in exchange for accuracy with speed.

With the recent developments in computer vision and deep learning, concepts
like attention mechanism [30], generative adversarial networks [25][35],
recursive & residual networks [20][33][1], and distillation layers
[12][11][3][29] also started to take part inside SR network architectures.
GANs and networks with attention mechanism mostly generate a high-quality SR
image by sacrificing speed, whereas RNNs and distilling networks try to
decrease the computational load.

Group convolutions. Group convolutions consist of groups of multiple
convolutional kernels placed within the same layer. The motivation behind
group convolutions emerged with AlexNet [22], desiring to distribute the model
over multiple GPUs to overcome hardware limitations. Later on, besides the
increase in speed in AlexNet, group convolutions are also observed to improve
classification accuracy when groups are accompanied by skipped connections
with ResNetX [36]. ShuffleNet [41] introduced shuffling the intermediate
tensors between group convolution blocks to increase feature extraction.
DeepRoots [16] and more recent studies use different convolutional kernels
inside groups, such as 1x1 depth wise convolutions [31] and dilated
convolutions [38][42]. In addition, unitary [43] and interleaved [40] group
convolutions also offer different perspectives on how to extract various
features from input images. Usage of group convolutions due to their
efficiency on super-resolution problems is also present [2][3][18].

Model Optimization. Hardware limitations and specifications may require the
model to be optimized via different techniques, such as quantization, pruning,
clustering, network architecture search (NAS), and many more. Quantization
refers to converting floating point values to integers, hence decreasing
memory usage and computational cost when re-accessing and/or updating the
mentioned values, at a cost of decreasing the precision. Quantization is
particularly useful on neural network models since it can decrease inference
times without sacrificing much inference accuracy if done correctly [17].
Models also can be quantized after quantization-aware training in floating
point precision [34][21], as well as training the network directly with low
precision multiplications [5]. Removing layers from a model having a minor
effect on inference is called pruning [37][10], and clustering is the method
of decreasing the number of unique weights by grouping weights and assigning
the centroid values for each group. All of these methods try to decrease
processor utilization or memory usage, or both. Besides optimizing an existing
network structure, finding the most possible optimal network structure in
search space is also a study area, known as network architecture search [44].

##  3 Method

In this section, XCAT is defined by its overall architecture and its
components. Details about the training techniques and the quantization
procedure are explained thoroughly as well.

###  3.1 XCAT’s Architecture

Figure 3: Network structure of XCAT. Numbers on arrows denote channel numbers,
and numbers inside blocks represent kernel sizes of convolutional blocks.
Dashed blocks represent tensors, whereas non-dashed ones represent operators
like convolution and normalization. Convolution with 1x1 identity kernel
performs the upsampling method visualized in Fig. 4. XCAT has m HXBlocks,
where m=2 for this study

As seen from Fig. 3, XCAT consists of 3 individual convolutional layers with
trainable 3x3 kernels, a single 1x1 convolutional layer with a fixed ”identity
kernel” to simulate nearest neighborhood upsampling operation, m HXBlocks, a
tensor addition layer, followed by a depth to space (D2S) layer and a clipped
ReLU activation layer. Input and output of XCAT are LR and SR, respectively,
where SR has x3 resolution of LR.

Each key component of XCAT will be detailed with their reasoning:

Group convolutions with heterogeneous filter groups and varying kernels. Group
convolutions, which include multiple convolutional kernels per layer, are
known to be able to extract and learn more varying features compared to a
single kernel [23]. XCAT inherits this idea of group convolution blocks to
replace single-layered convolutions in a repeated manner. However, as opposed
to initial approaches [36], convolutional layers inside the group convolution
blocks in XCAT have different layer dimensions and different kernel sizes
(Fig. 3). This allows to pass the same source information between different
convolutional layers and allows for less computationally demanding feature
extraction. The input tensor is split into two parts in channel dimension, one
processed by 1x1 and the other by 3x3 convolutional kernels. 1x1 convolution
”blends” the point-wise information from previous HXBlocks and extracts inter-
channel features, whereas 3x3 convolution considers in-channel correlation as
well. In addition, a relevant study of Lee et al.’s [24] logarithmic filter
groups in shallow CNNs shows the positive effect of dividing group convolution
input tensors unevenly.

Cross concatenation. First group convolutions in AlexNet [22] ended with max
pooling layers. However, group convolution designs such as DeepRoots [16]
started utilizing low-dimensional embeddings (like 1x1 convolutions) at the
end of the groups, with the inspiration taken from Lin. et. al. [28] and
Cogswell et. al. [4]. This was done to decrease the computational cost and
number of parameters without compromising accuracy. Later on, several
efficiency-oriented SR networks like XLSR [2] and IMDeception [3] also
utilized group convolutions ending with 1x1 depth-wise convolutions.

In XCAT, instead of using 1x1 depth wise convolutions for increasing the
spatial receptive field of each output of a group convolution block, the
output tensor of each group convolution block is cross concatenated. The
inspiration came from ShuffleNet [41] and Swin Transformer [29]; where channel
shuffling and convolutional layers are inserted between group convolutions in
the former, and window partitions are cyclic shifted to enable information
flow between windows in the latter. Each cross concatenation in XCAT
corresponds to a circular shift of one-fourth of the input tensor (Fig. 3).
This cyclic procedure allows the information to pass through from 1x1 and 3x3
convolutions inside XCAT’s group convolution blocks, hence having more chance
for feature extraction.

It is worthy to note that ShuffleNet [41]’s channel shuffling is similar to
the XCAT’s; however, XCAT has a cross concatenation operation represented with
cyclic shifts, whereas ShuffleNet has a shuffle operation dividing and
reorganizing tensors into many small partitions. This reorganization operation
is reflected onto the target device (Synaptics Dolphin NPU) as reshape and
transpose operations, which take much longer to process compared to XCAT’s
simpler yet effective approach.

During the experiments, it is observed that replacing cross concatenation
operations with 1x1 convolutions in XCAT increases the run time per frame, but
does not increase the PSNR test score considerably, making it less practical
for mobile networks.

Depth to space (D2S) operation. Shi et al. [32]’s pixel shuffling (depth to
space operator) is inserted at the end of the network, which aims to implement
sub-pixel convolutions in an efficient manner and is proven to increase PSNR
score in super-resolution problems in many studies.

Nearest neighborhood upsampling with fixed kernel convolutions. We observed
that providing the low-resolution input image to D2S with accompanying feature
tensors increases the robustness, as opposed to only providing the extracted
feature tensors to D2S. With this motivation, XCAT also adds repeated input
image tensors (where each channel of the input image is repeated 9 times) to
feature tensors and provides them to D2S. From the perspective of D2S, this
operation is equivalent to the nearest neighborhood upsampling.

A relevant study done by Du et. al. named ABPN [8] also utilizes the nearest
neighborhood upsampling to be fed to the D2S block. However, it uses tensor
copy operations while repeating and concatenating the input image in the
upsampling process, which are indeed expensive for mobile devices. For a
better alternative, a convolutional layer of 3 input channels and 27 output
channels is used, with a 1x1 non-trainable kernel which is set to serve the
same purpose as a tensor copy (Fig. 4). One point to note is that when this
1x1 kernel is set as trainable, it gets affected by the quantization process
and yields lower visual quality results.

Figure 4: Tensor copy operations done with convolutions. The identity kernel
of the 1x1 convolution is set in such a way that it reproduces the input
tensor of 3 channels 9 times, generating an output tensor of 27 channels

###  3.2 Training and Quantization Details

XCAT is trained in floating point precision and quantized afterward. However,
it is trained and designed with quantization in mind, with several techniques
to avoid PSNR decrease:

Intensity-augmented training. To minimize the PSNR difference while quantizing
the FP32 model to its UINT8, intensity values of the training images are
scaled with randomly chosen constants among (1, 0.7, 0.5). We have observed
that this strategy helped with quantization and avoided signal degradation, as
stated in [2].

Clipped ReLU. As proven and explained in [2], using clipped ReLU at the end of
the network allows better quantization while keeping the performance in the
real-time range.

Representative dataset selection. TensorFlow Lite requires a representative
dataset while quantizing a floating point Keras or TensorFlow model. As a rule
of thumb, this dataset consists of entire training images. However, it is
observed that selecting a subset of all training images as the representative
dataset affects the final PSNR test score of the quantized UINT8 model
immensely. Hence, to find the most suitable representative dataset, a linear
search is applied to all DIV2K training images, generating single image
representative datasets. For each representative dataset (or rather an image),
XCAT is quantized and PSNR test scores are measured. The highest scoring
quantized XCAT model is chosen as the best.

Training details. XCAT is trained twice in floating point precision and then
quantized. Training details are as follows:

  * •

DIV2K dataset is used for the first training, and Flickr2K dataset is added
alongside for the second training (fine tuning).

  * •

Intensity, rotation, random crop, and flip augmentations are used while
setting up the dataset for both of the training. HR images are cropped to
96x96 patches.

  * •

XCAT is trained with 50 epochs and 16 batches. Each epoch contains 10000 mini-
batches.

  * •

For the first training:

    * –

Charbonnier loss is used, where $C(x)=\sqrt{(x^{2}+{\epsilon}^{2})}$ and
$\epsilon=0.1$. Charbonnier loss is the smoother version of L1-loss having
better convergence characteristics than L2-loss.

    * –

Adam optimizer is used with initial learning rate = 0.001, $\beta_{1}=0.9$,
$\beta_{2}=0.999$, $\epsilon=1e^{-8}$.

    * –

Warm-up scheduler is used: Starting from the initial learning rate, at each
epoch, the learning rate is increased up to $25e^{-4}$ until the 5th epoch.
After the 5th epoch, the learning rate is linearly decreased at the end of
each epoch, where at the last epoch it decreases to $e^{-4}$.

  * •

For the second training for fine-tuning:

    * –

Mean square error is used as the loss.

    * –

Adam optimizer is used with an initial learning rate = 0.0001, and the same
beta and epsilon parameters.

    * –

Warm-up scheduler is used again, but with the new initial learning rate, and
the maximum learning rate of $12.5e^{-4}$ instead of $25e^{-4}$.

Table 1: Different XCAT-based models and their performance. Runtime is evaluated on Mali-G71 MP2 GPU via AI Benchmark 5 [15]. Score is the metric function described in (1). Note that the Config column describes the differences among models. m is the number of HXBlocks. X/Y shows the splitting ratio, where X+Y is the total number of channels of HXBlocks. axa/bxb represents the convolutional kernels inside the group convolution blocks, where the tensor with dimension X passes through axa, and Y through bxb Model | Config |  | PSNR
---
FP32/UINT8
| Runtime
---
(ms)
Score
XCAT |  | m=2, 21/7, 1x1/3x3
---
2 stage training
29.88/29.81 | 320 | 240
A |  | m=2, 21/7, 1x1/3x3
---
1 stage training
29.85/29.79 | 320 | 233
B |  | m=2, 21/7, 1x1/3x3
---
Cross Cat $\rightarrow$ 1x1 Conv.
29.92/29.84 | 340 | 235
C |  | m=2, 21/7, 3x3/3x3
---
Cross Cat $\rightarrow$ 1x1 Conv.
30.04/29.96 | 780 | 121
D |  | m=2
---
HXBlock $\rightarrow$ 3x3 Conv.
30.04/29.97 | 770 | 124
E |  | m=4, 21/7, 1x1/3x3
---
29.98/29.89 | 370 | 232
F |  | m=4, 21/7, 1x1/3x3
---
Conv after last HXBlock:
3x3 $\rightarrow$ 1x1
29.82/29.75 | 300 | 236
G |  | m=4, 21/7, 1x1/3x3
---
Conv after last HXBlock:
removed
29.81/29.72 | 290 | 234
H |  | m=4, 16/12, 1x1/3x3
---
Conv after last HXBlock:
removed
29.87/29.76 | 300 | 238
I |  | m=4, 7/21, 1x1/3x3
---
Conv after last HXBlock:
removed
30.03/29.88 | 520 | 163
J |  | m=4, 7/21, 3x3/3x3
---
Conv after last HXBlock:
removed
30.04/29.87 | 550 | 152
K |  | m=3, 7/21, 1x1/3x3
---
Conv after last HXBlock:
removed
29.95/29.81 | 430 | 179
L |  | m=4, 16/4, 1x1/3x3
---
Conv after last HXBlock:
removed
29.61/29.45 | 205 | 228
M |  | m=4, 16/4, 1x1/3x3
---
Replaced Add with Concat
29.63/29.15 | 298 | 103
Table 2: PSNR test scores of XCAT and other algorithms with public datasets. All models are FP32. To be consistent with the rest of the algorithms; XLSR, ABPN, and XCAT’s PSNR results are calculated using Luminance (Y) channel rather than RGB channels, except for DIV2K (*We performed our own training since the pre-trained FP32 model from the authors performed poorly, around 15dB for DIV2K(Val)) Dataset | Scale | Bicubic | FSRCNN | ESPCN | XLSR |  ABPN* |  | XCAT
---
(proposed)
Set5 | x3 | 30.44 | 32.73 | 32.59 | 33.09 | 33.45 | 33.02
Set14 | x3 | 27.63 | 29.30 | 29.18 | 29.59 | 29.73 | 29.54
B100 | x3 | 27.13 | 28.26 | 28.18 | 28.45 | 28.56 | 28.42
Urban100 | x3 | 24.43 | 26.03 | 25.87 | 26.48 | 26.73 | 26.38
Manga109 | x3 | 26.87 | 30.21 | 29.70 | 31.13 | 31.47 | 31.12
DIV2K(Val) | x3 | 28.82 | 29.67 | 29.54 | 30.10 | 30.10 | 29.88
Table 3: Effect of cross concatenation versus straight concatenation (no shuffling of the tensors while concatenating), the number of HXBlocks, and the tensor divisions. All models are based on XCAT. All parameter changes are mentioned on the table, and the rest are kept the same among all models. Runtime is evaluated on Mali-G71 MP2 GPU via AI Benchmark 5 [15]. Score is the metric function described in (1). Split and kernel definitions are stated in Tab. 1 Split | Kernel |  | m
---
(# of HXBlocks)
| Cross
---
Concat
| PSNR
---
(FP32)
| PSNR
---
(UINT8)
| Runtime
---
(ms)
Score
21/7 | 1x1/3x3 | 2 | ✓ | 29.88 | 29.81 | 320 | 240
21/7 | 1x1/3x3 | 2 | $\times$ | 29.87 | 29.79 | 320 | 233
21/7 | 1x1/3x3 | 4 | ✓ | 29.98 | 29.89 | 370 | 232
21/7 | 1x1/3x3 | 4 | $\times$ | 29.96 | 29.87 | 370 | 232
21/7 | 1x1/3x3 | 8 | ✓ | 30.04 | 29.97 | 480 | 200
21/7 | 1x1/3x3 | 8 | $\times$ | 30.01 | 29.89 | 480 | 179
21/7 | 1x1/3x3 | 12 | ✓ | 30.07 | 29.97 | 580 | 165
21/7 | 1x1/3x3 | 12 | $\times$ | 30.04 | 29.72 | 580 | 117
24/8 | 1x1/3x3 | 6 | ✓ | 30.02 | 29.92 | 410 | 218
24/8 | 1x1/3x3 | 6 | $\times$ | 30.01 | 29.89 | 410 | 209
56/8 | 1x1/3x3 | 4 | ✓ | 30.08 | 30.03 | 620 | 168
56/8 | 1x1/3x3 | 4 | $\times$ | 30.04 | 29.99 | 620 | 159

##  4 Experimental Results

During the development of XCAT, many modified versions were created and
tested. Numerical results of XCAT models and the ablation study done for
HXBlocks are given in Tab. 1. Comparative visual results are given in Fig. 5.

To choose the most successful model, we used the score function in (1), which
is officially published in the competition’s evaluation criteria.

$Score=\frac{2^{2(PSNR_{(UINT8)}-30)}}{t_{(UINT8)}10^{-5}}$ (1)

Comparative study. Tab. 4 and Fig. 5 reveal that the network architecture’s
suitability for the quantization procedure plays a big role in producing high-
quality, super-resolved images. Despite XLSR and ABPN having higher PSNR FP32
scores compared to XCAT, after the quantization, all three yielded similar
visual results and closer UINT8 PSNR scores to each other.

Ablation study. In Tab. 1, increasing layer number/sizes and parameter numbers
increased the PSNR score and decreased run time performance (E-G, G-L, B-C).
Decreasing number of groups had a negative effect on PSNR; however, the
positive effect on runtime surpassed (A-E). Using (I) dynamic kernels as
opposed to not using (J) had a significant runtime boost with PSNR scores
almost unchanged. Different heterogeneous divisions of filters (G-H) are also
tried. Logically, when the input size of the 3x3 convolution layer increased,
the PSNR score also increased. However, the penalty of runtime overcame the
positive benefits of the PSNR raise. Replacing cross concatenation layers with
1x1 convolutions (XCAT-B) had the same effect as in the previous case.

Tab. 3 shows the effect of using cross concatenation instead of directly
concatenating the intermediate tensors in HXBlocks, as well as using different
tensor divisions and number of HXBlocks. It is proven that using cross-
concatenation allows for better information flow and increases the PSNR score,
as opposed to using direct concatenation. This effect is more visible when the
number of HXBlocks increase.

|
---
|
---
|
---
|
---

(a) HR

(b) HR

(c) XCAT

(d) ESPCN

(e) ABPN

(f) FSRCNN

(g) XLSR

|
---
|
---
|
---
|
---

(h) (a) HR

(i) (b) HR

(j) (c) XCAT

(k) (d) ESPCN

(l) (e) ABPN

(m) (f) FSRCNN

(n) (g) XLSR

Figure 5: Comparative results of UINT8 quantized models on DIV2KVal dataset. The proposed method is applied for (c)’s representative dataset, whereas all DIV2KVal images are used for (d) and (f)’s. (e) and (g) are the pre-trained quantized models provided by the authors. Visual results of (c), (e), and (g) are indistinguishable; however, XCAT runs faster Table 4: PSNR (dB) drops for DIV2K(Val and Test, x3) before and after quantization, number of parameters, and runtime scores (ms) on Mali-G71 MP2 via AI Benchmark 5 [15] and Synaptics NPU (*FP32 and UINT8 scores are taken from the paper. In addition, the authors’ pre-trained .tflite model gave a concatenation error on AI Benchmark 4 & 5, about the source tensor not being able to be used multiple times. Hence, the model code is altered for ABPN, where the relevant tensor is manually hard-copied and concatenated) (+Tested in NCHW format) Metric |  FSRCNN [7] |  ESPCN [32] |  XLSR [2] |  ABPN [8]* |  | XCAT
---
(proposed)
Val, FP32 PSNR | 29.67 | 29.54 | 30.10 | 30.22 | 29.88
Val, UINT8 PSNR | 18.52 | 17.50 | 29.82 | 30.09 | 29.81
$\Delta$PSNR | 11.15 | 12.04 | 0.28 | 0.13 | 0.07
Test, UINT8 PSNR | - | - | 29.58 | 29.87 | 29.67
# of parameters | 25K | 31K | 22K | 42K | 16K
Synaptics Runtime+ | - | - | 44.8 | 36.9 | 8.8
Mali Runtime | 485 | 363 | 370 | 600 | 320
Score | 0.003 | 0.061 | 210 | 188 | 240

##  5 Conclusions & Future Studies

This study proposes a lightweight, quantized single image super-resolution
network named XCAT, submitted to _Mobile AI & AIM 2022 Real-Time Image Super-
Resolution Challenge_. XCAT offers heterogeneous group convolution blocks
which includes convolutional kernels with different kernels and input & output
tensor sizes. Compared to other studies which include group convolutions
ending with 1x1 layers, cross concatenating the intermediate tensors between
group convolutions offer runtime efficiency with tolerable sacrifice from PSNR
test scores. To further increase runtime performance on mobile devices,
upsampling done by tensor copy operations by default is replaced by a 1x1
convolutional layer with a non-trainable kernel. XCAT is also shown to be
robust to quantization, with a decrease of 0.07dB from FP32 to the UINT8
model.

Comparative experimental results on slightly modified XCAT models reveal that
the design choices proposed in this study offer the model to be deployed on
mobile devices efficiently. To further prove the effectiveness of the proposed
method, XCAT is evaluated with standardized datasets in comparison to other
mobile-friendly super-resolution networks. Visual results indicate that XCAT
can produce super-resolved images nearly identical to the other slower
networks’ outputs. Although HXBlock is designed for super-resolution problems,
we believe that it can help many heavy models to facilitate running on mobile
devices.

## References

  * [1] Ahn, N., Kang, B., Sohn, K.: Fast, accurate, and, lightweight super-resolution with cascading residual network. CoRR abs/1803.08664 (2018)
  * [2] Ayazoglu, M.: Extremely lightweight quantization robust real-time single-image super resolution for mobile devices. CoRR abs/2105.10288 (2021)
  * [3] Ayazoglu, M.: Imdeception: Grouped information distilling super-resolution network (2022)
  * [4] Cogswell, M., Ahmed, F., Girshick, R., Zitnick, L., Batra, D.: Reducing overfitting in deep networks by decorrelating representations (11 2015)
  * [5] Courbariaux, M., Bengio, Y., David, J.P.: Training deep neural networks with low precision multiplications (12 2015)
  * [6] Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision – ECCV 2014. pp. 184–199. Springer International Publishing, Cham (2014)
  * [7] Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision – ECCV 2016. pp. 391–407. Springer International Publishing, Cham (2016)
  * [8] Du, Z., Liu, J., Tang, J., Wu, G.: Anchor-based plain net for mobile image super-resolution (2021)
  * [9] Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: 2009 IEEE 12th International Conference on Computer Vision. pp. 349–356 (2009)
  * [10] He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: 2017 IEEE International Conference on Computer Vision (ICCV). pp. 1398–1406 (2017)
  * [11] Hui, Z., Gao, X., Yang, Y., Wang, X.: Lightweight image super-resolution with information multi-distillation network. pp. 2024–2032 (10 2019)
  * [12] Hui, Z., Wang, X., Gao, X.: Fast and accurate single image super-resolution via information distillation network. pp. 723–731 (06 2018)
  * [13] Ignatov, A., Timofte, R., Denna, M., Younes, A., Lek, A., Ayazoglu, M., Liu, J., Du, Z., Guo, J., Zhou, X., Jia, H., Yan, Y., Zhang, Z., Chen, Y., Peng, Y., Lin, Y., Zhang, X., Zeng, H., Zeng, K., Wang, S.: Real-time quantized image super-resolution on mobile npus, mobile ai 2021 challenge: Report. pp. 2525–2534 (06 2021)
  * [14] Ignatov, A., Timofte, R., Denna, M., Younes, A., et al.: Efficient and accurate quantized image super-resolution on mobile npus, mobile ai & aim 2022 challenge: Report. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2022)
  * [15] Ignatov, A., Timofte, R., Kulik, A., Yang, S., Wang, K., Baum, F., Wu, M., Xu, L., Van Gool, L.: Ai benchmark: All about deep learning on smartphones in 2019 (10 2019)
  * [16] Ioannou, Y., Robertson, D., Cipolla, R., Criminisi, A.: Deep roots: Improving cnn efficiency with hierarchical filter groups (07 2017)
  * [17] Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., Kalenichenko, D.: Quantization and training of neural networks for efficient integer-arithmetic-only inference. pp. 2704–2713 (06 2018)
  * [18] Jain, V., Bansal, P., Kumar Singh, A., Srivastava, R.: Efficient single image super resolution using enhanced learned group convolutions (08 2018)
  * [19] Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. CoRR abs/1511.04587 (2015)
  * [20] Kim, J., Lee, J.K., Lee, K.M.: Deeply-recursive convolutional network for image super-resolution. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1637–1645 (2016)
  * [21] Krishnamoorthi, R.: Quantizing deep convolutional networks for efficient inference: A whitepaper (06 2018)
  * [22] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C., Bottou, L., Weinberger, K. (eds.) Advances in Neural Information Processing Systems. vol. 25. Curran Associates, Inc. (2012)
  * [23] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C., Bottou, L., Weinberger, K. (eds.) Advances in Neural Information Processing Systems. vol. 25. Curran Associates, Inc. (2012)
  * [24] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C., Bottou, L., Weinberger, K. (eds.) Advances in Neural Information Processing Systems. vol. 25. Curran Associates, Inc. (2012)
  * [25] Ledig, C., Theis, L., Huszar, F., Caballero, J., Aitken, A.P., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. CoRR abs/1609.04802 (2016)
  * [26] Li, Y., Zhang, K., Timofte, R., Van Gool, L., Kong, e.a.: Ntire 2022 challenge on efficient super-resolution: Methods and results (2022)
  * [27] Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. CoRR abs/1707.02921 (2017)
  * [28] Lin, M., Chen, Q., Yan, S.: Network in network (2013)
  * [29] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows (2021)
  * [30] Niu, B., Wen, W., Ren, W., Zhang, X., Yang, L., Wang, S., Zhang, K., Cao, X., Shen, H.: Single image super-resolution via a holistic attention network (2020)
  * [31] Schwarz Schuler, J.P., Romaní, S., Abdel-nasser, M., Rashwan, H., Puig, D.: Grouped pointwise convolutions reduce parameters in convolutional neural networks. Mendel 28, 23–31 (06 2022)
  * [32] Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 1874–1883 (2016)
  * [33] Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2790–2798 (2017)
  * [34] Vanhoucke, V., Senior, A., Mao, M.: Improving the speed of neural networks on cpus (01 2011)
  * [35] Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Loy, C.C., Qiao, Y., Tang, X.: Esrgan: Enhanced super-resolution generative adversarial networks (2018)
  * [36] Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5987–5995 (2017)
  * [37] Xu, S., Huang, A., Chen, L., Zhang, B.: Convolutional neural network pruning: A survey. In: 2020 39th Chinese Control Conference (CCC). pp. 7458–7463 (2020)
  * [38] Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions (05 2016)
  * [39] Yu, J., Fan, Y., Yang, J., Xu, N., Wang, Z., Wang, X., Huang, T.S.: Wide activation for efficient and accurate image super-resolution. CoRR abs/1808.08718 (2018)
  * [40] Zhang, T., Qi, G.J., Xiao, B., Wang, J.: Interleaved group convolutions. In: 2017 IEEE International Conference on Computer Vision (ICCV). pp. 4383–4392 (2017)
  * [41] Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6848–6856 (2018)
  * [42] Zhang, Z., Wang, X., Jung, C.: Dcsr: Dilated convolutions for single image super-resolution. IEEE Transactions on Image Processing 28(4), 1625–1635 (2019)
  * [43] Zhao, R., Hu, Y., Dotzel, J., De Sa, C., Zhang, Z.: Building efficient deep neural networks with unitary group convolutions. pp. 11295–11304 (06 2019)
  * [44] Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8697–8710 (2018)


# The Varied Avatars of Time-delay Interferometry

Sanjeev Dhurandhar<EMAIL_ADDRESS>Inter University Centre for Astronomy and
Astrophysics, Ganeshkhind, Pune, 411 007, India  Prasanna Joshi
<EMAIL_ADDRESS>Max Planck Institute for Gravitational Physics
(Albert-Einstein-Institute), D-30167 Hannover, Germany  Massimo Tinto
<EMAIL_ADDRESS>University of California San Diego, Center for Astrophysics
and Space Sciences, 9500 Gilman Dr, La Jolla, CA 92093, U.S.A.  Divisão de
Astrofísica, Instituto Nacional de Pesquisas Espaciais, S. J. Campos, SP
12227-010, Brazil

###### Abstract

Time-Delay Interferometry (TDI) is the data processing technique that cancels
the large laser phase fluctuations affecting the one-way Doppler measurements
made by unequal-arm space-based gravitational wave interferometers. By taking
finite linear combinations of properly time-shifted Doppler measurements,
laser phase fluctuations are removed at any time $t$ and gravitational wave
signals can be studied at a requisite level of sensitivity.

In the past, other approaches to this problem have been proposed. Recently,
matrix based approaches have been put forward; two such approaches are by
Vallisneri et al. and Tinto, Dhurandhar and Joshi. In this paper we establish
a close relationship between these approaches. In fact we show that the
matrices involved in defining the operators in the two approaches exhibit an
isomorphism and therefore in both approaches one is dealing with matrix
representations of the time-delay operators.

###### pacs:

04.80.Nn, 95.55.Ym, 07.60.Ly

##  I Introduction

In ground based detectors of gravitational waves (GWs) the arms are chosen to
be of equal length. This is because the laser phase fluctuations experience
identical delays in the arm of the interferometer and cancel at the photo-
detector where the two returning beams are made to interfere. In space-based
detectors, on the other hand, the arm-lengths are unequal and time-dependent
as each spacecraft follows a trajectory determined by celestial mechanics. As
a result it becomes impossible to maintain the distances between spacecraft
equal and constant. Time-Delay Interferometry (TDI) is required to cancel the
laser phase noise, which is many orders of magnitude above the other residual
noise sources (such as shot noise, test mass acceleration noise, etc.)
affecting the heterodyne one-way measurements. TDI entails properly delaying
and linearly combining the different data streams so that the laser phase
fluctuations are suppressed below the residual noises and GW signals may be
observed.

In the past, other approaches have been proposed to compensate for the
inequality of the arms and achieve suppression of the laser noise below the
residual noise levels. The first, which was formulated in the Fourier domain
Giampieri et al. (1996), represented the delayed one-way measurements in terms
of their Fourier transforms multiplied by corresponding phasors. This approach
was incorrect for two fundamental reasons. Firstly, it neglected the time-
evolution of the delays, which we now know needs to be accounted for to
sufficiently suppress the laser noise. Secondly (and more importantly), it
made the erroneous assumption of taking infinitely long Fourier transforms of
the delayed one-way measurements. A finite-time Fourier transform of a delayed
measurement is not equal to the product of its Fourier transform with the
delay phasor. Rather it is equal to the Fourier domain convolution of the
Fourier transform of the data with the Fourier transform of the window of
integration. This implied the existence of residual laser noise terms in the
Fourier-domain laser-noise canceling algorithm that could be neglected only by
taking 6 months or longer Fourier transforms of the measurements Tinto and
Armstrong (1999).

A much neater approach by Tinto, Estabrook and Armstrong, where the delays
were represented by derivative-like symbols, namely commas Tinto and Armstrong
(1999); Armstrong et al. (1999); Tinto et al. (2002, 2004), was applied in the
context of the LISA mission Bender and et. al. (1998); Amaro-Seoane et al.
(2017). This notation (and its understanding) facilitated the algebra of the
TDI observables and led in principle, to a plethora of such observables that
could be obtained conveniently by just linearly combining the four Sagnac
observables $\alpha,\beta,\gamma,\zeta$. This work showed that the space of
TDI was a linear object and its elements could be obtained by linearly
combining four simple basic TDI observables. This is in fact a most important
result of this approach.

An exceptionally deep insight into time-delay interferometry was obtained,
when Dhurandhar, Nayak and Vinet Dhurandhar et al. (2002) found the exact
underlying mathematical structure of the TDI space. In this approach the time-
delay operation was promoted to operators acting on data streams or operators
acting on functions of time. The operators played the role of indeterminates
in a polynomial ring and the TDI space was none other than the first module of
syzygies Becker and Weispfenning (1993). It was shown therein that the TDI
space is a module over the polynomial ring of time-delay operators and hence
pinned down the linear structure. This work laid emphasis on the operators
rather than on the functions (data streams containing laser noise) since these
were subsidiary - the function space is the carrier space. This is a similar
situation as one has in matrix representations of groups; the matrices, which
are linear maps on the carrier vector space, represent the group elements.
This is interesting from the historical point of view because the first module
of syzygies was defined by Hilbert in 1890 in a different context. It is in
fact a kernel of a homomorphism. This is exactly what one desires - its
physical significance here is that elements in a kernel map to zero. In the
current context this is the zero of the laser noise: we are looking for those
data combinations that map the laser phase noise to zero. This approach
rigorously proved that all TDI observables can be obtained as a linear
combination of the four generators $\alpha,\beta,\gamma,\zeta$ 111A module in
general does not have a basis, but has generators which span the module though
they may not be linearly independent - one may not be able to reduce the
number of generators in general, because multiplicative inverses need not
exist in a ring.. The first work Dhurandhar et al. (2002) considered constant
arm-lengths. Later more general and realistic models of LISA were considered
Rajesh Nayak and Vinet (2005); Dhurandhar et al. (2008, 2010); Shaddock et al.
(2003); Tinto et al. (2004) which increased the complexity. These approaches
with generalizations have been reviewed comprehensively in Tinto and
Dhurandhar (2021).

In fairly recent years a novel approach was adopted by Vallisneri et al.
Vallisneri et al. (2021) in which the problem has been formulated in terms of
matrices (see also Romano and Woan (2006) where principal component analysis
has been employed). In this approach, the data are discretized and a design
matrix representing the delays is defined. But since data points may be
required in between the sample points for TDI to be effective, an
interpolating scheme must be employed for fractional delays. Here also a null
space is sought whose elements are then the TDI observables. Another matrix
approach was put forward by Tinto, Dhurandhar and JoshiTinto et al. (2021). In
this work, it was shown that the matrix approach is a ring representation
Burrow (1965) of the operator approach - there is a homomorphism between the
ring of operators into the ring of matrices. However, the matrix approach
seems to have an advantage because matrices are easy to manipulate (although
this has not been conclusively established). This is in the same spirit, as
one uses group representations rather than abstract group elements to perform
calculations. In this paper we show that there is in fact an isomorphism
(which is more than homomorphism - the map is also one to one and onto)
between the design matrices defined in Vallisneri et al. (2021) and the matrix
operators defined in our approach Tinto et al. (2021).

This paper is organized as follows. In Sec. II we describe the design matrices
defined in Vallisneri et al. (2021) but extract half the rows corresponding to
one of the arms because they have the basic structure we want to investigate.
In Sec. III we then prove the isomorphism between the matrices defined in
Tinto et al. (2021) and Vallisneri et al. (2021), while in Sec. IV we show how
to generalize the one-arm results to the two-arms configuration. In Sec. V we
finally present our concluding remarks, and emphasize that the isomorphism
existing between our matrix representation of the TDI delay operators and the
matrices introduced in Vallisneri et al. (2021) should help us in relating the
laser noise-free combinations identified by the two methods.

##  II Algebra of design matrices: the case of the single arm

In this section, we will consider just one of the arms, say arm 1, that is, we
do not consider the interleaving of $y_{1}$ and $y_{2}$ discussed in
Vallisneri et al. (2021). In section IV we will indicate how to generalise the
analysis to the case for two arms. Also we will only consider that part of the
matrix which describes the delay and therefore disregard the subtraction of
the unit matrix; that is, if $M$ is the design matrix for one arm, then we
consider the matrix $V=M+I$, where $I$ is the identity matrix. We denote the
matrix by $V$ since the design matrix has been introduced in the paper by
Vallisneri et al Vallisneri et al. (2021). Also for simplicity (as mentioned
in the same paper) we will take the sampling interval to be unity. The delayed
data are given by ${\mathcal{D}}_{1}y(t)=y(t-l_{1})$. It is first of all
easily verified that for integer delays the product of two operators say
${\mathcal{D}}_{1}$ and ${\mathcal{D}}_{2}$ corresponds to the product of the
matrices $D_{1}$ and $D_{2}$. These delay matrices for delays $\Delta t=1,2,3$
have been explicitly displayed in Tinto et al. (2021). The $V$ matrices for
integer delays can be taken to be identical to the $D$ matrices and so the
homomorphism for the $V$ matrices follows as shown in Tinto et al. (2021).
This disposes of the integer delays.

For fractional delays, we only need to exhibit a bijective map between the
matrices $D$ defined in Tinto et al. (2021) and the matrices $V$. We do so
below. For this purpose, we will first need to describe the structure of the
matrix $V$ for fractional delays. The Lagrange polynomial interpolation scheme
is chosen as in Vallisneri et al. (2021) with the degree $m$ of the
polynomials set equal to $6$. The data $y$ are labelled at integer nodes at
$t_{k}=k,~{}k=0,1,2,...$ and are denoted accordingly by $y_{k}=y(t_{k})$. Now
consider a fractional delay $\alpha$. The non-integer delay $\alpha$ is broken
up into its integer part $[\alpha]$ and the residual part $\delta\alpha$
accordingly as $\alpha=[\alpha]+\delta\alpha$. Now the interval containing $m$
nodes has to be chosen so that it covers the delayed time instant and such
that it lies somewhere near the centre of the interval. This depends at what
time instant we are evaluating the delayed data. If the time instant is $k$,
then we go back $[\alpha]+m/2=[\alpha]+3$ nodes, and it is at this node the
filter mask starts. For example, if $\alpha=2.2$, then $[\alpha]=2$ and one
must start the interpolating interval from $k-5$. If the data are measured
from $t=0$, then a full mask is possible only when $k\geq 5$. The first such
instant occurs at $k=5$ and the interval is $\\{0,1,2,3,4,5\\}$ with node
values $\\{y_{0},y_{1},...,y_{5}\\}$. The interpolated data is evaluated at
$t=5-\alpha=2.8\equiv\alpha^{\prime}$ say, or $y(2.8)$. For the sake of
completeness we give below the general expressions for the Lagrange
polynomials on $m$ nodes at $t_{k},~{}k=0,1,...,m-1$:

$l_{k}(t)=\frac{(t-t_{0})(t-t_{1})...(t-t_{k-1})(t-t_{k+1})...(t-t_{m-1})}{(t_{k}-t_{0})...(t_{k}-t_{k-1})(t_{k}-t_{k+1})...(t_{k}-t_{m-1})}\,,~{}~{}~{}~{}k=0,1,...,m-1.$
(1)

Note that $l_{k}(t)$ are polynomials of degree $m-1$. Here we have $m=6$ and
so the polynomials are of degree 5. Then the interpolated value of $y$ at
$t=\alpha^{\prime}$ is given by,

$y(\alpha^{\prime})=\sum_{k=0}^{5}l_{k}(\alpha^{\prime})y_{k}\,.$ (2)

In the $V$ matrix, the first row with 6 non-zero entries occurs first at the
5th row. Then we have $V_{5k}=l_{k}(\alpha^{\prime})$ for $0\leq k\leq 5$ and
$V_{5k}=0$ for $k>5$. In the next row $k=6$, the filter mask covers the
interval $\\{1,2,...,6\\}$ and the corresponding data points are
$\\{y_{1},y_{2},...,y_{6}\\}$. In the matrix $V$, the Lagrange polynomials are
shifted by one column to the right with $V_{60}=0$ and $V_{6k}=0$ for
$k=1,2,...,6$. Here for simplicity we have chosen the time-delay to be
constant (the time dependent case does not make much difference to the
homomorphism argument). As one proceeds down the rows, the Lagrange
polynomials get shifted to the right and so diagonally downwards. The matrix
$V$ looks as follows:

$V(\alpha^{\prime})=\left(\begin{array}[]{cccccccccc}\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\cdots\\\
l_{0}(\alpha^{\prime})&l_{1}(\alpha^{\prime})&l_{2}(\alpha^{\prime})&l_{3}(\alpha^{\prime})&l_{4}(\alpha^{\prime})&l_{5}(\alpha^{\prime})&0&0&0&\cdots\\\
0&l_{0}(\alpha^{\prime})&l_{1}(\alpha^{\prime})&l_{2}(\alpha^{\prime})&l_{3}(\alpha^{\prime})&l_{4}(\alpha^{\prime})&l_{5}(\alpha^{\prime})&0&0&\cdots\\\
0&0&l_{0}(\alpha^{\prime})&l_{1}(\alpha^{\prime})&l_{2}(\alpha^{\prime})&l_{3}(\alpha^{\prime})&l_{4}(\alpha^{\prime})&l_{5}(\alpha^{\prime})&0&\cdots\\\
\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\ddots\end{array}\right)$
(3)

The first non-trivial row shown is for $k=5$. For $\alpha^{\prime}=2.8$, the
$l_{k}$ take the numerical values $0.006336,-0.04928,0.22176,\break
0.88704,-0.07392,0.008064$ as $k$ ranges from $0$ to $5$ in steps of unity.

##  III The isomorphism between $V$ and $D$ matrices

For establishing the homomorphism this arrangement presents difficulties
because the target subspace, namely, the carrier space, changes (advances)
with each successive row. For $k=5$ the target subspace is the interval
$W_{0}=\\{0,1,2,3,4,5\\}$ while for $k=6$, the target subspace is
$W_{1}=\\{1,2,3,4,5,6\\}$ and so on. In order to establish homomorphism one
requires a fixed target subspace. We therefore fix a target subspace. There
are several choices for this; we make the following one. We fix the target
subspace to be $W_{0}$ and so refer all the Lagrange polynomials to $W_{0}$;
in effect we translate the Lagrange polynomials to $W_{0}$. This means, for
example, the entries in the 6th row ($k=6$), need to be shifted by one column
to the left. This is achieved by translating the Lagrange polynomials - that
is by adding $1$ to the argument $\alpha^{\prime}$. This makes sense because
for the value of $\alpha^{\prime}=2.8$, we evaluate the Lagrange polynomial at
$\alpha^{\prime}+1=3.8=6-2.2=6-\alpha$. Thus the translated Lagrange
polynomials are $l_{k}(\alpha^{\prime}+1)$. Similarly, for the next row $k=7$,
one needs to shift the Lagrange polynomials by two columns to the left and
therefore we must add 2 to the argument $\alpha^{\prime}$ resulting in
$l_{k}(\alpha^{\prime}+2)$. Thus the entries in rows $k=5,6,7,8,9,10$ are
shifted to the left by the appropriate number of columns with the arguments of
the Lagrange polynomials increased by the number equal to the number of
shifted columns. By carrying out this procedure we obtain the translated
matrix $V^{\rm trans}$ given below:

$V^{\rm
trans}(\alpha^{\prime})=\left(\begin{array}[]{cccccccccc}\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\cdots\\\
l_{0}(\alpha^{\prime})&l_{1}(\alpha^{\prime})&l_{2}(\alpha^{\prime})&l_{3}(\alpha^{\prime})&l_{4}(\alpha^{\prime})&l_{5}(\alpha^{\prime})&0&0&0&\cdots\\\
l_{0}(\alpha^{\prime}+1)&l_{1}(\alpha^{\prime}+1)&l_{2}(\alpha^{\prime}+1)&l_{3}(\alpha^{\prime}+1)&l_{4}(\alpha^{\prime}+1)&l_{5}(\alpha^{\prime}+1)&0&0&0&\cdots\\\
l_{0}(\alpha^{\prime}+2)&l_{1}(\alpha^{\prime}+2)&l_{2}(\alpha^{\prime}+2)&l_{3}(\alpha^{\prime}+2)&l_{4}(\alpha^{\prime}+2)&l_{5}(\alpha^{\prime}+2)&0&0&0&\cdots\\\
\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&0&0&0&\cdots\\\
l_{0}(\alpha^{\prime}+5)&l_{1}(\alpha^{\prime}+5)&l_{2}(\alpha^{\prime}+5)&l_{3}(\alpha^{\prime}+5)&l_{4}(\alpha^{\prime}+5)&l_{5}(\alpha^{\prime}+5)&0&0&0&\cdots\\\
\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\ddots\end{array}\right)$
(4)

We immediately recognise that the above block matrix of the Lagrange
polynomials is identical with the $D$ matrix, namely, Eq. (4.13), of reference
Tinto et al. (2021). Since as shown in there, the $D$ matrices form a
representation of the fractional delay operators, it follows that the block
matrices in $V^{\rm trans}$ also constitute a representation of the same
operators.

We can also perform the operation of shifting the rows. This is just
translating the polynomials by the required time stamps. If we shift the rows
’upwards’ by $r$ time samples, then we must subtract $r$ from the argument of
the Lagrange polynomials. For example, in the above example, if we shift by 2
time samples upwards, the arguments in any column of the $6\times 6$ block
will range from $\alpha^{\prime}-2$ to $\alpha^{\prime}+3$. Thus in our case
of $\alpha^{\prime}=2.8$, the arguments will range from $0.8$ to $5.8$ close
to the interpolation nodes of $W_{0}$. This is relevant when numerical
accuracy is a consideration. The opposite happens if we shift down and to the
right - shifting diagonally downwards and to the right (or upwards and to the
left) keeps the entries in each row the same - the arguments of the Lagrange
polynomials do not change. We have assumed here constant armlengths for
simplicity. We will remark later on time-dependent armlengths; they do not
cause any difficulty in principle to the homomorphism argument.

The above discussion was relevant to the first $6\times 6$ block. The next
$6\times 6$ block begins at the 11th row, that is, $j=11$ to $j=16$ and the
columns are from $k=6$ to $k=11$. Thus the target subspaces start from $W_{6}$
onwards. We may now fix the target subspace to be $W_{6}$ and repeat the steps
above. This will lead to an identical block matrix as the first one. Thus the
matrix represention of $V^{\rm trans}$ is block diagonal, each block a
$6\times 6$ matrix.

We can describe this representation directly in general for a filter mask on
$m$ nodes. Let $\alpha$ and $\beta$ be two time delays. Define
$r=[\alpha]+m/2$ and $s=[\beta]+m/2$, then $V^{\rm
trans}_{r+j,k}(\alpha^{\prime})=l_{j}(\alpha^{\prime}+k)$ and $V^{\rm
trans}_{s+k,n}(\beta^{\prime})=l_{k}(\beta^{\prime}+n)$ then:

$\sum_{k=0}^{m-1}V^{\rm trans}_{r+j,k}(\alpha^{\prime})V^{\rm
trans}_{s+k,n}(\beta^{\prime})=V^{\rm
trans}_{r+s+j,n}(\alpha^{\prime}+\beta^{\prime})\,.$ (5)

The above equation follows from the addition theorem for Lagrange polynomials.
Eq. (5) directly exhibits the homomorphism. This is in fact the same as
equation Eq. (4.15) given in reference Tinto et al. (2021). Since $m/2$ is
added both to $r$ and $s$, we may shift the rows by $m/2$ upwards by writing
the RHS of Eq. (5) as $V^{\rm
trans}_{r+s+j-m/2,n}(\alpha^{\prime}+\beta^{\prime}-m/2)$ to get it in the
required form as given in Vallisneri et al. (2021). This rule obtains if
$\delta\alpha+\delta\beta<1$. If this is not the case, that is, if
$\delta\alpha+\delta\beta\geq 1$ then the required upward shift is $m/2-1$.

The time dependent case follows exactly the discussion in Tinto et al. (2021).
If the delay $\beta$ is applied after $\alpha$, then $\beta$ becomes a
function of $\alpha$ and the composite delay is given by
$\alpha+\beta(\alpha)$. If the order of the delays is reversed, then the
composite delay is $\beta+\alpha(\beta)\neq\alpha+\beta(\alpha)$. Thus the
delay operators and their representative matrices do not commute in general.
This is the basic difference between the time dependent and time independent
cases. Rest of the discussion parallels the discussion for the time
independent case.

The only point remaining is to formally establish the correspondence between
$V$ and $V^{\rm trans}$. Let this correspondence be denoted by the mapping
$\psi$. We show below that $\psi$ is in fact an isomorphism (not merely a
homomorphism). Thus we have,

$\psi[V(\alpha^{\prime})]=V^{\rm trans}(\alpha^{\prime})\,.$ (6)

The above discussion shows that:

$\psi[V(\alpha^{\prime}+\beta^{\prime})]=V^{\rm
trans}(\alpha^{\prime}+\beta^{\prime})=V^{\rm trans}(\alpha^{\prime})\star
V^{\rm
trans}(\beta^{\prime})=\psi[V(\alpha^{\prime})]\star^{\prime}\psi[V(\beta^{\prime})]\,,$
(7)

where the $\star$ and $\star^{\prime}$ operations are defined as above.

Here some remarks are in order. It is first of all clear that $\psi$ is linear
and hence a homomorphism. Eq. (7) in fact defines the operation
$\star^{\prime}$. We will establish below that $\psi$ is bijective and hence
an _isomorphism_. We will explicitly establish this fact by exhibiting
formulae. In the literature Burrow (1965), the map $\psi$ or more
appropriately its extension $\tilde{\psi}$ is called an _intertwinor_ and the
representation an intertwining representation. Thus the matrices $V$ form an
intertwining representation. In the appendix A we indicate how the map $\psi$
is extended to the intertwining map $\tilde{\psi}$.

Let us consider first, the first $6\times 6$ block of $V$. It is easily shown
that the $6$ row vectors of the block matrix, namely, $R_{k},~{}k=0,1,...,5$
are linearly independent and therefore span a $6$ dimensional subspace
$B_{1}=R_{0}\oplus R_{1}\oplus R_{2}\oplus...\oplus R_{5}$, where the symbol
$\oplus$ denotes direct sum. $B_{1}$ is the domain of the map $\psi$. Now we
come to the range of $\psi$ which is the first block $B^{\rm trans}_{1}$ of
$V^{\rm trans}$. $B^{\rm trans}_{1}$ also consists of 6 linearly independent
row vectors, because the inverse $[V^{\rm trans}(\alpha)]^{-1}=V^{\rm
trans}(-\alpha)$ in principle always exists - we can always undo the delay by
reversing the situation. The existence of the inverse implies that the row
space of $B^{\rm trans}_{1}$ must be $6$ dimensional. This proves that the map
$\psi$ is bijective where $\psi(B_{1})=B^{\rm trans}_{1}$. This map can be
extended in an obvious way to the rest of the blocks of $V$ and $V^{\rm
trans}$, hence establishing formally the correspondence as desired.

The isomorphism can also be proved explicitly by computing the translation
matrices. We show this only for the block $B_{1}$. The argument extends in an
obvious way to $V$. But before we apply the translation matrices, we need to
shift the rows of $B_{1}$ to the left by the appropriate number of columns.
This is done by projecting out each row at a time by applying the projection
operators $P_{k},~{}k=0,1,...,5$ and then shifting to the left by the required
number of columns. The matrices $P_{k}$ are $6\times 6$ and have all entries
$0$, except for $1$ on the $k^{\text{th}}$ row and column. Applying $P_{k}$ on
the left of $B_{1}$ picks out the $k^{\text{th}}$ row, zeroing out other rows.
Then we need to shift the rows by $k$ columns to the left. This is achieved by
applying shift matrices $S_{k}$ which are $11\times 6$. These are nothing but
essentially the delay operators $D_{k}$ for integer valued delays. Finally, we
apply the translation matrices $T_{k}$ which map a row vector
$L_{0}\longrightarrow L_{k}$, where we define the row vector
$L_{k}=[l_{0}(\alpha+k,l_{1}(\alpha+k),...,l_{5}(\alpha+k)]$. Thus, we write
$L_{k}=L_{0}T_{k}$. The $T_{k}$ are $6\times 6$ matrices. We therefore obtain:

$B^{\rm trans}_{1}=\sum_{k=0}^{5}P_{k}B_{1}S_{k}T_{k}\,.$ (8)

This is the map $\psi$.

We can also invert the above relation. We only need to multiply the above sum
by the projection operator $P_{j}$ from the left; because
$P_{j}P_{k}=P_{j}\delta_{jk}$. This picks out the $j^{\text{th}}$ term zeroing
out the other terms. Thus we obtain:

$B_{1}=\sum_{k=0}^{5}P_{k}B^{\rm trans}_{1}T_{k}^{-1}S_{-k}\,.$ (9)

This is the inverse map $\psi^{-1}$. Here $S_{-k}$ is the operator which
shifts the elements to the right by $k$ columns and undoes the effect $S_{k}$.
Further, the $T_{k}$ matrices are invertible - in fact $\det{T_{k}}=1$. We can
compute them explicitly. We can write any $l_{n}(\alpha+k)$ as linear
combinations of $l_{j}(\alpha)$. For example,
$l_{0}(\alpha+1)=-l_{5}(\alpha),l_{1}(\alpha+1)=l_{0}(\alpha)+6l_{5}(\alpha),...$.
This equation can be written in matrix form: $L_{1}=L_{0}T_{1}$, where,

$T_{1}=\left[\begin{array}[]{ccccccc}0&1&0&0&0&0\\\ 0&0&1&0&0&0\\\
0&0&0&1&0&0\\\ 0&0&0&0&1&0\\\ 0&0&0&0&0&1\\\
-1&6&-15&20&-15&6\end{array}\right]\,.$ (10)

The other translation matrices $T_{k}$ can be obtained easily from the
addition theorem for Lagrange polynomials. All the translation matrices
$T_{k}$ are non-singular and in fact have determinant $1$. This is because the
vectors are rigidly translated, keeping the volume of the parallelepiped
defined by those vectors invariant.

Thus the isomorphism can be explicitly established directly.

##  IV The generalisation to two arms

In this section we indicate how to generalise to the case of two unequal arms.
Here we have in general two different delays $\alpha_{1}$ and $\alpha_{2}$
corresponding to the arms $1$ and $2$ respectively. The mathematical structure
is that of the product ring. We will do this for a group ${\cal G}$. The
product is denoted by ${\cal G}\times{\cal G}$. It is defined as follows.

Let ${\cal G}$ be a group and let $g_{1},g_{2}\in{\cal G}$ then the element of
${\cal G}\times{\cal G}$ is the ordered pair $(g_{1},g_{2})$ or we write
$(g_{1},g_{2})\in{\cal G}\times{\cal G}$. The composition law in ${\cal
G}\times{\cal G}$ is defined in the following way: Let $(g_{1},g_{2})$ and
$(h_{1},h_{2})$ belong to ${\cal G}\times{\cal G}$, then
$(g_{1},g_{2})\cdot(h_{1},h_{2})=(g_{1}h_{1},g_{2}h_{2})$. Clearly the product
so defined is in ${\cal G}\times{\cal G}$. It is easily shown that under this
composition law ${\cal G}\times{\cal G}$ is a group.

The next point to consider is a matrix representation of ${\cal G}$ which
associates a matrix $T_{g}$ with each element $g\in{\cal G}$. The matrices
$T_{g}$ are actually linear maps from a vector space $T_{g}:W\longrightarrow
W$. A representation is a homomorphism $\phi$ which takes $g\in{\cal G}$ to
$T_{g}$ or $\phi(g)=T_{g}$, such that for all $g,h\in{\cal G}$,
$\phi(gh)=T_{g}T_{h}$ and $\phi(e)=I$ or the identity of the group $e$ is
mapped to the identity matrix.

From the above considerations, we may easily define a representation of ${\cal
G}\times{\cal G}$. Consider a finite dimensional representation, that is,
$\text{dim}(W)=n$. Then $T_{g}$ is a $n\times n$ matrix. Now consider an
element $(g,h)\in{\cal G}\times{\cal G}$, then we have the corresponding
$n\times n$ matrices $T_{g}$ and $T_{h}$. We define the product representation
$\phi\otimes\phi$ by the block diagonal $2n\times 2n$ matrix:

$\phi\otimes\phi[(g,h)]=\left[\begin{array}[]{cc}T_{g}&0\\\
0&T_{h}\end{array}\right]\,.$ (11)

It is easy to show that $\phi\otimes\phi$ constitutes a representation of
${\cal G}\times{\cal G}$. It is important to note that the two block matrices
are essentially independent of each other.

For the two arm case there are in general two independent time-delays, say
$\alpha_{1}$ and $\alpha_{2}$ (these are the $g$ and $h$ in the above
discussion). Under the representation homomorphism they map to $6\times 6$
matrices $B^{\rm trans}_{1}(\alpha_{1})$ and $B^{\rm trans}_{1}(\alpha_{2})$
respectively or if one considers the entire matrices $V^{\rm
trans}(\alpha_{1})$ and $V^{\rm trans}(\alpha_{2})$. Under the isomorphism
$\psi$, $V^{\rm trans}$ matrices map to the $V$ matrices. These can be
arranged as $n\times n$ block diagonal matrices as in Eq. (11) to obtain the
product representation matrix which is $2n\times 2n$. Then by the following
matrix transformation below, this block diagonal matrix is converted into a
$2n\times n$ matrix:

$\left[\begin{array}[]{cc}V(\alpha_{1})&0\\\
0&V(\alpha_{2})\end{array}\right]\left[\begin{array}[]{c}I_{n}\\\
I_{n}\end{array}\right]=\left[\begin{array}[]{c}V(\alpha_{1})\\\
V(a_{2})\end{array}\right]\,,$ (12)

where $I_{n}$ is the $n\times n$ unit matrix. This is not an isomorphism.
Finally, the design matrix $M$ (except for the subtraction of the identity
matrix) is obtained by a permutation of the rows of the matrix on the RHS of
Eq. (12). The permutation operation can also be represented by a matrix
$P_{\rm interleave}$ consisting of $0$ s and $1$ s. The matrix $P_{\rm
interleave}$ is just
$\\{\delta_{2m-1,m},\delta_{2m,n+m}\\},~{}~{}~{}m=1,2,...,n$. It is non-
singular and hence also an isomorphism. Thus we have an intimate relation
between the design matrices $M$ of Vallisneri et al. (2021) and the matrices
$V^{\rm trans}$ which form a representation of the delay operators.

##  V Concluding Remarks

Future space-based gravitational wave interferometers will rely on the use of
TDI to achieve their baseline sensitivities. The matrix representations of the
TDI delay operators discussed in this article should simplify and make the
implementation of TDI more efficient and consequently the analysis of
gravitational wave signals that we are searching for. In the process we have
made a detailed analysis of the design matrices. In this article we have shown
that the matrix representation of the delay operators derived in Tinto et al.
(2021) is isomorphic (i.e. one-to-one and onto) to the matrices introduced in
Vallisneri et al. (2021) to cancel the laser noise. The isomorphism we have
just established should help us in identifying a systematic way for relating
the laser noise-free combinations identified by TDI to those obtained by the
method proposed in Vallisneri et al. (2021). This will be the subject of a
forthcoming investigation.

## Acknowledgments

The authors would like to thank Hemant Bhate for useful conversations on group
representation theory. M.T. thanks the Center for Astrophysics and Space
Sciences (CASS) at the University of California San Diego (UCSD, U.S.A.) and
the National Institute for Space Research (INPE, Brazil) for their kind
hospitality while this work was done. S.V.D. acknowledges the support of the
Senior Scientist Platinum Jubilee Fellowship from NASI, India.

##  Appendix A The interwining map $\tilde{\psi}$

Although the map $\psi$ exhibits closure property and associativity, it does
not map the identity to identity because $B_{1}$ is not a square matrix. This
is easily remedied by suitably augmenting the matrices by identity and zero
matrices. The map $\psi$ is extended to $\tilde{\psi}$ as follows. In Eqs. (8)
and $(\ref{psiinv})$ we make all matrices $11\times 11$. In the projection
matrix $P_{k}$ we let $k=0,1,..,10$ and call it $\tilde{P}_{k}$. We also
define $\tilde{B}_{1}$, $\tilde{S}_{k}$, $\tilde{T}_{k}$ by appropriately
adjoining $5\times 5$ identity matrices and zero matrices as follows:

$\tilde{B}_{1}^{\rm trans}=\left[\begin{array}[]{cc}B^{\rm trans}_{1}&0\\\
0&I_{5}\end{array}\right]\,,~{}~{}~{}\tilde{B}_{1}=\left[\begin{array}[]{ccc}&B_{1}&\\\
0&&I_{5}\end{array}\right]\,,~{}~{}~{}\tilde{S}_{k}=\left[\begin{array}[]{cc}&0\\\
S_{k}&\\\
&I_{5}\end{array}\right]\,,~{}~{}~{}\tilde{T}_{k}=\left[\begin{array}[]{cc}T_{k}&0\\\
0&I_{5}\end{array}\right]\,,$ (13)

where $I_{5}$ is a $5\times 5$ identity matrix and the block matrices $0$ have
appropriate dimensions to make all the matrices $11\times 11$. Then the
corresponding quantities in Eq. (8) and $(\ref{psiinv})$ can be replaced by
the quantities with tildes, and the sum over $k$ over 6 terms is replaced by a
sum over 11 terms, $k=0,1,2,...,10$. It is now easy to check that
$\tilde{\psi}$ not only satisfies the properties of $\psi$ but also maps
identity to identity, that is $\tilde{\psi}(I_{11})=I_{11}$, where $I_{11}$,
the $11\times 11$ identity matrix.

## References

  * Giampieri et al. (1996) G. Giampieri, R. W. Hellings, M. Tinto, and J. E. Faller, Opt. Commun. 123, 669 (1996).
  * Tinto and Armstrong (1999) M. Tinto and J. W. Armstrong, Phys. Rev. D 59, 102003 (pages 11) (1999).
  * Armstrong et al. (1999) J. W. Armstrong, F. B. Estabrook, and M. Tinto, Astrophys. J. 527, 814 (1999).
  * Tinto et al. (2002) M. Tinto, F. B. Estabrook, and J. W. Armstrong, Phys. Rev. D 65, 082003 (pages 12) (2002).
  * Tinto et al. (2004) M. Tinto, F. B. Estabrook, and J. W. Armstrong, Phys. Rev. D 69, 082001 (2004).
  * Bender and et. al. (1998) P. L. Bender and et. al. (LISA Study Team), Tech. Rep. MPQ-233, Max-Planck-Institut für Quantenoptik, Garching (1998), URL ftp://ftp.ipp-garching.mpg.de/pub/grav/lisa/pdd/.
  * Amaro-Seoane et al. (2017) P. Amaro-Seoane et al., ArXiv e-prints (2017), eprint 1702.00786.
  * Dhurandhar et al. (2002) S. V. Dhurandhar, K. Rajesh Nayak, and J.-Y. Vinet, Phys. Rev. D 65, 102002 (pages 16) (2002).
  * Becker and Weispfenning (1993) T. Becker and V. Weispfenning, _Gröbner Bases: A Computational Approach to Commutative Algebra_ , vol. 141 of _Graduate Texts in Mathematics_ (Springer, New York, 1993).
  * Rajesh Nayak and Vinet (2005) K. Rajesh Nayak and J.-Y. Vinet, Class. Quantum Grav. 22, S437 (2005).
  * Dhurandhar et al. (2008) S. V. Dhurandhar, J.-Y. Vinet, and K. Rajesh Nayak, Class. Quantum Grav. 25, 245002 (pages 17) (2008).
  * Dhurandhar et al. (2010) S. V. Dhurandhar, K. Rajesh Nayak, and J.-Y. Vinet, Class. Quantum Grav. 27, 135013 (2010), eprint 1001.4911.
  * Shaddock et al. (2003) D. A. Shaddock, M. Tinto, F. B. Estabrook, and J. W. Armstrong, Phys. Rev. D 68, 061303 (pages 4) (2003).
  * Tinto and Dhurandhar (2021) M. Tinto and S. V. Dhurandhar, Living Reviews in Relativity 24, 6 (2021), ISSN 1433-8351, URL https://doi.org/10.1007/s41114-020-00029-6.
  * Vallisneri et al. (2021) M. Vallisneri, J.-B. Bayle, S. Babak, and A. Petiteau, Phys. Rev. D 103, 082001 (2021), URL https://link.aps.org/doi/10.1103/PhysRevD.103.082001.
  * Romano and Woan (2006) J. D. Romano and G. Woan, Phys. Rev. D 73, 102001 (2006), URL https://link.aps.org/doi/10.1103/PhysRevD.73.102001.
  * Tinto et al. (2021) M. Tinto, S. V. Dhurandhar, and P. Joshi, _Matrix representation of time-delay interferometry_ (2021), eprint 2105.02054.
  * Burrow (1965) M. Burrow, _Representation Theory of Finite Groups_ (Academic Press Inc., New York, 1965).