# Enabling and Accelerating Dynamic Vision Transformer Inference for Real-Time
Applications

Kavya Sreedhar  Stanford University  Jason Clemons  NVIDIA  Rangharajan
Venkatesan<EMAIL_ADDRESS>NVIDIA  Stephen W. Keckler<EMAIL_ADDRESS>NVIDIA  Mark Horowitz<EMAIL_ADDRESS>Stanford University

###### Abstract

Many state-of-the-art deep learning models for computer vision tasks are based
on the transformer architecture. Such models can be computationally expensive
and are typically statically set to meet the deployment scenario. However, in
real-time applications, the resources available for every inference can vary
considerably and be smaller than what state-of-the-art models use. We can use
dynamic models to adapt the model execution to meet real-time application
resource constraints. While prior dynamic work has primarily minimized
resource utilization for less complex input images while maintaining accuracy
and focused on CNNs and early transformer models such as BERT, we adapt vision
transformers to meet system dynamic resource constraints, independent of the
input image. We find that unlike early transformer models, recent state-of-
the-art vision transformers heavily rely on convolution layers. We show that
pretrained models are fairly resilient to skipping computation in the
convolution and self-attention layers, enabling us to create a low-overhead
system for dynamic real-time inference without additional training. Finally,
we create a optimized accelerator for these dynamic vision transformers in a
5nm technology. The PE array occupies 2.26mm2 and is 17 times faster than a
NVIDIA TITAN V GPU for state-of-the-art transformer-based models for semantic
segmentation.

##  I Introduction

The transformer architecture [57] has lead to state-of-the-art models for
computer vision tasks, including image classification [24, 8, 16, 55], object
detection [7, 35, 81, 86] and semantic segmentation [37, 49, 71, 83]. Such
models typically have a fixed execution path. This execution can be
computationally expensive, requiring millions of parameters and billions of
floating point operations (FLOPs) [22, 29]. However, real-time systems for
applications such as autonomous driving [18, 47], video conferencing [62], and
speech recognition [2] have dynamic system loads that change as the
surrounding environment changes. As a result, the resources available each
time a task needs to be performed vary, and at times can be smaller than what
state-of-the-art models require [40].

In these cases, the model execution path can be dynamically adapted to ensure
that the resource utilization matches the application resource availability.
Since real-time applications can handle occasional accuracy degradation [3, 5,
15, 43], accuracy can be traded for more efficient execution to ensure that
inference completes with the available resources.

While dynamic models have been previously developed, they have been primarily
used to shorten the model execution based on the input and estimate the
complexity of classifying it, with many approaches exiting early when internal
predictions have not changed through a few layers [12, 21, 36, 19, 54, 73, 85,
53]. Such approaches aim to minimize the execution time or energy while
maintaining model accuracy for “easier” inputs, which does not address our
problem of ensuring that the model execution meets a given dynamic execution
time or energy constraint.

Instead, we build on other work which has presented various parameterizations
of the same static model architecture, corresponding to varying levels of
model complexity and accuracy. Such work has relied on removing redundant
computation [7, 26, 41, 60], and has achieved significant computation savings
with modest decreases in accuracy. Recent work called Once-For-All (OFA) [6]
has developed efficient training techniques to produce many competitive sets
of model weights in a single run, removing the need for multiple training
runs. These prior works focus on CNNs and BERT (consisting of just the
transformer encoder) [13], so we need to extend this work to vision
transformers.

We find that unlike early transformer-based models which are convolution-free
and dominated by self-attention [13, 16], recent vision transformers rely on
convolutional backbones and convolutions in the transformer decoder. Thus, we
can leverage existing CNN dynamic techniques and further explore the accuracy
resiliency of pretrained models to pruning in convolutional and self-attention
layers in order to enable dynamic inference with real-time application
constraints.

We use this resilience to build a dynamic real-time (DRT) inference engine
that augments pretrained models with alternative lower-cost execution paths
that meet given dynamic resource constraint targets, using the same model
weights. During runtime, given an input dynamic resource constraint, we
minimally add the cost of choosing which path to execute. This approach
requires no additional training or changes to training. Thus, determining how
to adapt pretrained models is much faster than the time required to retrain
models in prior work. We compare these results to the trained models available
to determine when switching between sets of trained models would be more
advantageous than dynamically pruning pretrained models for maintaining
accuracy.

We then explore the impact of executing dynamic vision transformer models on
the accelerator design space. With dynamic models, we need to choose how to
optimize the hardware since the characteristics of each application can vary.
We use SegFormer [71] and Swin Transformer [37] with our DRT approach as case
studies for semantic segmentation, and the Once-For-All tradeoff curve for
ResNet-50 [6] as a case study for the computationally-dominant CNN backbones
for DETR [7] and Deformable DETR [86] for object detection.

We make the following contributions:

  * •

We find that convolutions, not self-attention layers, are computationally
dominant when considering the full application pipeline for state-of-the-art
object-detection and semantic-segmentation transformer models.

  * •

We show that state-of-the-art transformer models for semantic segmentation are
fairly resilient to modifications in the model execution path without any
additional training and create a low-overhead engine that leverages this
resiliency to adapt models to maximize model accuracy while meeting dynamic
resource constraints.

  * •

We create an optimized accelerator for DRT vision transformers, and show that
the optimal architecture does not depend on the model complexity.

To better understand the required computation in vision transformers, the next
section reviews the evolution of these applications to use more convolutional
computation. This trend focuses our attention in Section III on examining the
resiliency of these application’s key layers in pretrained models to bypassing
computation. This resiliency allows us to create a dynamic real-time inference
engine, which we describe in Section IV. Sections V and VI then use this
engine to explore the impact of executing such dynamic models on accelerator
design.

##  II Computation in Vision Transformers

We focus on object detection [87] and semantic segmentation [77] since these
tasks are widely used [2, 62, 18, 47] and are more complicated than image
classification. Object detection outputs a bounding box and class label for
each object identified. Semantic segmentation outputs an image that assigns
class labels to each pixel in the input. Thus, for semantic segmentation, the
output typically has the same resolution as the input.

State-of-the-art models for these tasks rely on transformers, an encoder-
decoder architecture that consists of multi-head self-attention layers and
fully-connected feedforward networks (FFN) [57]. Following from ViT, the first
vision transformer, such models reshape 2D images into a 1D sequence of
flattened image patches, which are linearly projected into an embedding
dimension [16]. Throughout the layers, models progressively increase the
embedding dimension and reduce the spatial dimensions to capture details at
various image resolutions, including finer spatial details, for higher
accuracy.

While ViT is convolution-free, most modern vision transformers use
convolutions to achieve higher performance [72, 83, 71, 37]. For example,
SegFormer [71], a state-of-the-art semantic segmentation model, uses
convolutions to overlap image patches, reduce the complexity of attention, and
replace fixed position encoding to achieve higher segmentation accuracy and
reduce the number of parameters. DETR [7] is a modern object detector and uses
a CNN backbone to extract image features compactly, a conventional
transformer, and a FFN to return the detection prediction. Many recent
transformer-based models build upon DETR for different visual processing tasks
[11, 82, 86, 75, 44].

Figure 1: Execution time breakdown between the ResNet-50 backbone and the transformer in DETR [7] and Deformable DETR [86] across various batch sizes on NVIDIA TITAN V GPU locked to 1005 MHz. Actual execution time in ms is labeled on the bars. Note that these results are for the COCO dataset [34] with image size 640 by 820. Model | Parameters (Millions) | Dataset | Image size | GFLOPs | Latency (ms) | FPS | mIoU / AP | Task
---|---|---|---|---|---|---|---|---
SegFormer B2 ADE [71] | 27.6 | ADE20K [84] | 512 by 512 | 62.6 | 58 | 17.2 | 0.4651 | SS
SegFormer B2 Cityscapes [71] | 27.6 | Cityscapes [10] | 1024 by 2048 | 705 | 415 | 2.4 | 0.8098 | SS
Swin Tiny [37] | 60 | ADE20K [84] | 512 by 512 | 237 | 215 | 4.7 | 0.4451 | SS
DETR [7] | 41 | COCO [34] | 640 by 480 | 86 | 162 | 6.2 | 0.401 | OD
Deformable DETR [86] | 40 | COCO [34] | 640 by 480 | 173 | 119 | 5.8 | 0.445 | OD
TABLE I: State-of-the-art vision transformer model summary for a batch size of
1 and profiling on a NVIDIA TITAN V GPU with clocks locked to 1005 MHz. SS =
semantic segmentation. OD = object detection. Accuracy metrics: mIoU for the
ADE20K dataset [84] (SS) and AP, with IoU from 0.5 to 0.95 in increments of
0.05 for the COCO-2017 dataset [34] (OD).  Figure 2: Layers in SegFormer [71]
model for semantic segmentation. The Swin Transformer [37] model follows the
same high-level encoder structure, with a more optimized module. The decoder
used with Swin has a layer similar to Conv2DFuse in SegFormer, which we refer
to as fpn_bottleneck_Conv2D.

To enable DRT inference, we identify components in vision transformer models
that minimally contribute to accuracy but require extensive computation. To
gather this information, given a pretrained model, we first profile the FLOPs
and execution time distribution among the model layers for inference on an
NVIDIA TITAN V GPU with clocks locked to 1005 MHz across various batch sizes
and image sizes. Since these models are large, requiring millions of
parameters and billions of FLOPS [22, 29], this step narrows the design space
to explore for enabling DRT inference. We use mmdetection [9], mmsegmentation
[1], torchprof [68], and the PyTorch Profiler to profile execution time.

We use mean intersection over union (mIoU), a common accuracy metric for
semantic segmentation, to quantify model accuracy. IoU is defined as the area
of overlap between the prediction and the ground truth divided by the area for
both the predicted segmentation and ground truth. Thus, it follows that mIoU
is the average of the IoU for every class. This value ranges from zero to one,
where values closer to one indicate more overlapping segmentation and thus
better accuracy.

###  II-A Object Detection

We use DETR [7], and Deformable DETR [86], which further optimizes the
attention module for higher detection accuracy, for our case study given their
widespread use and state-of-the-art accuracy. Figure 1 plots normalized
inference execution time across various batch sizes for these applications and
shows that the ResNet-50 backbone dominates the execution time on the TITAN V
GPU for both these models, while the transformer component comprises only 6.1%
to 12.4% of the total execution time for DETR, depending on the batch size
used. Similarly, the transformer comprises 6.1% to 18.4% of the total
execution time for Deformable DETR. Notice that the ResNet-50 CNN backbone
comprises more of the total execution time as batch size increases, confirming
that the convolution layers have better locality than the unblocked attention
layers. Clearly for DRT operation, we need to modulate the work done in this
CNN backbone. Fortunately, this is a well-studied area. We can apply existing
efficient inference techniques for CNNs (Section VII-A) for this class of
models and we use the Once-For-All approach [6] for ResNet-50 in Section V for
our accelerator experiments.

###  II-B Semantic Segmentation

Figure 3: FLOPs and execution time distribution across layers in SegFormer B2
model [71] for batch size of one on the ADE20K dataset on NVIDIA TITAN V GPU
with clocks locked to 1005 MHz. Figure 4: FLOPs and execution time
distribution across layers in Swin Tiny model [37] for batch size of one on
the ADE20K dataset on NVIDIA TITAN V GPU with clocks locked to 1005 MHz.
Figure 5: Image size vs proportion of FLOPs or latency (across batch sizes) in
fpn_bottleneck_conv, the large convolution in the decoder fusing information
at different image resolutions from the encoder stage outputs together. This
single convolution layer comprises a majority of FLOPs and latency for image
sizes in the ADE20K (512 by 512) [84] and Cityscapes (1024 by 2048) datasets
[10] for segmentation.

We study two models, SegFormer and Swin (Figure 2). Similar to SegFormer, Swin
Transformer [37] includes convolutions to include local continuity information
and further optimizes attention. We use these models as case studies for
semantic segmentation since they are modern transformers which achieve state-
of-the-art model accuracy. Figures 3 and 4 report the inference FLOPs and
execution time distribution for the SegFormer B2 model [71] (denoted
SegFormer) and the Swin-Tiny model [37], respectively. We find that 68% and
89% of the total FLOPs are in convolution layers in SegFormer and Swin-Tiny,
in contrast to the zero convolutions in ViT [16] and BERT (which only consists
of a transformer encoder) [13]. Few convolutions have been added to the
encoder (5% of convolutions are in the SegFormer encoder and 1% are in Swin
encoder) to overlap image patches to incorporate local continuity information
and reduce the size of inputs and the complexity of attentions. Attention
layers in the decoder have been replaced with large convolutions to fuse
information at different resolutions for the input image together for the
final pixel-level classification (Conv2DFuse in SegFormer and
fpn_bottleneck_conv in Swin, which uses the UPerNet decoder head [70] for
segmentation). SegFormer specifically chose to replace attention in the
decoder with convolution and linear layers for simplicity and higher accuracy.

Looking at SegFormer’s decoder, where most of the FLOPs are used, the Conv2D
Fuse, Conv2D Pred, and Decoder Linear connected to output of Encoder Stage 0
layers individually comprise $62\%$, $3\%$, and $1.3\%$ of the total number of
FLOPs for a 512 by 512 image, respectively, as shown in Figure 3. In
particular, Conv2D Fuse alone constitutes the majority of FLOPs, considerably
more so than any other individual layer in the model. The Conv2D Fuse layer
has 3072 input channels, 768 output channels and a 1 by 1 stencil and its
function in this model is to fuse information about the input image at various
image resolutions from the output of each encoder stage.

As we saw in the unblocked DETR, the number of FLOPs does not map to execution
time on the TITAN V GPU. Figure 3 reports the execution time across layer
types using the PyTorch autograd profiler. Convolutions comprise a quarter of
the total execution time (despite comprising 68% of the FLOPs), suggesting
that the GPU implementation is well-optimized for reusing convolution weights
as expected. Matrix multiplications take an equal amount of time compared to
convolutions.

Prior work, like Swin, has focused on optimizing encoder backbones [63, 64,
37] that can be used with different decoders and task-specific heads for
various tasks including classification, segmentation, and detection. Thus, the
core contribution of Swin and similar works is the backbone. For segmentation,
Swin uses the UPerNet decoder head [70]. However, we find that when
considering the full application pipeline, 89% of FLOPs are in the decoder and
99% of convolutions in the full model are in the decoder. Thus, when
considering segmentation applications, there is a need to prioritize
optimizing the decoder and the task-specific computation over the encoder
backbone since these operations dominate for segmentation. Our observations
can be more widely applicable to models that choose to use attention-dominant
backbones with the UPerNet decoder head for segmentation as well.

Similarly, prior transformer accelerator work has heavily focused on
accelerating attention for the same reason. Attention consists of matrix
multiplications and softmax, but those operations constitute 27% of the total
execution time and a much smaller proportion of FLOPs in the full application
pipeline. Thus, it is important to optimize accelerators balance the tradeoffs
between executing matrix multiplication and convolutions efficiently for
transformer-based models for semantic segmentation.

##  III Resiliency of Pretrained Semantic Segmentation Models

Since existing work [6, 26] has already explored the resiliency of the
ResNet-50 backbone which dominates object detection computation, this section
focuses on understanding how resilient pretrained semantic segmentation models
are to bypassing computation in the encoder and decoder of the transformer. To
accomplish this goal, we modify the original pretrained model execution graph
to allow for executing different subsets of the original model. Some examples
of alternative execution paths with the modified execution graph include:

Figure 6: Tradeoff between normalized mIoU (accuracy) and normalized execution time when dynamically pruning SegFormer B2 Models [71] with the Cityscapes and on the ADE20K datasets separately with no additional training. Please refer to Table II for the dynamic model configurations for labeled points. |  | Conv2DFuse | Normalized | Normalized
---|---|---|---|---
Label | Depths | In Channels | Resource Util | mIoU
A | 3, 4, 6, 3 | 3072 | 1 | 1
B | 3, 4, 6, 3 | 1920 | 0.88 | 0.98
C | 2, 4, 6, 3 | 1664 | 0.83 | 0.96
D | 2, 3, 6, 3 | 1408 | 0.78 | 0.92
E | 2, 3, 5, 3 | 1024 | 0.73 | 0.82
F | 3, 2, 5, 2 | 896 | 0.69 | 0.72
G | 2, 3, 4, 3 | 512 | 0.66 | 0.63
H | 2, 4, 6, 3 | 2432 | 0.76 | 0.98
I | 2, 4, 5, 3 | 2048 | 0.72 | 0.95
J | 2, 4, 5, 3 | 1280 | 0.68 | 0.90
K | 2, 4, 5, 3 | 896 | 0.66 | 0.81
L | 2, 4, 5, 3 | 384 | 0.63 | 0.69
TABLE II: SegFormer model execution path configurations corresponding to
various levels of resource utilization and model accuracy, as shown in Figure
6. Depths corresponds to number of encoder layers in Stage 0 through Stage 3
from left to right. Label A corresponds to the full model execution; labels B
through G are for the model trained on ADE20K; laels H through L are for the
model trained on Cityscapes.

  * •

Bypassing a model layer (i.e., performing an identity operation instead of the
original layer’s operation)

  * •

Reducing the number of channels input to a layer

  * •

Reducing the number of hidden dimensions (channels) output by a layer

  * •

Reducing sampling scale factors

Leveraging our understanding of the distribution of computation in SegFormer
B2 (27.6 million parameters) and Swin-Tiny (60 million parameters), from
Section II, we target these modifications on relatively expensive layers. To
quantify the trade-off between the decrease in resource utilization as well as
the resulting accuracy loss with these modifications, we run inference with
these alternative execution paths individually, sweeping parameters such as
the number of model layers in each encoder stage and the number of channels
input to expensive layers. Since we propagate bypassed computation backwards
through the model execution graph, skipping computation associated with input
channels to a layer can allow for skipping computation associated with output
channels in previous layers. Following these experiments, we then identify the
Pareto-optimal execution paths yielding higher accuracy and lower resource
utilization. We use execution time as an example of a dynamic resource
constraint.

We further explore the resiliency of SegFormer across image size (with
different datasets) by running our experiments for the SegFormer B2 model
trained with the CityScapes dataset [10], which uses larger image sizes (1024
by 2048) and thus has longer inference execution times than the ADE20K
dataset. As shown in Table I, the SegFormer B2 model corresponding to the
Cityscapes dataset has higher accuracy on the Cityscapes validation dataset
(0.8098) compared to model trained on and validated with the ADE20K dataset
(0.4651). We examine the resiliency of these models using all the datasets
that the model papers reported accuracy on: Swin Transformer published models
with the ADE20K dataset while SegFormer published models for the ADE20K and
CityScapes datasets. In this section, model layers are denoted according to
the labels used in Figure 2, and Table II. Note that fpn_bottleneck_Conv2D
refers a large convolution in the Swin decoder. As in the prior section, all
the experiments were run on a NVIDIA TITAN V GPU, with clocks locked to 1005
MHz.

###  III-A SegFormer-B2

Since 68% of FLOPs are in the decoder, we find that modifications such as
reducing the sampling scale factor in spatial-reduction attention [63] layers
or solely skipping encoder layers negligibly lower execution time and energy
but often substantially degrade accuracy without any retraining. Thus the
pretrained model is not resilient to such changes and such modified execution
paths are not ideal for DRT inference.

The trade-offs between model accuracy and resource utilization can be obtained
by reducing the number of input and/or output channels for Conv2DFuse,
Conv2DPred, and Decoder Linear connected to output of Encoder Stage 0, denoted
as DecodeLinear0. Reducing the number of input channels from 64 for
DecodeLinear0 layer does not allow for any further skipped computation in
earlier layers since the full output of Encoder Stage 0 must still be computed
as input for Encoder Stage 1. For the same reason, since the interpolation of
the output of the Linear layers after the Encoder Stages is reversed and then
concatenated, reducing input channels from 3072 to Conv2DFuse only allows for
skipped computation to propagate backwards if the model execution path uses
less than or equal to 768 input channels (corresponding to the contribution
from Encoder Stage 3, which is not input to another encoder stage and only
input to Conv2DFuse). In contrast, if we remove computation associated with
some of the 768 input channels to Conv2DPred, we can propagate that skipped
computation backwards through the preceding BatchNorm and ReLU layers, and
avoid computing the corresponding output channels of Conv2DFuse.

As more channels are removed from the inputs to the Conv2DFuse, Conv2DPred,
and DecodeLinear0 layers, the marginal accuracy drop increases. However, if we
additionally bypass encoder layers, we can shift these trade-off curves more
to the left than down and produce new Pareto-optimal execution paths. Thus,
Figure 6 shows the piecewise concatenation of these shifted curves to produce
the final trade-off curves for the Cityscapes and ADE20K datasets, with
execution time as a dynamic resource constraint. The configurations used to
obtain these points are shown in Table II. We find that combinations of
varying the number of encoder layers in each stage and input channels to
Conv2DFuse and Conv2DPred resulted in Pareto-optimal points.

We see that this model trained on the ADE20K dataset is fairly resilient to
skipping computation in the encoder layers and the Conv2DFuse and Conv2DPred
layers: we can save 17% of execution time (which drops energy consumption by
28% ) with less than a 6% drop in accuracy. Moreover, the marginal drop in
accuracy is less than the marginal drop in execution time and energy with
these Pareto-optimal points, as can be seen with light dashed $y=x$ reference
line in Figure 6. Surprisingly, through this exploration, we identified one
model configuration (with 736 instead of 768 input channels to Conv2DPred)
that achieves slightly better mIoU (0.4655 without any retraining and 0.4698
after training) than the original SegFormer B2 model (0.4651) and is 2.6%
faster.

The model trained on the Cityscapes dataset is even more resilient: we can
save 28% of execution time with less than a 5% drop in accuracy. This result
is likely due to the fact that the Cityscapes model is trained and executed on
larger image sizes (1024 by 2048 vs 512 by 512 in ADE20K), so the full
SegFormer B2 model trained on Cityscapes achieves 1.74X higher mIoU (0.8098 vs
0.4651) to begin with. Thus, there is likely more redundancy in these model
weights, allowing for a smoother accuracy degradation curve compared to the
model trained on ADE20K.

For reference, we plot the original SegFormer B0, B1, and B2 models (labeled
as ADE20K Trained Models and Cityscapes Trained Models) as large squares in
Figure 6. These points show the tradeoff provided by SegFormer with retrained
parameters for smaller versions of this model architecture. We see that this
dynamic pruning approach without retraining is competitive up until we have
saved around 25% of the execution time. For 50% saving (2x performance), one
should switch to retrained models to maintain accuracy. In addition, our
tradeoff curve for the ADE20K model results in slightly higher accuracy for 8%
to 20% of execution time savings.

###  III-B Swin

Figure 7: Tradeoff between normalized mIoU (accuracy) and normalized execution time when dynamically pruning Swin Base and Swin Tiny Models [37] with the Cityscapes and on the ADE20K datasets separately with no additional training. For Swin Tiny, the channels preserved as input to fpn_bottleneck_Conv2D are labels on the graph. Please refer to Table III for the dynamic model configurations for Swin Base points. | fpn_bottleneck_Conv2D | Normalized | Normalized
---|---|---|---
Depths | In Channels | Resource Util | mIoU
2, 2, 18, 2 | 2048 | 1 | 1
2, 2, 18, 2 | 1920 | 0.998 | 0.98
2, 2, 18, 2 | 1792 | 0.99 | 0.94
2, 2, 16, 2 | 1920 | 0.98 | 0.85
2, 2, 14, 2 | 1792 | 0.90 | 0.81
2, 2, 16, 2 | 1152 | 0.81 | 0.78
2, 2, 13, 2 | 1536 | 0.74 | 0.76
2, 2, 12, 2 | 1536 | 0.62 | 0.74
2, 2, 11, 2 | 1536 | 0.52 | 0.72
TABLE III: Swin Base execution path configurations corresponding to various
levels of resource utilization and model accuracy, as shown in Figure 7.
Depths corresponds to number of encoder layers in Stage 0 through Stage 3 from
left to right.

In Swin-Tiny, the smallest parameterization of Swin Transformer, the fpn
bottleneck Conv2D, fpn convs 0 Conv2D and fpn convs 1 Conv2D layers comprise
the majority of the total number of FLOPs, at $65\%$, $16\%$, and $4\%$ for a
512$\times$512 image, respectively, as shown in Figure 4. The fpn bottleneck
Conv2D layer in Swin is similar to the Conv2DFuse layer in SegFormer B2 as it
is also a large convolution in the decoder fusing information at various image
resolutions from the encoder stage outputs. The fpn bottleneck Conv2D layer is
computationally dominant for this model across various image sizes, as shown
in Figure 5. Thus, similar to the SegFormer case, we experiment with varying
the input and output channels in these layers along with various combinations
of bypassing encoder layers. As with SegFormer-B2, we propagate this skipped
computation backwards through the model to the extent possible to determine
the trade-off between model accuracy and execution time as well as energy.

Skipping even a few encoder layers in Swin-Tiny leads to a higher relative
drop in model accuracy compared to execution time, as shown in Figure 7.
Intuitively, we observing that increasing the batch size pushes this curve to
the left and that with a batch size of 16, we can save 27% of the execution
time for these dynamic model configurations. Compared to SegFormer B2, Swin-
Tiny has 25% fewer encoder layers and a significantly higher proportion of
FLOPs in the decoder (89% vs 68%). Thus, it appears that the encoder layers do
not contain as much redundant information as the SegFormer encoder and that
the model accuracy is more sensitive when bypassing self-attention layers in
the encoder. We observe that as a result, the resiliency in the encoder with
SegFormer does not carry over to the Swin Model architecture (despite Swin-
Tiny overall requiring three times more parameters). In addition, since most
of the computation is in the decoder, skipping computation associated with
input channels in only a few convolution layers does not lead to significant
execution time savings after a focused design space exploration.

We also observe that Swin Base, the largest Swin model, which requires twice
as many parameters as Swin Tiny, is more resilient to skipping computation
associated with input channels in fpn bottleneck Conv2D, as shown in Figure 7.
This layer has the same number of input channels (2048) in both Swin Tiny and
Swin Base, but Swin Base has 18 layers in the third encoder stage compared to
only two layers in Swin Tiny. Thus, we can bypass encoder layers as well as we
did in SegFormer to achieve a better tradeoff curve. Selected dynamic model
configurations for Swin Base corresponding to various accuracy thresholds are
shown in Table III. For Swin Base, we find that as with SegFormer, the
marginal drop in accuracy is less than the margin drop in execution time with
these Pareto-optimal points, as can be seen with the light dashed $y=x$
reference line in Figure 7.

As with SegFormer, for reference, we plot the original Swin Tiny, Small, and
Base Models (labeled as Trained Swin Models) as large squares in Figure 7 and
see that if an application requires more than 20% of execution time savings,
it is advantageous to switch models from Swin Base to Swin Tiny at that point.
However, there is not a clear advantage to switch from Swin Base to Swin Small
as dynamically pruning Swin Base without any training is still competitive
with the Swin Small execution time.

##  IV Dynamic Real-time Inference Engine

Figure 8: Our dynamic real-time inference engine.

Our DRT inference engine, shown in Figure 8, focuses on maximizing accuracy
given a resource constraint, instead of minimizing resource utilization given
an accuracy target as in most prior static and dynamic work on efficient
inference. Given a precomputed accuracy versus resource utilization estimate,
for every inference, an image and resource utilization target are input. The
dynamic inference algorithm chooses the model configuration that meets the
resource utilization target while maximizing accuracy and runs inference using
the corresponding model execution path. The engine outputs an estimate of the
resulting model accuracy when constrained to meet the input resource target
and the output image.

Using the accuracy versus resource utilization tradeoff curves from Section
III, we build a lookup table (LUT) that contains Pareto optimal model
configurations given a resource constraint (the ’A’ block in Figure 8).111If
available resources vary widely, it might be necessary to have multiple model
parameters available in the engine. For example, one model configuration may
bypass one layer in the encoder and reduce the number of input channels by 20%
for a layer in the decoder. Then, the dynamic inference algorithm (D in Figure
8) can look up the appropriate configuration that meets the resource target
input and launch it. Finally, the engine outputs the model accuracy estimate
from the LUT and the output image from the model execution. Since our
implementation uses readily-available pretrained models, this information can
be pre-determined for a given model. Thus, we minimally add the cost of this
look up to the model execution.

Importantly, this approach requires no additional training or training data
and models do not need to be trained any differently for use with our DRT
inference engine. As a result, we are able to generate the LUT information
using only inference experiments. Importantly, these experiments are much
faster compared to the overhead of a training a model. For example, in the
time that it takes to train the original SegFormer B2 model [71] on an NVIDIA
TITAN V GPU, we can run over 800 inference experiments with modified execution
paths for this model, using a standard 2000 images from the ADE20K dataset
[84]. Since we directly use pretrained models, we build an extensible engine
that can be easily used for models developed in the future as well. Finally,
we note that our approach still allows inference to be performed with the
original model without any modification when there are enough resources
available.

##  V Accelerator

Figure 9: MAGNet [59] block diagram that we used for our accelerator. The
system view is shown on the left, with the PE shown in detail on the right.

⬇

# temporal tiling at PE array

for k2 in range(K2):

for p2 in range(P2):

for q2 in range(Q2):

# number of PEs

parallel_for p2s in range(P2S):

parallel_for q2s in range(Q2S):

parallel_for k2s in range(K2S):

parallel_for c2s in range(C2S):

# temporal tiling in PE

for p1 in range(P1):

for q1 in range(Q1):

for k1 in range(K1):

# output stationary: R * S * C1

for r in range(R):

for s in range(S):

for c1 in range(C1):

# local weight stationary: Q0

for q0 in range(Q0):

# number of vector MACs

parallel_for k0 in range(K0):

# vector size

parallel_for c0 in range(C0):

Listing 1: Scheduling and tiling loop nests for executing layers in our
accelerator (please see Figure 9). We allow for spatial and temporal tiling
and use an output-stationary local-weight-stationary dataflow.

We use the MAGNet framework [59] for our accelerator experiments since it
provides a configurable deep neural network accelerator template that has
previously been successfully used for CNNs [59] and BERT [28]. As shown in
Figure 9, we have three levels of compute with vector MACs, processing
elements (PEs), and the PE array. For a convolution layer, we refer to the
number of output channels as K, the number of input channels as C, the output
image height as P, the output image width as Q, the kernel width as R and the
kernel height as S. We allow for spatial and temporal tiling in the K and C
dimensions as well as the P and Q dimensions, if needed, as shown in Listing
1. Each vector MAC allows for the parallel execution of C0 multiplications,
where C0 $*$ C1 $*$ C2 is the total number of input channels. Then, with K0
vector MACs in each PE, there are C0 $*$ K0 parallel operations executed in
each PE on every cycle, as shown in the innermost parallel loops of Listing 1.
Note that K0 $*$ K1 $*$ K2 is the total number of output channels. Our memory
hierarchy has four levels with small register files within each vector MAC to
enable weight and output partial sum reuse at this level of compute, buffers
within each PE for storing inputs, weights, and outputs, a global buffer at
the PE array level, and an off-chip DRAM. The inputs are shared across the
vector MACs with one input buffer PE while there are K0 weight buffers in a PE
with one weight buffer per vector MAC. We use INT-8 quantization as shown in
Figure 9.

We use an output-stationary local-weight-stationary (OS-LWS) dataflow to
enable weight reuse at the lowest level of the compute hierarchy in the vector
MAC. With this multi-level dataflow, we compute Q0 partial sums by temporally
reusing weights Q0 times (LWS) and reusing these partial sums R $*$ S $*$ C1
times (OS), as shown in Listing 1. This dataflow choice is particularly useful
for our linear transformation layers, which multiply a large set of inputs by
the same set of shared weights. We allow for temporal tiling to reuse weights
in the weight buffer (captured by the P1 $*$ Q1 loops) and input activations
in the input buffer (captured by K1 loop) to produce a tile of output image of
height P1, width Q1 $*$ Q0 and K0 $*$ K1 output channels. We exploit data and
model parallelism across P, Q, C, and K dimensions to parallelize across
multiple PEs. We further temporally tile the weights (split by output channels
with K2 loop) and activations (split by height and width with the P2 and Q2
loops, respectively) if needed.

Thus, we can map a convolutional layer by specifying the input image height
and width, the input and output channels, and the kernel size and stride in
each direction. Note that the output image height and width can be computed
from these parameters. We include padding in the input image height and width.
To execute a matrix multiplication $A*B=C$, where $A$ is $m$ by $n$, $B$ is
$n$ by $o$ and $C$ is $m$ by $o$, we set the input image dimensions to $1$ by
$m$, the kernel size to $1$ by $1$, the input channels to $n$, the output
channels to $o$, and the output image dimensions to $1$ by $m$. This
parameterization captures the computation required for matrix multiplications.
For ReLU and pooling layers, each PE also contains a post-processing unit that
fuses these layers with convolutions when required.

Given this base architecture, we conduct an accelerator design space
exploration across the vectorization of compute units and the sizes of the
buffers in each PE exploring whether sizing resources for accelerators
targeting specific dynamic model configurations instead of the full model
execution affects the optimal architecture.

To achieve optimal performance, we consider three optimizations. First, we
exploit model-level parallelism outside of self-attention layers, where layers
in the decoder can execute in parallel with later encoder stages for semantic
segmentation models (e.g., in SegFormer B2, Stage 1 Overlap Patch Embedding
Conv2D can be executed in parallel with Decoder Linear that consumes the
output of Stage 0, as labeled in Figure 2). Second, for larger layers whose
computation cannot be split and contained within individual PEs, we allow for
cross-PE reduction so that outputs from a PE can be sent to another PE. This
decision results in a small increase in energy but allows for executing larger
layers in model configuration that do not fit on accelerators targeting
smaller model configurations. Third, we adjust the limit of the tiling size
which affects the maximum bound possible for the temporal tiling parameters P1
and Q1 in Listing 1 to allow for temporal tiling for larger layers.

We enforce that a constant number of MACs (16384) can be computed in parallel
for fair comparison of accelerators with different parameterizations targeting
different model configurations. We vary how these are split across vector
depth in each vector MAC, the number of vector MACs in each PE, and the number
of PEs. For example e.g., K0 = C0 = 16 with a 8 by 8 PE array and K0 = C0 = 32
with a 4 by 4 PE array are both valid parameterizations for our compute. This
flexibility helps explore the tradeoff between the divisibility of layer sizes
among the vectorization factor and the energy cost of fetching more
instructions with less vectorization.

##  VI Evaluation

We present post-synthesis results in TSMC5nm for executing SegFormer B2 and
Swin Tiny for semantic segmentation with our dynamic real-time inference
approach as described in Sections III and IV as well as for executing the OFA
(OFA) ResNet-50 parameterizations for object detection models such as DETR and
Deformable DETR that heavily rely on the ResNet-50 backbone. We denote weight
memory depth (per PE) as WM and activation memory depth (per PE) as AM and
continue using K0 to represent the number of vector MACs per PE and C0 to
represent the number of multiplications computed in parallel in a vector MAC.
We compare various vectorization factors and memory sizes and limit our
largest accelerator to be one that requires no temporal tiling of the weights
(i.e., K2 = 0 in Listing 1).

###  VI-A SegFormer B2

Figure 10: Execution time and total energy distribution across layers in
SegFormer B2 model [71] for batch size of one on accelerator with K0 $=32$, C0
$=32$, WM $=1024$ kB, and AM $=64$ kB. Figure 11: Energy per FLOP when
executing SegFormer B2 on accelerator with on accelerator with K0 $=32$, C0
$=32$, WM $=1024$ kB, and AM $=64$ kB. Figure 12: Tradeoff between normalized
mIoU (accuracy) and cycles on accelerators when dynamically pruning pretrained
ADE SegFormer B2 [71]. Cycles can be a dynamic resource constraint. The
accelerators have K0 = 32, C0 = 32, and AM = 64 kB, with WM labeled in the
legend. Table II has the dynamic model configurations for labeled points.
Figure 13: Tradeoff between normalized mIoU (accuracy) and total energy on
accelerators when dynamically pruning ADE SegFormer B2 [71]. Energy can be a
dynamic resource constraint. Total energy is normalized to the energy for the
Conv2DFuse layer, which required the most energy. The accelerators have K0 =
32, C0 = 32, and AM = 64 kB, with WM labeled in the legend. Table II has the
dynamic model configurations for labeled points. Figure 14: Normalized total
energy comparison for executing SegFormer B2 model [71] on accelerators with
different K0, C0, WM, and AM. Data labels are WM, AM. K0 = C0 = 32
accelerators have lower total energy.

From our design space exploration, we find that the accelerator with the
lowest latency and energy targeting the full model (point A in Table II) is
with K0 $=32$, C0 $=32$, WM $=1024$ kB, and AM $=64$ kB. We denote this as
accelerator_A. We found that the Conv2DFuse layer was the limiting layer in
sizing the weight memory to ensure that all weights were stored on-chip,
without temporal tiling of the weights. We report the execution time
distribution and energy distribution of the layers when executed on
accelerator_A in Figure 10. We observe that execution time and energy
distribution is much closer to the FLOPs distribution shown in Figure 3.

The total execution requires 4415208 cycles with a synthesized clock frequency
of 1.25 GHz, so the total execution time on accelerator_A is 3.5ms. Thus,
accelerator_A is 16.6X faster than the 58ms for the SegFormer B2 execution on
the NVIDIA TITAN V GPU. We execute the dynamic model configurations for the
ADE20K model in Table II on accelerator_A as shown in Figure 12. We observe
that there is a better tradeoff between accuracy and execution time with
accelerator_A compared to the TITAN V GPU. For example, with a 2% drop in
accuracy, accelerator_A enables saving 20% instead of 11% of execution time
when executing point B compared to the GPU.

When looking at energy per FLOP as shown in Figure 11, we observe that five
convolution layers have significantly higher energy per FLOP compared to all
the other layers in the model. These outliers can be attributed to a very
small number of input channels (three for the input channels for an input
image to the model in PatchEmbed1_Conv2D and one for the DWConv layers due to
how we incorporate parallelism in this depthwise convolution in how our model)
and low utilization along C0 (Listing 1), which we use to determine how many
multiplications to execute in parallel in each vector MAC. Since these layers
constitute 17% of the total energy, we do not focus on optimizing this number
further.

Figure 14 shows that more vectorization inside our vector MACs and inside our
PEs results in lower energy for the full model execution, so we focus on K0 =
C0 = 32 accelerators.

We next consider an accelerator with comparable execution time and energy but
smaller area, with K0 $=32$, C0 $=32$, WM $=128$ kB, and AM $=64$ kB. We
denote this as accelerator*. We see a very similar FLOPs and execution time
distribution for accelerator* as shown in Figure 10 for accelerator_A. We
observe that the PE array area of accelerator* is 4.3X smaller compared to
accelerator_A and the full model execution is less than 3% slower and consumes
0.5% more energy. We note that given that both accelerator_A and accelerator*
are capable of computing the same number of MACs in parallel, as by experiment
design, the area in accelerator_A is dominated by the weight memories. For
point G, which has 50% of the number of FLOPs compared to point A, we observe
that the execution is 5% slower (Figure 12) and consumes 2.7% more energy
(Figure 13). From Figure 12, we see that the optimal architecture is the same
among various dynamic configurations with different model complexity when
comparing execution time. In Figure 13, we see that the accelerator
architecture does not affect the total energy consumption because the number
of MACs executed are the same given a dynamic configuration.

The total execution for point A requires 4540195 cycles, translating into a
total execution time of 3.6ms, which is 16X faster than the NVIDIA TITAN V
GPU. We also see from Figures 12 and 13 that we can efficiently execute
dynamic model configurations for SegFormer with one set of model weights on
this hardware with dynamic execution time or energy constraints. We observe
that the limiting layer when sizing the weight memory becomes the Overlap
Patch Embedding Conv2D layers (Figure 2) when executing smaller dynamic model
configurations like point G since the weights required for point G’s
Conv2DFuse are six times smaller compared to the full model (point A)
Conv2DFuse layer. Thus, we conclude that it is advantageous to use
accelerator* to target dynamic real-time inference for this model due to the
significant area savings and low increase in execution time and energy, and
that scaling the accelerator architecture for more complicated configurations
does not significantly improve performance.

###  VI-B Swin-Tiny

Figure 15: Execution time distribution across layers in Swin Tiny model [37]
for batch size of one on accelerator with K0 $=32$, C0 $=32$, WM $=128$ kB,
and AM $=64$ kB.

Following from our accelerator results for SegFormer, we execute Swin-Tiny on
accelerator*. The total execution requires 15482594 cycles (12.4ms), which is
17X faster than the 215ms on the NVIDIA TITAN V GPU. Similarly to the
SegFormer results, we find that the execution time distribution, as shown in
Figure 15, closely matches the FLOPs distribution for this model from Section
II. As noted earlier, fpn_bottleneck_conv2D in Swin serves a similar purpose
to SegFormer’s Conv2DFuse and takes the majority of the execution time. In
fact, 89% of the execution time on the accelerator was spent on convolutions,
primarily in the decoder head. We can similarly execute various dynamic model
configurations, including the full model configuration on accelerator* and
find that the tradeoff between model accuracy and execution time again
improves compared to the GPU results.

###  VI-C Once-For-All ResNet-50

|  |  | PE Array | Normalized
---|---|---|---|---
Accelerator | WM (kB) | AM (kB) | Area ($mm^{2}$) | Energy
OFA1 | 1024 | 64 | 8.33 | 16.5
OFA2 | 128 | 64 | 2.26 | 14.3
OFA3 | 64 | 32 | 1.66 | 14.6
TABLE IV: Accelerator parameterizations with K0 = 32 and C0 = 32 considered
for OFA ResNet-50 models [6].  Figure 16: Tradeoff between normalized mIoU
(accuracy) and cycles on accelerators when executing various OFA ResNet-50
model parameterizations [6]. Cycles (or execution time) can be a dynamic
resource constraint. The accelerators have K0 = 32 and C0 = 32.

We accelerate various OFA models [6] for ResNet-50 [26] since it is the
computationally dominant backbone for state-of-the-art object detection
models, as shown in Section II. Note that we use 640 by 480 image sizes
corresponding to the COCO dataset [34] for object detection with OFA instead
of ImageNet, which is typically used for image classification with ResNet-50.
We compare three accelerator parameterizations for these models, as summarized
in Table IV. Figure 16 shows that accelerator_OFA1 has the lowest execution
time for all configurations. However, accelerator_OFA2 and accelerator_OFA3
are only 1.5% (for the most accurate OFA model) to 4.5% (for the least
accurate OFA model shown) slower and they are 3.7X and 5X smaller than
accelerator_OFA1. In addition, Table IV shows that the energy is slightly
higher for accelerator_OFA1 compared to the other two options, likely due to
the larger memories. Since all of these accelerators are capable of executing
the same number of MACs in parallel for fair comparison, we note that the
memories dominate total PE array area for accelerator_OFA3.

Importantly, we see the benefit of using OFA ResNet-50 to enable dynamic real-
time inference for object detection models. On accelerator_OFA2, this approach
saves 57% of the execution time with less than a 5% drop in accuracy. Note
that accelerator_OFA2 has the same parameterization as accelerator* from the
previous section, so accelerator* serves as one accelerator capable of
efficiently executing SegFormer, Swin Transformer, and OFA ResNet-50 models.

We observe that similarly to the segmentation models, there are two layers
(the input layer and the classification head) that have significantly higher
energy per FLOP compared to all the other layers in the full OFA due to a low
number of input channels and underutilization in the C0 dimension.
Fortunately, these layers contribute 13% of the total energy, so we do not
focus on optimizing this number further in this case as well. Finally, we
observe that the execution time and energy breakdown are not dominated by
single layers in OFA models as was true for the segmentation models. Given
that OFA is based on ResNet-50, these distributions follow based on the size
of the convolution layers.

##  VII Related Work

###  VII-A Efficient Inference Approaches

Given a base model, prior work produces either more efficient static models or
dynamic models for execution time and energy savings. Static approaches have
focused on CNNs and BERT and retrain models with pruning and scaling [6, 14,
17, 23, 41, 60, 61, 74], knowledge distillation [25, 45, 51, 56], and
quantization [4, 27, 39, 46, 67, 69, 79]. Since the same model is used for
each inference, the model execution does not adapt to real-time application
scenarios when resources are further constrained or more resources are
available, allowing for more accurate inference. Static techniques can be used
with our approach to improve the tradeoff between model accuracy and resource
utilization as we found that it may be more advantageous to switch to trained
models after 20% to 50% of execution time savings depending on the pretrained
model used. To demonstrate the effectiveness of such an approach, we use the
Once-For-All [6] ResNet-50 model parameterizations in Section VI to explore
the impact of using such a technique to enable dynamic execution on the
accelerator design space.

On the dynamic side, adaptive computation time approaches train an additional
network that uses the output from preceding layers to decide whether to exit
early at each layer [12, 21, 36, 19]. BranchyNet [54] trains early exit paths
added in a model. SkipNet [65] trains a gating network to decide which
convolutional layers to skip based on the input image. DeeBERT [73] trains
extra classification layers between each transformer layer to enable early
exit. Patience-based BERT [85] augments each layer with an internal classifier
and ends inference when the intermediate predictions of these classifiers
stays the same for a fixed number of steps. EdgeBERT [53] uses dynamic
voltage-frequency scaling to enable early exit.

These dynamic approaches have focused on reducing execution time or energy for
CNNs and BERT while maintaining accuracy for inputs that do not require the
full model execution. We instead enforce that a given input resource
constraint is met regardless of the input and allow for trading average
accuracy to achieve this goal, as shown in Section IV. These prior works also
introduce more model parameters and require retraining the model or training
an additional model to dynamically make early-exit decisions, while our
approach considers the tradeoffs between no additional training and retraining
a subset of existing model parameters, as shown in Section III.

###  VII-B Vision Transformer Accelerators

Prior work on accelerating vision transformers focuses on efficiently
executing attention in image classification models. DiViT [32] for ViT [16]
exploits patch locality to reduce redundant computation in attention layers.
Other work [66] removes redundant attention heads and FFN layers in DeiT [56].
ViTCoD [76] considers sparsity-accuracy tradeoffs to accelerate DeiT [56],
LeViT [20], and Strided Transformer [31]. One paper accelerates the Swin [37]
encoder backbone for image classification, while we also accelerate the
UPerNet decoder head [70] for semantic segmentation. Thus, this prior work
focuses on multi-head self-attention and fully-connected layers instead of
convolutions, which we find to be the bottleneck on the NVIDIA Titan V GPU and
our accelerator. On the FPGA side, a survey overviews FPGA works for object
detection, but these papers do not support DETR or DETR-based models and
accelerate older models that require more complicated components [80]. While
these works target vision models that heavily rely on self-attention or now
out-dated custom detection components, we target state-of-the-art detection
and segmentation models that rely on efficiently executing convolutions and
self-attention, as shown in Section II. Thus, solely optimizing self-attention
will lead to limited improvement and such works are complementary to our work.

Other work focuses on accelerating softmax, GeLU, and normalization layers in
BERT, which minimally contribute to the models we consider [58, 48, 30, 78].
Further work focuses on quantization for BERT [38] and ViT [50], pruning [33,
42], and clustering [52] methods for more efficient inference. These works are
complementary to our approach and we do use INT-8 quantization in our
accelerator in Section V. Additionally, we present a dynamic pruning approach
instead of static pruning in Section IV for more efficient inference.

##  VIII Conclusion

While state-of-the-art vision transformer models for object detection and
semantic segmentation are typically large and designed for static execution,
real-time applications that perform these tasks require dynamic models that
can adhere to strict constraints and adapt to varying workloads. Unlike prior
work that focuses on shortening execution time or energy based on the input
image and the convergence of internal predictions, we adapt such models for
dynamic inference to meet an input resource constraint for any input image.
Unlike early vision transformers such as BERT and ViT, we find that state-of-
the-art object detection models spend most of their execution time on a CNN
backbone before the transformer execution while state-of-the-art segmentation
models have shifted to replacing attention layers with convolutions in the
decoder for higher performance. With this insight, we explore the resiliency
of pretrained models in attention and convolution layers, and find that
depending on the model, we can save 20% to 50% of execution time by
dynamically pruning with one set of model weights before switching to
retrained models to maintain accuracy. Finally, we build an efficient
accelerator capable of executing dynamic real-time inference for vision
transformer models 17X times faster than a NVIDIA TITAN V GPU. We find that
the model complexity does not change the underlying optimal accelerator
architecture since we trade at most 5% of execution time and energy for four
times smaller hardware. Our work is one of the first works considering
hardware acceleration for non-attention-dominant vision transformers and helps
bridge the gap between large state-of-the-art vision transformer models and
deployment in resource-constrained real-time systems.

## References

  * [1] “MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark,” https://github.com/open-mmlab/mmsegmentation, 2020.
  * [2] M. Alam, M. D. Samad, L. Vidyaratne, A. Glandon, and K. M. Iftekharuddin, “Survey on deep neural networks in speech and vision systems,” _Neurocomputing_ , vol. 417, pp. 302–321, 2020.
  * [3] C. Alippi, V. Piuri, and M. Sami, “Sensitivity to errors in artificial neural networks: A behavioral approach,” _IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications_ , vol. 42, no. 6, pp. 358–361, 1995.
  * [4] H. Bai, W. Zhang, L. Hou, L. Shang, J. Jin, X. Jiang, Q. Liu, M. Lyu, and I. King, “Binarybert: Pushing the limit of bert quantization,” _arXiv preprint arXiv:2012.15701_ , 2020.
  * [5] A. Bosio, P. Bernardi, A. Ruospo, and E. Sanchez, “A reliability analysis of a deep neural network,” in _2019 IEEE Latin American Test Symposium (LATS)_. IEEE, 2019, pp. 1–6.
  * [6] H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han, “Once-for-all: Train one network and specialize it for efficient deployment,” _arXiv preprint arXiv:1908.09791_ , 2019.
  * [7] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European conference on computer vision_. Springer, 2020, pp. 213–229.
  * [8] C.-F. R. Chen, Q. Fan, and R. Panda, “Crossvit: Cross-attention multi-scale vision transformer for image classification,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 357–366.
  * [9] K. Chen, J. Wang, J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu, Z. Zhang, D. Cheng, C. Zhu, T. Cheng, Q. Zhao, B. Li, X. Lu, R. Zhu, Y. Wu, J. Dai, J. Wang, J. Shi, W. Ouyang, C. C. Loy, and D. Lin, “MMDetection: Open mmlab detection toolbox and benchmark,” _arXiv preprint arXiv:1906.07155_ , 2019.
  * [10] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [11] X. Dai, Y. Chen, J. Yang, P. Zhang, L. Yuan, and L. Zhang, “Dynamic detr: End-to-end object detection with dynamic attention,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2988–2997.
  * [12] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and Ł. Kaiser, “Universal transformers,” _arXiv preprint arXiv:1807.03819_ , 2018.
  * [13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [14] Z. Dong, Y. Gao, Q. Huang, J. Wawrzynek, H. K. So, and K. Keutzer, “Hao: Hardware-aware neural architecture optimization for efficient inference,” in _2021 IEEE 29th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM)_. IEEE, 2021, pp. 50–59.
  * [15] F. F. dos Santos, P. F. Pimenta, C. Lunardi, L. Draghetti, L. Carro, D. Kaeli, and P. Rech, “Analyzing and increasing the reliability of convolutional neural networks on gpus,” _IEEE Transactions on Reliability_ , vol. 68, no. 2, pp. 663–677, 2018.
  * [16] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [17] A. Fan, E. Grave, and A. Joulin, “Reducing transformer depth on demand with structured dropout,” _arXiv preprint arXiv:1909.11556_ , 2019.
  * [18] D. Feng, C. Haase-Schütz, L. Rosenbaum, H. Hertlein, C. Glaeser, F. Timm, W. Wiesbeck, and K. Dietmayer, “Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 3, pp. 1341–1360, 2020.
  * [19] M. Figurnov, M. D. Collins, Y. Zhu, L. Zhang, J. Huang, D. Vetrov, and R. Salakhutdinov, “Spatially adaptive computation time for residual networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1039–1048.
  * [20] B. Graham, A. El-Nouby, H. Touvron, P. Stock, A. Joulin, H. Jégou, and M. Douze, “Levit: a vision transformer in convnet’s clothing for faster inference,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 12 259–12 269.
  * [21] A. Graves, “Adaptive computation time for recurrent neural networks,” _arXiv preprint arXiv:1603.08983_ , 2016.
  * [22] K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu, Z. Yang, Y. Zhang, and D. Tao, “A survey on vision transformer,” _IEEE transactions on pattern analysis and machine intelligence_ , 2022.
  * [23] S. Han, H. Mao, and W. Dally, “Compressing deep neural networks with pruning, trained quantization and huffman coding. arxiv 2015,” _arXiv preprint arXiv:1510.00149_ , 2015.
  * [24] A. Hatamizadeh, H. Yin, J. Kautz, and P. Molchanov, “Global context vision transformers,” _arXiv preprint arXiv:2206.09959_ , 2022.
  * [25] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , vol. 2, no. 7, 2015.
  * [26] S. Jastrzębski, D. Arpit, N. Ballas, V. Verma, T. Che, and Y. Bengio, “Residual connections encourage iterative inference,” _arXiv preprint arXiv:1710.04773_ , 2017.
  * [27] X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, “Tinybert: Distilling bert for natural language understanding,” _arXiv preprint arXiv:1909.10351_ , 2019.
  * [28] B. Keller, R. Venkatesan, S. Dai, S. G. Tell, B. Zimmer, W. J. Dally, C. T. Gray, and B. Khailany, “A 17–95.6 tops/w deep learning inference accelerator with per-vector scaled 4-bit quantization for transformers in 5nm,” in _2022 IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits)_. IEEE, 2022, pp. 16–17.
  * [29] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” _ACM Computing Surveys (CSUR)_ , 2021\.
  * [30] S. A. Koohpayegani and H. Pirsiavash, “Sima: Simple softmax-free attention for vision transformers,” _arXiv preprint arXiv:2206.08898_ , 2022.
  * [31] W. Li, H. Liu, R. Ding, M. Liu, P. Wang, and W. Yang, “Exploiting temporal contexts with strided transformer for 3d human pose estimation,” _IEEE Transactions on Multimedia_ , 2022.
  * [32] Y. Li, Y. Hu, F. Wu, and K. Li, “Divit: Algorithm and architecture co-design of differential attention in vision transformer,” _Journal of Systems Architecture_ , p. 102520, 2022.
  * [33] Z. Li, S. Ghodrati, A. Yazdanbakhsh, H. Esmaeilzadeh, and M. Kang, “Accelerating attention through gradient-based learned runtime pruning,” in _Proceedings of the 49th Annual International Symposium on Computer Architecture_ , 2022, pp. 902–915.
  * [34] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [35] S. Liu, F. Li, H. Zhang, X. Yang, X. Qi, H. Su, J. Zhu, and L. Zhang, “Dab-detr: Dynamic anchor boxes are better queries for detr,” _arXiv preprint arXiv:2201.12329_ , 2022.
  * [36] W. Liu, P. Zhou, Z. Zhao, Z. Wang, H. Deng, and Q. Ju, “Fastbert: a self-distilling bert with adaptive inference time,” _arXiv preprint arXiv:2004.02178_ , 2020.
  * [37] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 012–10 022.
  * [38] Z. Liu, G. Li, and J. Cheng, “Hardware acceleration of fully quantized bert for efficient natural language processing,” in _2021 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, 2021, pp. 513–516.
  * [39] Z. Liu, Y. Wang, K. Han, W. Zhang, S. Ma, and W. Gao, “Post-training quantization for vision transformer,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [40] M. Merenda, C. Porcaro, and D. Iero, “Edge machine learning for ai-enabled iot devices: A review,” _Sensors_ , vol. 20, no. 9, p. 2533, 2020.
  * [41] P. Michel, O. Levy, and G. Neubig, “Are sixteen heads really better than one?” _Advances in neural information processing systems_ , vol. 32, 2019\.
  * [42] H. Peng, S. Huang, T. Geng, A. Li, W. Jiang, H. Liu, S. Wang, and C. Ding, “Accelerating transformer-based deep learning models on fpgas using column balanced block pruning,” in _2021 22nd International Symposium on Quality Electronic Design (ISQED)_. IEEE, 2021, pp. 142–148.
  * [43] V. Piuri, “Analysis of fault tolerance in artificial neural networks,” _Journal of Parallel and Distributed Computing_ , vol. 61, no. 1, pp. 18–48, 2001.
  * [44] B. Roh, J. Shin, W. Shin, and S. Kim, “Sparse detr: Efficient end-to-end object detection with learnable sparsity,” _arXiv preprint arXiv:2111.14330_ , 2021.
  * [45] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” _arXiv preprint arXiv:1910.01108_ , 2019.
  * [46] S. Shen, Z. Dong, J. Ye, L. Ma, Z. Yao, A. Gholami, M. W. Mahoney, and K. Keutzer, “Q-bert: Hessian based ultra low precision quantization of bert,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 05, 2020, pp. 8815–8821.
  * [47] M. Siam, M. Gamal, M. Abdel-Razek, S. Yogamani, M. Jagersand, and H. Zhang, “A comparative study of real-time semantic segmentation for autonomous driving,” in _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , 2018, pp. 587–597.
  * [48] J. R. Stevens, R. Venkatesan, S. Dai, B. Khailany, and A. Raghunathan, “Softermax: Hardware/software co-design of an efficient softmax for transformers,” in _2021 58th ACM/IEEE Design Automation Conference (DAC)_. IEEE, 2021, pp. 469–474.
  * [49] R. Strudel, R. Garcia, I. Laptev, and C. Schmid, “Segmenter: Transformer for semantic segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 7262–7272.
  * [50] M. Sun, H. Ma, G. Kang, Y. Jiang, T. Chen, X. Ma, Z. Wang, and Y. Wang, “Vaqf: Fully automatic software-hardware co-design framework for low-bit vision transformer,” _arXiv preprint arXiv:2201.06618_ , 2022.
  * [51] S. Sun, Y. Cheng, Z. Gan, and J. Liu, “Patient knowledge distillation for bert model compression,” _arXiv preprint arXiv:1908.09355_ , 2019.
  * [52] H. Tabani, A. Balasubramaniam, S. Marzban, E. Arani, and B. Zonooz, “Improving the efficiency of transformers for resource-constrained devices,” in _2021 24th Euromicro Conference on Digital System Design (DSD)_. IEEE, 2021, pp. 449–456.
  * [53] T. Tambe, C. Hooper, L. Pentecost, T. Jia, E.-Y. Yang, M. Donato, V. Sanh, P. Whatmough, A. M. Rush, D. Brooks, and G.-Y. Wei, “Edgebert: Sentence-level energy optimizations for latency-aware multi-task nlp inference,” in _MICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture_ , 2021, pp. 830–844.
  * [54] S. Teerapittayanon, B. McDanel, and H.-T. Kung, “Branchynet: Fast inference via early exiting from deep neural networks,” in _2016 23rd International Conference on Pattern Recognition (ICPR)_. IEEE, 2016, pp. 2464–2469.
  * [55] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jegou, “Training data-efficient image transformers & distillation through attention,” in _International Conference on Machine Learning_ , vol. 139, July 2021, pp. 10 347–10 357.
  * [56] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 10 347–10 357.
  * [57] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [58] I. Vasyltsov and W. Chang, “Efficient softmax approximation for deep neural networks with attention mechanism,” _arXiv preprint arXiv:2111.10770_ , 2021\.
  * [59] R. Venkatesan, Y. S. Shao, M. Wang, J. Clemons, S. Dai, M. Fojtik, B. Keller, A. Klinefelter, N. Pinckney, P. Raina, Y. Zhang, B. Zimmer, W. J. Dally, J. Emer, S. W. Keckler, and B. Khailany, “Magnet: A modular accelerator generator for neural networks,” in _2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_. IEEE, 2019, pp. 1–8.
  * [60] E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov, “Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned,” _arXiv preprint arXiv:1905.09418_ , 2019.
  * [61] H. Wang, Z. Wu, Z. Liu, H. Cai, L. Zhu, C. Gan, and S. Han, “Hat: Hardware-aware transformers for efficient natural language processing,” _arXiv preprint arXiv:2005.14187_ , 2020.
  * [62] W. Wang, T. Zhou, F. Porikli, D. Crandall, and L. Van Gool, “A survey on deep learning technique for video segmentation,” _arXiv preprint arXiv:2107.01153_ , 2021.
  * [63] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 568–578.
  * [64] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, “Pvt v2: Improved baselines with pyramid vision transformer,” _Computational Visual Media_ , vol. 8, no. 3, pp. 415–424, 2022.
  * [65] X. Wang, F. Yu, Z.-Y. Dou, T. Darrell, and J. E. Gonzalez, “Skipnet: Learning dynamic routing in convolutional networks,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 409–424.
  * [66] X. Wang, L. L. Zhang, Y. Wang, and M. Yang, “Towards efficient vision transformer inference: a first study of transformers on mobile devices,” in _Proceedings of the 23rd Annual International Workshop on Mobile Computing Systems and Applications_ , 2022, pp. 1–7.
  * [67] Y. Wei, X. Pan, H. Qin, W. Ouyang, and J. Yan, “Quantization mimic: Towards very tiny cnn for object detection,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 267–283.
  * [68] A. W. Wong, “torchprof,” 12 2020, https://github.com/awwong1/torchprof. [Online]. Available: https://github.com/awwong1/torchprof
  * [69] J. Wu, C. Leng, Y. Wang, Q. Hu, and J. Cheng, “Quantized convolutional neural networks for mobile devices,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 4820–4828.
  * [70] T. Xiao, Y. Liu, B. Zhou, Y. Jiang, and J. Sun, “Unified perceptual parsing for scene understanding,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 418–434.
  * [71] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “Segformer: Simple and efficient design for semantic segmentation with transformers,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [72] E. Xie, W. Wang, W. Wang, P. Sun, H. Xu, D. Liang, and P. Luo, “Segmenting transparent object in the wild with transformer,” _arXiv preprint arXiv:2101.08461_ , 2021.
  * [73] J. Xin, R. Tang, J. Lee, Y. Yu, and J. Lin, “Deebert: Dynamic early exiting for accelerating bert inference,” _arXiv preprint arXiv:2004.12993_ , 2020\.
  * [74] T.-J. Yang, A. Howard, B. Chen, X. Zhang, A. Go, M. Sandler, V. Sze, and H. Adam, “Netadapt: Platform-aware neural network adaptation for mobile applications,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 285–300.
  * [75] Z. Yao, J. Ai, B. Li, and C. Zhang, “Efficient detr: improving end-to-end object detector with dense prior,” _arXiv preprint arXiv:2104.01318_ , 2021\.
  * [76] H. You, Z. Sun, H. Shi, Z. Yu, Y. Zhao, Y. Zhang, C. Li, B. Li, and Y. Lin, “Vitcod: Vision transformer acceleration via dedicated algorithm and accelerator co-design,” _arXiv preprint arXiv:2210.09573_ , 2022.
  * [77] H. Yu, Z. Yang, L. Tan, Y. Wang, W. Sun, M. Sun, and Y. Tang, “Methods and datasets on semantic segmentation: A review,” _Neurocomputing_ , vol. 304, pp. 82–103, 2018.
  * [78] J. Yu, J. Park, S. Park, M. Kim, S. Lee, D. H. Lee, and J. Choi, “Nn-lut: Neural approximation of non-linear operations for efficient transformer inference,” _arXiv preprint arXiv:2112.02191_ , 2021.
  * [79] O. Zafrir, G. Boudoukh, P. Izsak, and M. Wasserblat, “Q8bert: Quantized 8bit bert,” in _2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurIPS Edition (EMC2-NIPS)_. IEEE, 2019, pp. 36–39.
  * [80] K. Zeng, Q. Ma, J. W. Wu, Z. Chen, T. Shen, and C. Yan, “Fpga-based accelerator for object detection: A comprehensive survey,” _The Journal of Supercomputing_ , pp. 1–41, 2022.
  * [81] H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H.-Y. Shum, “Dino: Detr with improved denoising anchor boxes for end-to-end object detection,” _arXiv preprint arXiv:2203.03605_ , 2022.
  * [82] M. Zheng, P. Gao, R. Zhang, K. Li, X. Wang, H. Li, and H. Dong, “End-to-end object detection with adaptive clustering transformer,” _arXiv preprint arXiv:2011.09315_ , 2020.
  * [83] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr, and L. Zhang, “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 6881–6890.
  * [84] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, “Scene parsing through ade20k dataset,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 633–641.
  * [85] W. Zhou, C. Xu, T. Ge, J. McAuley, K. Xu, and F. Wei, “Bert loses patience: Fast and robust inference with early exit,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 18 330–18 341, 2020.
  * [86] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable detr: Deformable transformers for end-to-end object detection,” _arXiv preprint arXiv:2010.04159_ , 2020.
  * [87] Z. Zou, Z. Shi, Y. Guo, and J. Ye, “Object detection in 20 years: A survey,” _arXiv preprint arXiv:1905.05055_ , 2019.


# Tau Decays in Pythia $8$

Philip Ilten  School of Physics, University College Dublin

###### Abstract

As of version $8.150$ of Pythia, the isotropic decay model of $\tau$-leptons
has been replaced with sophisticated $\tau$-lepton decay machinery. The decays
and spin correlations for $\tau$-leptons in Pythia $8$ are described,
including the spin correlation algorithm, the available $\tau$-lepton
production processes, the $\tau$-lepton decay models, the user interface, and
the implementation.

###### keywords:

Monte Carlo , tau decays , polarization , hadronic currents

††journal: Nuclear Physics B Proceedings

##  1 Introduction

The role of $\tau$-leptons in Higgs boson measurements [1] and beyond the
Standard Model searches [2] is becoming increasingly important, due to the
enhanced coupling of the $\tau$-lepton in many of these physics models.
Consequently, it is necessary for current Monte Carlo generators to ensure
accurate modeling of $\tau$-lepton decays.

Prior to version $8.150$ of Pythia [3], $\tau$-lepton decays in Pythia were
performed using a leptonic or generic hadronic current matrix element without
including spin correlations, and more sophisticated $\tau$-lepton modeling was
only possible through external packages such as Tauola [4]. Now, in Pythia
version $8.150$ and above, fully modeled hadronic currents with spin
correlations are available, based on prior $\tau$-lepton modeling work in
Tauola and Herwig++ [5]. Currently, all known $\tau$-lepton decays with a
branching fraction greater than $0.04\%$ are modeled.

In this review, the spin correlation algorithm used in Pythia for
$\tau$-leptons is described, as well as the available $\tau$-lepton production
mechanisms, summarized in Table 1, and $\tau$-lepton decays, given in Table 2.
An introduction to the user interface, as well as the technical implementation
is given.

##  2 Correlations

The spin correlation algorithm used in Pythia for $\tau$-lepton decays is
based on the algorithm proposed by Collins [6] and Knowles [7], and expanded
by Richardson [8]. The algorithm separates spin correlations from the hard
process, parton shower, and hadronization phases of the Monte Carlo generator,
while maintaining full correlations, and can be broken into the following
steps.

  1. 1.

The $2\to n$ hard process is generated according to its matrix element
$\mathcal{M}$.

  2. 2.

One of the outgoing particles is selected and its helicity density matrix is
calculated,

$\displaystyle\rho_{\lambda_{j}\lambda_{j}^{\prime}}^{j}={}$
$\displaystyle\rho_{\kappa_{1}\kappa_{1}^{\prime}}^{1}\rho_{\kappa_{2}\kappa_{2}^{\prime}}^{2}\mathcal{M}_{\kappa_{1}\kappa_{2};\lambda_{1}\ldots\lambda_{n}}\mathcal{M}_{\kappa_{1}^{\prime}\kappa_{2}^{\prime};\lambda_{1}^{\prime}\ldots\lambda_{n}^{\prime}}^{*}$
(1) $\displaystyle\times\prod_{k\neq
j}\mathcal{D}_{\lambda_{k}\lambda_{k}^{\prime}}^{k}$

and the trace is normalized. Here, $\rho^{1,2}$ are the helicity density
matrices of the incoming particles with helicity $\kappa_{1,2}$, $\mathcal{M}$
is the matrix element with outgoing particle helicities $\lambda_{k}$, and
$\mathcal{D}^{k}$ are the decay matrices of the outgoing particles,
initialized to the identity. If the particle is from a decay, the helicity
density matrix is the same as above but without $\rho^{2}$ and $\kappa_{2}$.

  3. 3.

The selected particle is decayed using the weight,

$\mathcal{W}=\rho_{\lambda_{0}\lambda_{0}^{\prime}}\mathcal{M}_{\lambda_{0};\lambda_{1}\ldots\lambda_{n}}\mathcal{M}_{\lambda_{0}^{\prime};\lambda_{1}^{\prime}\ldots\lambda_{n}^{\prime}}^{*}\prod_{k=1,n}\mathcal{D}_{\lambda_{k}\lambda_{k}^{\prime}}^{k}$
(2)

where $\rho$ is the helicity density matrix of the decaying particle with
helicity $\lambda_{0}$ and the decay matrix element $\mathcal{M}$.

  4. 4.

Steps 1 through 2 are performed until a decay is reached with no unstable
particles.

  5. 5.

The decay matrix of the last decayed particle is calculated,

$\mathcal{D}_{\lambda_{0}\lambda_{0}^{\prime}}=\mathcal{M}_{\lambda_{0};\lambda_{1}\ldots\lambda_{n}}\mathcal{M}_{\lambda_{0}^{\prime};\lambda_{1}^{\prime}\ldots\lambda_{n}^{\prime}}^{*}\prod_{k=1,n}\mathcal{D}_{\lambda_{k}\lambda_{k}^{\prime}}^{k}$
(3)

and the trace is normalized.

  6. 6.

An undecayed particle from the decay above is randomly selected and steps 1
through 3 are repeated.

  7. 7.

Step 6 is repeated until all unstable particles are decayed.

##  3 Production

Spin correlated $\tau$-lepton decays can be generated in Pythia from a variety
of hard processes, summarized in Table 1. The spin of the $\tau$-lepton can
also be provided from an externally generated hard process, or explicitly set
for all $\tau$-leptons or $\tau$-leptons from a specified parent.

For electroweak processes, spin correlations are fully handled if the incoming
fermions producing the $\gamma$, $Z$, or $W$ are known. If the $W$ is not
produced from fermions or the fermions are unavailable, the $\tau$-lepton is
given a polarization of $\mathcal{P}=-1$, e.g. $H\to W\to\tau{\nu_{\tau}}$.
For $\gamma$ or $Z$ hard processes not produced from fermions or with
unavailable fermion information, the $\gamma$ or $Z$ is assumed to be
unpolarized, e.g. $H\to ZZ\to 4\tau$.

Figure 1: Distribution of the fractional energy of the pion from a
$\tau^{-}\rightarrow\nu_{\tau}\pi^{-}$ decay in the rest frame of the
producing boson. The $\tau$-leptons are produced from $Z$ (dashed black), $H$
(dotted red), $W$ (dash-dotted green), and $H^{\pm}$ (dash-dot-dotted blue)
bosons.

Because the Higgs is spin zero, the production information is not necessary
for calculating $\tau$-lepton spin correlations. However, the type of Higgs
must be known. Currently, spin correlations of $\tau$-leptons produced from
the $\mathcal{CP}$-even ($H$, $h^{0}$, $H^{0}$), $\mathcal{CP}$-odd ($A^{0}$),
or charged ($H^{\pm}$) Higgses are calculated.

The $\tau$-lepton can also be produced from a variety of $B$ and $D$ meson
decays. For these decays, the constituent quarks of the meson are
approximated, and the spin correlations are calculated using an
$f\bar{f}^{\prime}\to W\to f\bar{f}^{\prime}$ matrix element. For any
$\tau$-lepton from an unknown process, the $\tau$-lepton is assumed to be
unpolarized.

The slope of the energy distribution for pions from
$\tau^{-}\rightarrow\nu_{\tau}\pi^{-}$ decays in the rest frame of the parent
bosons is proportional to the average $\tau$-lepton polarization. In Figure 1
this distribution, from Pythia simulation, is given for $\tau$-leptons
produced from electroweak and Higgs processes. Note the the opposite
polarization of $\tau$-leptons produced from a $W$ and $H^{\pm}$, the
unpolarized $\tau$-leptons produced from neutral Higgses, and the slight
polarization of $\tau$-leptons produced from $Z$ bosons at a $pp$ collider.

Type | Processes
---|---
electroweak |  $f\bar{f}\to\gamma\to f\bar{f}$, $f\bar{f}\to Z\to f\bar{f}$,
$f\bar{f}\to\gamma^{*}/Z\to f\bar{f}$, $f\bar{f}^{\prime}\to W\to
f\bar{f}^{\prime}$,
$Z\to f\bar{f}$, $W\to f\bar{f}^{\prime}$
Higgs |  $H\to f\bar{f}$, $h^{0}\to f\bar{f}$, $H^{0}\to f\bar{f}$
$A^{0}\to f\bar{f}$, $H^{\pm}\to f\bar{f}^{\prime}$
other | $B/D\to f\bar{f}^{\prime}+X$
Table 1: Production mechanisms in Pythia for which full spin correlations of
$\tau$-lepton decays are automatically calculated.

##  4 Decays

The matrix element for the decay of the $\tau$-lepton can be written as,

$\mathcal{M}=\frac{g_{w}^{2}}{8m_{W}^{2}}L_{\mu}J^{\mu}$ (4)

where $g_{W}$ is the $\mathit{SU(2)}$ coupling, $m_{W}$ the $W$ mass,
$L_{\mu}$ the leptonic current of the $\tau$-lepton, and $J_{\mu}$ a leptonic
or hadronic current dependent upon the decay. The $\tau$-lepton current
$L_{\mu}$ is $\bar{u}_{\nu_{\tau}}\gamma_{\mu}(1-\gamma^{5})u_{\tau}$ where
$u_{\tau}$ is the $\tau$-lepton spinor, dependent upon momentum and helicity,
and $\bar{u}_{\nu_{\tau}}$ is the $\tau$-lepton neutrino spinor. A full list
of the available $J_{\mu}$ currents modeled in Pythia is given in Table 2.

For the two-body decays of the $\tau$-lepton, $J_{\mu}$ is $fq$, where $f$ is
a constant and $q$ is the momentum of the hadron. For the three-body leptonic
decays, $\tau^{-}\rightarrow\nu_{\tau}e^{-}{\bar{\nu}_{e}}$ and
$\tau^{-}\rightarrow\nu_{\tau}\mu^{-}{\bar{\nu}_{\mu}}$, $J_{\mu}$ is of the
same form as $L_{\mu}$. Two hadronic three-body decay models are available, a
decay via a vector resonance and a decay via a vector and scalar resonance.

Four-body $\tau$-lepton decays in Pythia are implemented in four different
models. The primary four-body decays,
$\tau^{-}\rightarrow\nu_{\tau}\pi^{0}\pi^{0}\pi^{-}$ and
$\tau^{-}\rightarrow\nu_{\tau}\pi^{-}\pi^{-}\pi^{+}$, are modeled using the
CLEO fit. These decays can also be performed using a generic hadronic four-
body model from Decker, et al. The four-body decays with kaons in the final
state are calculated using a model from Finkemeier and Mirkes.

The five-body decays of the $\tau$-lepton to pions are produced with the
Novosibirsk model, a phenomenological fit of four pion production from
electron-positron annihilation. The six-body decays of the $\tau$-lepton are
handled through a model proposed by Kühn and Wa̧s.

A comparison of the hadronic invariant mass distributions, generated with
Pythia, between the dominant three, four, five, and six-body decays of the
$\tau$-lepton is given in Figure 2. The
$\tau^{-}\rightarrow\nu_{\tau}\pi^{0}\pi^{0}\pi^{-}$ distribution was
generated using the default CLEO model. The expected upwards shift for higher
multiplicity decays can be seen, as well the distinct difference in shapes due
to the differing propagators of the models.

Figure 2: Distribution of the invariant mass of the hadronic current for the dominant three-body $\pi^{0}\pi^{-}$ (dashed black), four-body $\pi^{0}\pi^{0}\pi^{-}$ (dotted red), five-body $\pi^{0}\pi^{-}\pi^{-}\pi^{+}$ (dash-dotted green), and six-body $\pi^{-}\pi^{-}\pi^{-}\pi^{+}\pi^{+}$ (dash-dot-dotted blue) decays of the $\tau$-lepton. Multiplicity | Model | $\mathcal{M}$ | Decay Products
---|---|---|---
$2$-body | single hadron |  | 1521 |  $\pi^{-}$, $K^{-}$
$3$-body | leptonic |  | 1531 |  $e^{-}{\bar{\nu}_{e}}$, $\mu^{-}{\bar{\nu}_{\mu}}$
Kühn and Santamaria | [9] | 1532 |  $\pi^{0}\pi^{-}$, $K^{0}K^{-}$, $\eta K^{-}$
Finkemeier and Mirkes | [10] | 1533 |  $\pi^{-}\bar{K}^{0}$, $\pi^{0}K^{-}$
$4$-body | CLEO | [11] | 1541 |  $\pi^{0}\pi^{0}\pi^{-}$, $\pi^{-}\pi^{-}\pi^{+}$
Finkemeier and Mirkes | [12] | 1542 |  $K^{-}\pi^{-}K^{+}$, $K^{0}\pi^{-}\bar{K}^{0}$, $K_{S}^{0}\pi^{-}K_{S}^{0}$, $K_{L}^{0}\pi^{-}K_{L}^{0}$, $K_{S}^{0}\pi^{-}K_{L}^{0}$,
$K^{-}\pi^{0}K^{0}$, $\pi^{0}\pi^{0}K^{-}$, $K^{-}\pi^{-}\pi^{+}$,
$\pi^{-}\bar{K}^{0}\pi^{0}$
Decker, et al. | [13] | 1543 |  $\pi^{0}\pi^{0}\pi^{+}$, $\pi^{-}\pi^{-}\pi^{+}$, $K^{-}\pi^{-}K^{+}$, $K^{0}\pi^{-}\bar{K}^{0}$, $K^{-}\pi^{0}K^{0}$,
$\pi^{0}\pi^{0}K^{-}$, $K^{-}\pi^{-}\pi^{+}$, $\pi^{-}\bar{K}^{0}\pi^{0}$,
$\pi^{-}\pi^{0}\eta$
Jadach, et al. | [14] | 1544 | $\gamma\pi^{0}\pi^{-}$
$5$-body | Novosibirsk | [15] | 1551 |  $\pi^{0}\pi^{-}\pi^{-}\pi^{+}$, $\pi^{0}\pi^{0}\pi^{0}\pi^{-}$
$6$-body | Kühn and Wa̧s | [16] | 1561 |  $\pi^{0}\pi^{0}\pi^{-}\pi^{-}\pi^{+}$, $\pi^{0}\pi^{0}\pi^{0}\pi^{0}\pi^{-}$, $\pi^{-}\pi^{-}\pi^{-}\pi^{+}\pi^{+}$
Table 2: Summary of available $\tau$-lepton decay models in Pythia $8$ sorted
by multiplicity. For each model the internal Pythia matrix element mode
identifier ($\mathcal{M}$) is given, as well as the decays available through
the model and the reference of the model. The implicit ${\nu_{\tau}}$ is
omitted for brevity.

##  5 Interface

Documentation for $\tau$-lepton decays in Pythia can be found under the Tau
Decays subsection of Particle Decays in the Pythia HTML manual. The
$\tau$-lepton decay mechanism is configured by setting the mode
sophisticatedTau of ParticleDecays. This mode default is 1, where the decays
are performed using the sophisticated $\tau$-lepton decay machinery. When
sophisticated decays are not necessary, or a comparison with the isotropic
Pythia model is needed, the mode can be set to 0.

If the mode is set to 1, and a LHEF file is read, the spin of the
$\tau$-lepton will be set to the SPINUP information. However, if no SPINUP
information is provided, and the $\tau$-lepton is from a known production
process, Pythia will automatically calculate the polarization. If the
$\tau$-lepton is from an unknown process the sophisticated decay will be
performed assuming the $\tau$-lepton is unpolarized.

It is possible to force the polarization of the $\tau$-leptons and override
either the Pythia calculated polarization or the polarization from a LHEF
file. If the mode is set as 2, all $\tau$-leptons produced from a parent with
an identification code specified by tauMother will be given a polarization set
by the parameter tauPolarization. If mode 3 is selected, all $\tau$-leptons
will be produced with the polarization set by tauPolarization.

The models used to decay the $\tau$-leptons for each channel can be changed by
switching the matrix element mode of the channel. The syntax takes the form
15:channel:meMode = mode where 15 is the $\tau$-lepton particle identification
code, channel specifies the decay channel number as listed under Particle
Data, and mode is the new matrix element mode. For example, the default CLEO
model used for the $\tau^{-}\rightarrow\nu_{\tau}\pi^{0}\pi^{0}\pi^{-}$ decay
can switched to the Decker, et al. model using 15:9:meMode = 1543. However,
caution is advised in switching models, as it is possible to set a channel to
use an incompatible matrix element, causing unpredicted behavior.

##  6 Implementation

The implementation of the $\tau$-lepton decay machinery in Pythia is intended
to be complete and fast, yet easily extensible for new decay models or
production processes. A helicity framework is available in the Pythia source
files HelicityBasics where methods for calculating the helicity dependent
wavefunctions for particles, both fermions and bosons, is provided, as well as
the corresponding $\gamma$-matrices.

Using this framework, the helicity matrix element for the decay of a
$\tau$-lepton into a pion,

$\mathcal{M}=\bar{u}_{\nu_{\tau}}\gamma_{\mu}(1-\gamma^{5})u_{\tau}f_{\pi}q_{\pi}^{\mu}$
(5)

can be written in pseudo-code as,

$\displaystyle\mathcal{M}=\sum_{\mu}$
$\displaystyle{\nu_{\tau}}\mathtt{.waveBar(}\lambda_{\nu_{\tau}}\mathtt{)*GammaMatrix(}\mu\mathtt{)}$
(6)
$\displaystyle\mathtt{*(1-GammaMatrix(5))*}\tau\mathtt{.wave(}\lambda_{\tau}\mathtt{)*}f_{\pi}$
$\displaystyle\mathtt{*GammaMatrix(4)(}\mu\mathtt{,}\mu\mathtt{)*Wave4(}q_{\pi}\mathtt{)(}\mu\mathtt{)}$

where $\gamma^{4}$ provides the $+---$ metric.

Both the production matrix elements and $\tau$-lepton decay matrix elements
are written using this framework and are provided in the
HelicityMatrixElements files of the Pythia source. For the $\tau$-lepton decay
matrix elements, it is only necessary to implement the $J_{\mu}$ current. The
spin correlation and decays of the $\tau$-leptons are implemented in the
TauDecays files of the Pythia source.

##  7 Conclusion

Fully modeled $\tau$-lepton decays with spin correlations are now available in
Pythia, using fully extensible code. Further development of $\tau$-lepton
production mechanisms and decay models is underway, and feedback on requested
features is welcome.

##  8 Acknowledgments

The author would like to acknowledge Torbjörn Sjöstrand and the Pythia team
for all of their invaluable help. The funding for this project was provided by
Lund University and MCNet through a Marie Curie grant, contract MRTN-
CT-2006-035606.

## References

  * [1] ATLAS Collab., Tech. Rep. ATLAS-CONF-2012-160.
  * [2] CMS Collab., Tech. Rep. CMS-PAS-HIG-12-050.
  * [3] T. Sjöstrand, et al., Comput.Phys.Commun. 178 (2008) 852–867.  arXiv:0710.3820.
  * [4] P. Golonka, et al., Comput.Phys.Commun. 174 (2006) 818–835.  arXiv:hep-ph/0312240.
  * [5] D. Grellscheid, P. Richardson. arXiv:0710.1951.
  * [6] J. C. Collins, Nucl.Phys. B304 (1988) 794.
  * [7] I. Knowles, Comput.Phys.Commun. 58 (1990) 271–284.
  * [8] P. Richardson, JHEP 0111 (2001) 029.  arXiv:hep-ph/0110108.
  * [9] J. H. Kühn, A. Santamaria, Z.Phys. C48 (1990) 445–452.
  * [10] M. Finkemeier, E. Mirkes, Z.Phys. C72 (1996) 619–626.  arXiv:hep-ph/9601275.
  * [11] CLEO Collab., Phys.Rev. D61 (2000) 012002.  arXiv:hep-ex/9902022.
  * [12] M. Finkemeier, E. Mirkes, Z.Phys. C69 (1996) 243–252.  arXiv:hep-ph/9503474.
  * [13] R. Decker, et al., Z.Phys. C58 (1993) 445–452.
  * [14] S. Jadach, et al., Comput.Phys.Commun. 76 (1993) 361–380.
  * [15] A. Bondar, et al, Comput.Phys.Commun. 146 (2002) 139–153.  arXiv:hep-ph/0201149.
  * [16] J. H. Kühn, Z. Wa̧s, Acta Phys.Polon. B39 (2008) 147–158.  arXiv:hep-ph/0602162.