# Adaptive Threshold for Online Object Recognition and Re-identification Tasks

Bharat Bohara
deHumlaTech AI Research
<EMAIL_ADDRESS>
###### Abstract

Choosing a decision threshold is one of the challenging job in any
classification tasks. How much the model is accurate, if the deciding boundary
is not picked up carefully, its entire performance would go in vain. On the
other hand, for imbalance classification where one of the classes is dominant
over another, relying on the conventional method of choosing threshold would
result in poor performance. Even if the threshold or decision boundary is
properly chosen based on machine learning strategies like SVM and decision
tree, it will fail at some point for dynamically varying databases and in case
of identity-features that are more or less similar, like in face recognition
and person re-identification models. Hence, with the need for adaptability of
the decision threshold selection for imbalanced classification and incremental
database size, an online optimization-based statistical feature learning
adaptive technique is developed and tested on the LFW dataset and self-
prepared athletes dataset. This method of adopting adaptive threshold resulted
in $12-45\%$ improvement in the model accuracy compared to the fixed threshold
$\\{0.3,0.5,0.7\\}$ that are usually taken via the hit-and-trial method in any
classification and identification tasks. Source code for the complete
algorithm is available at: https://github.com/Varat7v2/adaptive-threshold.git

Keywords: Adaptive threshold, Imbalanced classification, Online face
recognition, Online person re-identification

##  1 Introduction

Most of the research is mainly focused on revising the performance of
classification architectures, optimization algorithms, and loss functions, but
hardly done any work on the improvement over adaptability of the decision
threshold that separates the boundary-line between the classes. Although there
are myriad state-of-the-art models to extract discriminative features, it is
still a common trend to choose a classification threshold by the hit-and-trial
method. As per the author’s best practice, for a handful of feature vectors in
the database, it works pretty well, but it becomes ineffective with the
increasing database’s size. If the chosen threshold is futile, how much
accurate the model is, its state-of-the-art performance would go in vain. On
the contrary, if the threshold is chosen wisely and is updated iteratively as
soon as a new feature vector is registered in the database, even less accurate
model will do a better job. In this project, an optimization-based statistical
feature learning algorithm has been developed to boost-up the performance of
any recognition and re-identification models through decision boundary.

A decision threshold is a numerical value that dichotomizes different classes.
Different thresholds yield a different number of true/false positives and
true/false negatives, and consequently different precision, recall, and
f1-score for a given dataset. In this project, a decision threshold is
selected to maximize the f1-score (that is, the harmonic average of precision
and recall). For instance in cancer diagnostic problem, it acts as a specified
cut-off for an observation to be classified as either 0 (no cancer) or 1 (has
cancer). Choosing an optimal threshold value is a challenging task as it is
case-specific, i.e., different for different objectives and datasets. In the
context of verification and identification tasks, as the identity-database
like in face recognition and person re-identification gets updated with a
newer identity raises the need for tuning the threshold with database update.
This threshold works not only as the deciding factor for verification and
identification but also as the gatekeeper to update the identities in
database.

##  2 Related work

There is not much work done yet in selecting the classifier threshold
adaptively; however, Chou et. al (2018) [1] has developed an online face
registration mechanism with a distinctive threshold per face. With this
mechanism, they achieved 22% accuracy improvement on the LFW dataset as
compared to the common method of choosing the threshold - fixed value
throughout the inference time. In brief, they have proposed an adaptive
thresholding technique that assigns a specific threshold per registered face
in the database – that gets adopted accordingly with the new entries in the
database. However, this method is limited only to the evaluation task but not
favorable to the real-time inference.

Receiver operating characteristics (ROC) curve plots the true-positive rate
against the false-positive rate at particular thresholds. A diagonal line
indicates that the model predicts all the cases as the majority class
delineating its poor performance over the test-dataset; above the curve
indicates a threshold with higher model accuracy and below it with poor
accuracy. ROC curve helps to understand the trade-off in the true-positive
rate and false-positive rate for different threshold values. The area under
the ROC curve is known as the area under the curve (AUC) that depicts the
model’s overall performance.

Zou et al. (2016) [2], have analyzed drawbacks of using ROC-curve as the sole
measure of selecting the threshold for an imbalanced classification. A novel
framework - sampling-based threshold auto-tuning method, for finding the best
classification threshold is proposed yielding $20.63\%$ improvement over the
conventional method of choosing a default threshold of 0.5. Though ROC curve
and AUC values reflect ranking power of positive prediction probability, Zou
et al. claim that classifier performance including precision, recall, and
f1-score might not be perfect even-though AUC value exceed 0.9. They have
employed f1-score and AUC for Liao’s protein remote homology detection such
that the classification threshold is tuned for the best f1-score. Similarly,
Lipton et al. (2014) [3] has proposed an optimal classification threshold
selection through maximization of f1-score value for binary and multi-label
classification problem - claiming improvement in predicting 26,853 labels of
Medline documents compared to the traditional method.

Al Hartmann [4] has filed a patent on an adaptive threshold for detecting spam
email messages based on ratios between clean and spam emails received at
previous time-periods and misclassification ratio cost. Bauer et al. (2015)
[5] has applied a Bayesian model of neurofeedback and reinforcement learning
for evaluating the impact of adaptive classification threshold on optimizing
restorative brain-computer interfaces (BCI). It is claimed that a threshold
adaptation is superior to any-of-the fixed threshold throughout the
experimental results.

##  3 Methodology

For statistical feature learning of a probability density function, different
metrics like cosine distance, cosine similarity, and euclidean distance as
mentioned in equations [1,2,3] are used. Nonetheless similarity metric seems
to fit best to our objective problem. Cosine similarity is a metric for
measuring the distance between two vectors of an inner product space. In other
words, it is the cosine of the angle between the two vectors projected in a
multi-dimensional space, also commonly known as the inner dot product of those
vectors - both normalized to have a unity length. The vectors’ similarity is
between 0 to 1; 0 means entirely dissimilar, and 1 means completely similar.
So higher the value, higher will be the similarity between the vectors.
Similarly, euclidean distance is a metric to measure distance or length
between the two points in a euclidean space. The significance of all these
metrics, especially in the machine learning arena, is to compute how similar
or dissimilar the feature vectors are. In this project, since there are
multiple feature vectors for the same identity (i.e., face or person),
vectorial similarities and distances are organized into auto and cross-
category - inspired from the auto/cross-correlation terminologies.

Figure 1: Auto similarities pairing Figure 2: Cross similarities pairing
Figure 3: Flow-diagram of online optimization based adaptive threshold search
method

Similarities and distances between the feature vectors of the same (here
denoted as auto) and different (here denoted as a cross) identities (here
identities denote faces, persons, or any other subject of interest) are
computed simultaneously. Figures [1,2] show all the possible pairing of the
identities in order to compute auto and cross distance/similarity between the
query, and database identities. While finding an adaptive threshold, query
embeddings ($Q_{\imath}$) are cloned from gallery embeddings ($G_{\jmath}$) to
find the similarity among same (auto-similarity) and different identities
(cross-similarity). Hence during the query mode, $G_{\imath}$ and $Q_{\jmath}$
will be different images or embeddings, whereas during optimal-threshold
search mode, query embeddings ($Q_{\imath}$) will be an exact copy of database
embeddings ($G_{\jmath}$). Different statistical features like minima, maxima,
mean, standard deviation, variance, and much more information can be drawn
from auto and cross distance/similarity distributions. From such statistical
information, an adaptive threshold will be adjusted once a new identity is
added or deleted from the database. Since calculating statistical measures
along with euclidean distance or cosine similarity between the feature vectors
from a large pool of distributions is computationally expensive, an adaptive
threshold adjustment can be done periodically once the number of newer
identity registration exceeds a certain specified number.

$\begin{split}similarity(\cos\theta)&=\frac{\textbf{X}\cdot\textbf{Y}}{\left\|\textbf{X}\right\|\left\|\textbf{Y}\right\|}\\\
&=\frac{\sum_{i=1}^{n}X_{i}Y_{i}}{\sqrt{\sum_{i=1}^{n}X_{i}^{2}}\sqrt{\sum_{i=1}^{n}Y_{i}^{2}}}\end{split}$
(1)

$D_{cosine}=1-\cos\theta$ (2)

$D_{euclidean}(X,Y)=\sqrt{\sum_{i=1}^{n}\left(X_{i}-Y_{i}\right)^{2}}$ (3)

Referring to figure 3, once auto and cross similarity distribution is
obtained, their probability of occurrences is depicted in a histogram. With a
mean and a standard deviation of auto and cross probability distributions, a
Gaussian function is estimated for each of them, as shown in equation 4. The
main reason behind estimating auto and cross Gaussian distribution function is
to choose a value between their means such that none of the distribution is
dominated or biased while choosing the threshold. We can either choose an
average of auto and cross means directly without estimating their normal
distribution, but doing so might unnecessarily shift our threshold towards a
curve with a higher peak. In order to find the point of intersection of the
estimated auto and cross Gaussian distribution functions, we need to equate
and solve them, as shown in quadratic equation 5.

$\displaystyle
f_{1}(x)=\frac{1}{\sigma_{1}\sqrt{2\pi}}e^{-\frac{1}{2}\left(\frac{x_{1}-\mu_{1}}{\sigma_{1}}\right)^{2}}$
(4) $\displaystyle
f_{2}(x)=\frac{1}{\sigma_{2}\sqrt{2\pi}}e^{-\frac{1}{2}\left(\frac{x_{2}-\mu_{2}}{\sigma_{2}}\right)^{2}}$

where $f_{1}(x)$ is auto Gaussian function, and $f_{2}(x)$ is cross Gaussian
function, and $\mu_{1},\sigma_{1},x_{1}$ and $\mu_{2},\sigma_{2},x_{2}$ are
mean, standard-deviation, and x-coordinate of the intersection point of the
auto and cross Gaussian distribution functions respectively.

Equating and solving equations 4 can be arranged in a condensed quadratic
form: $Ax^{2}+Bx+C=0$, where its coefficients are obtained as shown in
equation 5.

$\begin{split}A=\sigma{1}^{2}-\sigma{2}^{2}=\nu_{1}-\nu_{2}\\\
B=2(\mu_{1}\sigma_{2}^{2}-\mu_{2}\sigma_{1}^{2})=2(\mu_{1}\nu_{2}-\mu_{2}\nu_{1})\\\
C=\mu_{2}^{2}\sigma_{1}^{2}-\mu_{1}^{2}\sigma_{2}^{2}-2\sigma_{1}^{2}\sigma_{2}^{2}\log\left(\frac{\sigma_{1}}{\sigma_{2}}\right)\\\
=\nu_{1}\mu_{2}^{2}-\nu_{2}\mu_{1}^{2}-\nu_{1}\nu_{2}\log\left(\frac{\nu_{1}}{\nu_{2}}\right)\end{split}$
(5)

where, $\sigma_{1},\nu_{1}$, and $\sigma_{2},\nu_{2}$ are standard deviation
and variance for auto and cross Gaussian distribution functions. Since,
$\lambda\in\\{0,1\\}$, we can ignore roots lying out of this bound; otherwise,
select either of the roots and evaluate the model, and consider the root with
higher model accuracy. If the point of intersection lies between $\mu_{cross}$
and $\mu_{auto}$, we can proceed ahead to compute model accuracy, otherwise
take an average of them as shown in equation 6.

$threshold=\begin{cases}\lambda,&\text{if}\
\mu_{cross}\leq\lambda\leq\mu_{auto}\\\
\frac{\mu_{cross}+\mu_{auto}}{2},&\text{otherwise}\end{cases}$ (6)

We used different performance metrics, such as precision, recall, f1-scores,
and accuracy for model evaluation. Their computation requires the count of
true/false positives and negatives at given threshold $\lambda$, that are
calculated as shown in the equation 7.

$\displaystyle
TP(\lambda)=\sum_{\imath=1}^{m}\sum_{\jmath=1}^{n}\mathds{1}\\{(i,j)\in\mathbb{P}_{same},\mathbb{S}_{max}(x_{i},y_{j})\geq\lambda\\}$
(7) $\displaystyle
FP(\lambda)=\sum_{\imath=1}^{m}\sum_{\jmath=1}^{n}\mathds{1}\\{(i,j)\in\mathbb{P}_{diff},\mathbb{S}_{max}(x_{i},y_{j})\geq\lambda\\}$
$\displaystyle
FN(\lambda)=\sum_{\imath=1}^{m}\sum_{\jmath=1}^{n}\mathds{1}\\{(i,j)\in\mathbb{P}_{same},\mathbb{S}_{max}(x_{i},y_{j})\leq\lambda\\}$
$\displaystyle
TN(\lambda)=\sum_{\imath=1}^{m}\sum_{\jmath=1}^{n}\mathds{1}\\{(i,j)\in\mathbb{P}_{diff},\mathbb{S}_{max}(x_{i},y_{j})\leq\lambda\\}$

where
$TP(\lambda)$ is the total no. of times when the model correctly predicts the
positive class with a maximum similarity between feature vectors greater than
equal to the threshold, $TN(\lambda)$ is the total no. of times when the model
correctly predicts the negative class with a maximum similarity between
feature vectors lesser than equal to the threshold, $FP(\lambda)$ is the total
no. of times when the model incorrectly predicts the positive class with a
maximum similarity between feature vectors greater than equal to the
threshold, $FN(\lambda)$ is the total no. of times when the model incorrectly
predicts the negative class with a maximum similarity less than equal to the
threshold, $\mathbb{S}_{max}(x_{i},y_{j})$ is the maximum cosine similarity
between identities $x_{i}$ and $y_{j}$, $\mathbb{P}_{same}$ is auto-pairing,
and $\mathbb{P}_{diff}$ is cross-pairing between the $i^{th}$ and $j^{th}$
identities.

Using the count of true/false positives and negatives at specified threshold
$\lambda$ from equation 7, model precision, recall, f1-score, and model
overall accuracy can be computed as shown in equation 8.

$\displaystyle Precision(\lambda)=\frac{TP(\lambda)}{TP(\lambda)+FP(\lambda)}$
(8) $\displaystyle
Recall(\lambda)=\frac{TP(\lambda)}{TP(\lambda)+FN(\lambda)}$ $\displaystyle
F1score(\lambda)=2*\frac{Precision(\lambda)*Recall(\lambda)}{Precision(\lambda)+Recall(\lambda)}$
$\displaystyle Accuracy(\lambda)=\frac{TP+TN}{TP+TN+FP+FN}$

Similarly, true positive rate ($TPR$) and false positive rate ($FPR$) for
plotting ROC curve is computed with the same count of true/false positives and
negatives - resulting from the model evaluation at given threshold $\lambda$
as shown in equation 9.

$\displaystyle TPR(\lambda)=\frac{\sum TP}{\sum
CP}=\frac{TP(\lambda)}{TP(\lambda)+FP(\lambda)+\epsilon}$ (9) $\displaystyle
FPR(\lambda)=\frac{\sum FP}{\sum
CN}=\frac{FP(\lambda)}{FP(\lambda)+TN(\lambda)+\epsilon}$

where, $CP$ is a cumulative sum of all positives count, $CN$ is a cumulative
sum of all negatives count, $\epsilon$ is added to prevent from possible
division by zero error.

If model accuracy at specified threshold $\lambda$ exceeds the targeted value,
we can select it as an optimal threshold and proceed ahead with model
inference; otherwise, we need to search for the optimal value with an
objective to maximize f1-score as shown in equation 10.

$\lambda_{opt}=\begin{cases}\lambda,&\text{if}\ f1score(\lambda)\geq\tau\\\
\lambda,&\text{if}\ f1score(\lambda)\geq f1score(\lambda_{old})\\\
\lambda_{old},&\text{otherwise}\end{cases}$ (10)

For optimizing threshold at which the f1-score is below the targeted value,
bounded minimization method is used to maximize the trade-off between $TPR$
and $FPR$ as shown in equation 11. In other words, using the optimization
technique, we are trying to pick the best threshold value from the ROC curve
for which we intend to maximize the number of true positives while suppressing
the count of false positives.

$\displaystyle\max_{\lambda}$ $\displaystyle\left\lvert
TPR(\lambda)-FPR(\lambda)\right\rvert$ (11) s.t.
$\displaystyle\lambda\in\\{0,1\\}$

However, in this study, we have directly taken maximization of f1-score as our
objective function constrained to $\lambda\in\\{0,1\\}$ and bounded by
$\mu_{cross}$ and $\mu_{auto}$. With an assumption that choosing a threshold
value between the means of auto and cross-similarities density functions would
maximize the number of true positives and minimize true negatives count, we
tried both methods of bounded ($\mu_{cross}\leq\lambda\leq\mu_{auto}$) and
unbounded optimization. As a result, it was found that unbound method i.e.,
$\lambda\in\\{0,1\\}$ resulted in higher model accuracy. Hence, we tweaked our
objective function 12 by making search space flexible to $[0,1]$.

$\displaystyle\max_{\lambda}$
$\displaystyle\Bigg{\\{}{f1score=2*\frac{P(\lambda)*R(\lambda)}{P(\lambda)+R(\lambda)}\Bigg{\\}}}$
(12) s.t. $\displaystyle\lambda\in\\{0,1\\}$

At optimization termination, if precision or f1-score at the converged
threshold ($\lambda$) is still less than that of the point of intersection of
the Gaussian distribution functions, we choose the $\lambda_{old}$ as our
working threshold for model inference as depicted in equation 10. This
procedure as shown in figure 3 repeats if there is any change in the database,
i.e., addition or deletion of any feature vector; otherwise, the chosen
threshold will be used for inference unless any change in the database is
observed. On the contrary, since searching optimal threshold is
computationally expensive, we can run this algorithm once the model
accuracy/f1-score falls below a certain value. However, for areas like
medicine where a perfect threshold is a must, continuous running of this
technique as illustrated in figure 3 would benefit significantly.

### Model Inference

Once the optimal threshold is chosen based on an adaptive algorithm as
portrayed in figure 3, we proceed ahead with model inference. In this section,
as unfolded in figure 4, the feature vector of an image either from webcam or
directly from the image itself is generated using pre-trained recognition or
re-identification model. Here, for face recognition, facenet [6] and dlib [7]
models are taken into account, and joint discriminative and generative
learning model for person re-identification. For a given query image, we first
detect face or person alignment using CNN models. Once a face is detected, we
feed it into the pre-trained model to extract its 128D or 512D facial feature
vector. Consequently, we perform dot product of this feature vector with an
entire pool of vectors stored in the database to calculate the similarities
between each of them. Upon comparison of the similarity metric for a query
vector, if it is greater than the chosen decision threshold, it is assumed
that the identity has been identified or recognized, and model inference
continues. Conversely, if the similarity of the query identity does not match
with any of the gallery vectors, it is recorded as a new identity in the
database.

Figure 4: Flow-diagram of identity recognition/re-identification

##  4 Experiments

This method has been tested on labeled faces in the wild (LFW) [8], that is a
de facto face verification dataset. Since most of the identities in the LFW
dataset comprise at most two images per identity, full effectiveness and
robustness of an adaptive threshold algorithm is not observed. A separate
dataset of the top 100 highly-paid athletes referred to Forbes’s 2018 list [9]
is prepared with more than 20 images per identity with an online automated
dataset generator [10]. Since the same set-of-identities is used for galley
and query embeddings, only identities with two or more than two images are
taken into account. On the other hand, since the whole model accuracy depends
on the number of images covering an entire $180^{\circ}$ facial orientation of
a person, there needs to be a flexibility in adjusting it. If headpose is
incorporated, we can choose face oriented only at a particular direction so
that 3-4 images would cover entire possible regions of detection. For
instance, the most accurate CNN-based face detector can detect a face rotated
up to $90^{\circ}$ in either direction w.r.t a perpendicular line protruded
out of the face. The other important point to be noted is that since the same
dataset, i.e., LFW and Athletes, is used for model evaluation, to simulate a
real-time dynamic nature of the database or gallery size, the number of
identities is increased one-by-one starting with two at the beginning. That is
to say that during the start of the algorithm, two identities are accounted to
compute a threshold; then substantially all the available identities are added
to the galley one at a time; simultaneously computing the appropriate
threshold with the momentary database size. Hence, using the proposed
algorithm, the threshold learns to adapt to a changing gallery/database
identities’ size.

Figure 5: Auto-similarity distribution among faces of same identities

From figure 5, cosine similarities between the same identities (auto) is
distributed in such a way that maximal distribution lies within the range of
$[0.2-0.8]$.

Figure 6: Cross-similarity distribution among faces of different identities

From figure 6, it is observed that mostly similarity among different
identities is distributed between the range $[-0.4,0.8]$. In reference to the
distribution pattern of the auto/cross similarities observed in figures [5,
6], and histogram 7, if a single threshold ought to be chosen would be in
between their means i.e., $[0.2-0.6]$. If we chose the lower bound of a choice
list, the threshold would perform better at identifying similar identities,
whereas bad at differentiating dissimilar ones, and vice versa for the upper
bound. Therefore, in this project, we have used an online optimization-based
threshold technique that can adapt to the changes in the database size and
their probability density functions.

Figure 7: Probability of occurrences of similarities among same (auto) and
different identities (cross) Figure 8: Estimated actual and Gaussian
distribution functions for standardized auto and cross similarities functions

Figure 7 shows probability density of similarity function for same and
different identities for an Athletes dataset. From the observed histogram, a
normal distribution function is estimated using a mean and standard deviation
of the actual auto and cross similarity distributions. Figure 8 shows that the
point of intersection of the actual auto and cross similarity distribution
functions separates the curves in a most unbiased way. Higher model accuracy
is assumed to be achieved if it is taken as a threshold. There is no such
defined function for those actual probability distribution functions. However,
their estimated Gaussian distribution function mimics the distribution to a
larger extent, though there is a slight shift in the point-of-intersection.
The intersection point of the estimated normal distribution functions is
computed by solving a quadratic equation with coefficients shown in equation
5. This point can be taken as an initial value for an adaptive threshold and
perform its conditional inspection. If model accuracy (here f1-score) is
greater than the targeted value (here $\geq 80\%$), we take it as an optimal
threshold, whereas if it is not, then we proceed ahead with an optimum search
via bounded optimization method as shown in the objective function 12. Upon
the termination of an optimal search, if f1-score at that point ($\lambda$) is
greater than that of the point of intersection of the estimated Gaussian
functions, it is taken as an optimal threshold; otherwise, the intersection
point ($\lambda_{old}$) is taken as an optimal threshold.

Figure 9: Model performance at different threshold (adaptive) Figure 10:
Comparative study of fixed threshold @{0.3, 0.5, 0.7} with adaptive threshold

The model performance at the chosen optimum threshold is compared with its
performance at various fixed threshold values that are usually taken via hit-
and-trial method. In this study, the pre-specified fixed threshold points are
chosen from auto and cross similarity distribution functions, i.e., point at
maximum density or occurrences. From the auto and cross probability density
function referred to figure 7, $\mu_{auto}$ and $\mu_{cross}$ for the Athletes
dataset is observed to be 0.3 and 0.6, respectively. From figure 10, it is
observed that model accuracy at adaptive threshold overshadows the performance
at fixed threshold ($=0.5$) (which is the as-usual chosen threshold in the
context of conventional classification problem) and $0.7$ \- at which model
precision is highest among all. However, looking at f1-score, model
performance at adaptive threshold is more likely comparable to that of the
fixed threshold ($=0.3$). It might be because of its higher recall value; in
contrast, looking at the comparative precision curve, it is worst among all
the chosen threshold, including adaptive one. Hence, therefore, inspecting
precision, recall, and f1-score separately, adaptive threshold outperforms all
the chosen fixed thresholds believed that they are the best and conventionally
taken as-usual thresholds. From the figure 10, one of the noticeable things of
an adaptive threshold is that once the number of identities in the gallery
increases, resulting in a drastic drop in the model accuracy, there is an
abrupt adaptation in the threshold to keep the model constraints within check.
In this particular test case, model accuracy at the fixed threshold ($=0.3$)
is quite similar to that of an adaptive threshold, however, for a large number
of identities in the database, its adaptive nature will start outperforming
rest of all the fixed values. Hence, with a larger dataset or in case of
continuous running of the model, it is evident that gallery size will increase
indefinitely; consequently, the adaptive threshold will adjust for better
performance while the best-fixed threshold deteriorates relentlessly over
time-span.

Figure 11: Precision curve at fixed threshold @{0.3, 0.5, 0.7} and adaptive
threshold

From figure 11, it is observed that the model precision is highest at fixed
threshold ($=0.7$) followed by fixed threshold ($=0.5$), but from figure 12,
recall at those values are worst. Hence, looking at the harmonic mean of
precision and recall at those fixed points can be discarded. Among all of
those chosen points, model performance is good at the adaptive threshold and
fixed threshold ($=0.3$). Since this experiment has been done only on a small
dataset, its performance at the lowest fixed threshold seems comparable to
that of adaptive. Nevertheless, there is a higher probability of getting worse
for a larger dataset because of its worsening precision with drastically
increasing database size.

Figure 12: Recall curve at fixed threshold @{0.3, 0.5, 0.7} and adaptive
threshold Figure 13: F1-score curve at fixed threshold @{0.3, 0.5, 0.7} and
adaptive threshold Figure 14: ROC curve at fixed threshold @{0.3, 0.5, 0.7}
and adaptive threshold

From ROC curve 9, it is observed that the ratio of true positives to false
positives is higher with an adaptive threshold as compared to any of the fixed
thresholds $\\{0.3.0.5,0.7\\}$. Looking at the curve, if we ought to choose
any one of the fixed thresholds, then it would be $0.5$ as its area under the
curve is highest among all the fixed ones. Thus, the ROC curve’s comparative
plot validates that the adaptive method is a good choice for choosing a
threshold for the chosen dataset.

Table 1: Comparative model performance at different thresholds | Threshold
---
Type
| AUC
---
(unit sq.)
| Model
---
Accuracy
(%)
| F1score $\geq$ 0.8
---
(%)
Adaptive | 0.72 | 88.34 | 20.32
<EMAIL_ADDRESS>| 0.62 | 66.89 | 19.54
<EMAIL_ADDRESS>| 0.68 | 78.22 | 5.31
<EMAIL_ADDRESS>| 0.64 | 60.53 | 0.0

Table 1 shows that the model performs well with an adaptive threshold compared
to any of the fixed thresholds. Though f1-score is $\geq 80\%$ for most of the
time for a fixed threshold at ($=0.3$), its precision is horrible. The higher
f1-score might be because of its higher recall value. On the contrary, a fixed
threshold of ($=0.7$) does have higher precision than any of the other bands,
as vivid in figure 11, but due to its bad recall, the final f1-score is lowest
among all. If we look solely at the model accuracy, as shown in equation 8, a
model with an adaptive threshold outperforms all the other fixed threshold
values. Similarly, if we look at the area under the ROC-curve, the adaptive
threshold occupies maximum area that further validates model’s better
performance with the adaptive threshold.

##  5 Conclusion

Most of the research in machine learning focuses on improving the existing
model architectures and modifying the optimization algorithms and loss
functions. However, there is hardly any research done in making the decision
threshold adaptive. Once the model outputs confidence scores, they need to be
sieved with a decision threshold to categorize them to their respective
classes. If the threshold is not chosen wisely, the entire model performance
will go in vain. However, in most cases, this significant deciding value is
taken via hit-and-trial method; or randomly some fixed value just by analogy
(mostly 0.5 by default). Mostly for examples like classifying different
objects - where each object’s features are vividly distinct, it might work,
but not for all cases.

Nonetheless, for identification tasks like in face-recognition and person re-
identification - where features between different identities are more or less
identical and inseparable, the conventional method of choosing a fixed
threshold might not work. To counteract the idea of selecting an optimum
threshold based on ROC-curve, it might fail for temporally increasing
identity-database size. For a deep learning model that encounter
classification/identification of the objects with a higher degree of features-
similarity, a threshold that could adapt to the varying database size would
play a crucial role. Therefore, an online optimization-based statistical
feature learning technique is proposed to formulate an adaptive threshold in
this project. Whenever there is any update on the identity-database size, a
new decision threshold will be calculated based on the estimated Gaussian
distribution functions of the auto/cross-similarity distributions and bound-
constrained optimization search method. The proposed algorithm was tested on
Labeled Faces in the Wild (LFW), a de-facto face verification dataset, and a
self-prepared dataset of top 100 highly-paid athletes published by Forbes
magazine (2018). The reason behind preparing own dataset is that the algorithm
requires a large number of samples per identity for probing its robustness
over widely distributed features. The method achieved $32.07\%$, $12.94\%$,
and $45.94\%$ higher accuracy compared to the fixed thresholds at $0.3$,
$0.5$, and $0.7$, respectively. Looking at model accuracy, with adaptive
threshold f1-score was observed to $20.32\%$ times $\geq 80\%$ w.r.t the total
number of samples; whereas $19.54\%$, $5.31\%$, and $0.00\%$ for fixed
thresholds at 0$.3$, $0.5$, and $0.7$ respectively. This project has been
tested on a concise dataset, so in the future, researchers can explore more on
such techniques that can make decision threshold adapt to varying identities-
database size and indistinguishable features.

## Acknowledgement

The author declares that the research was conducted in the absence of grants
or financial support from any institution or organization. However,
deHumlaTech AI Research team would like to thank Mr. Bibek Raj Shrestha and
Mr. Baibhav Raj Shrestha, for providing the necessary hardware components like
RAM-16GB and SSD-500GB to encounter its higher computational requirements.

## References

  * [1] H.-R. Chou, J.-H. Lee, Y.-M. Chan, and C.-S. Chen, “Data-specific Adaptive Threshold for Face Recognition and Authentication,” Tech. Rep.
  * [2] Q. Zou, S. Xie, Z. Lin, M. Wu, and Y. Ju, “Finding the Best Classification Threshold in Imbalanced Classification,” _Big Data Research_ , vol. 5, pp. 2–8, sep 2016.
  * [3] Z. C. Lipton, C. Elkan, and B. Naryanaswamy, “Optimal thresholding of classifiers to maximize F1 measure,” in _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ , vol. 8725 LNAI, no. PART 2. Springer Verlag, 2014, pp. 225–239. [Online]. Available: https://link.springer.com/chapter/10.1007/978-3-662-44851-9{_}15
  * [4] A. Hartmann, “Adaptive threshold based spam classification,” sep 2009. [Online]. Available: http://outlook.spambully.com/about.php
  * [5] R. Bauer and A. Gharabaghi, “Reinforcement learning for adaptive threshold control of restorative brain-computer interfaces: A Bayesian simulation,” _Frontiers in Neuroscience_ , vol. 9, no. FEB, 2015.
  * [6] F. Schroff and J. Philbin, “FaceNet: A Unified Embedding for Face Recognition and Clustering,” Tech. Rep.
  * [7] D. E. King, “Dlib-ml: A Machine Learning Toolkit,” Tech. Rep., 2009.
  * [8] G. B. Huang and E. Learned-Miller, “Labeled Faces in the Wild: Updates and New Reporting Procedures,” Tech. Rep. [Online]. Available: http://vis-www.cs.umass.edu/
  * [9] “dataremixed/the-worlds-highest-paid-athletes-list — Workspace — data.world.” [Online]. Available: https://data.world/dataremixed/the-worlds-highest-paid-athletes-list/workspace/file?filename=The+World{%}27s+Highest-Paid+Athletes+List.xlsx
  * [10] B. Bohara, “Online-dataset-maker.” [Online]. Available: https://github.com/Varat7v2/online-dataset-maker


# Take the aTrain. Introducing an Interface for the Accessible Transcription
of Interviews.

Armin Haberl  Jürgen Fleiß  Dominik Kowald  Stefan Thalmann

###### Abstract

aTrain is an open-source and offline tool with an easy-to-use graphical
interface for transcribing audio data in multiple languages with CPU and
NVIDIA GPUs. It is designed for researchers using qualitative data generated
from various forms of speech interactions. aTrain requires no programming
skills, runs on most computers, does not require an internet connection, and
was verified not to upload data to any server. aTrain combines OpenAI’s
Whisper model with speaker recognition to provide output that integrates with
MAXQDA and ATLAS.ti. It is provided through the Microsoft Store for simple
installation. The source code is freely available on GitHub. Having developed
aTrain with a focus on speed on local computers, we show that the
transcription time on current mobile CPUs is around 2 to 3 times the duration
of the audio file using highest-accuracy transcription models. With an entry-
level graphics card, transcription speed increases to 20% of the audio
duration.

###### keywords:

transcription , local , Whisper , AI , machine learning , qualitative research
, interview transcription , qualitative data analysis

###### JEL:

C65 , C88 , Z19

††journal: Journal of Behavioral and Experimental Finance

[inst1]organization=Business Analytics and Data Science-Center, University of
Graz,addressline=Universitätsstraße 15, city=Graz, postcode=8010,
country=Austria [inst2]organization=Know Center & Graz University of
Technology,addressline=Sandgasse 36, city=Graz, postcode=8010, country=Austria

##  1 Introduction

> ”Transcribing interviews is time-consuming and potentially costly work. It
> can be facilitated by using a transcribing machine that has a foot pedal and
> earphones”.
>
> (Seidman, 2013, p. 118)

In the ten years after the foot pedal was hailed as the state of the art for
effectively transcribing interviews, the tools available have developed
rapidly with the rise of artificial intelligence (AI). While the transcription
of an interview of one hour requires up to six hours of manual work (Bell et
al., 2018), advances in AI-based tools have sped up this process,
significantly reducing the necessary transcription work to a fraction compared
to manual transcription. An example of such an AI-based transcription model is
Whisper, developed by OpenAI in 2023 (Radford et al., 2023). Whisper uses a
transformer model (Vaswani et al., 2017) to facilitate fast automated
transcriptions with accuracy and robustness comparable to human transcribers
(Radford et al., 2023). However, speed and accuracy of the transcription are
not the only criteria relevant to qualitative researchers. When comparing the
available automated transcription tools, Wollin-Giering et al. (2023) recently
outlined several criteria for the comparison of automatic transcription tools:
data protection, accuracy, time spent, and costs.

Although open-source models such as Whisper perform generally well in all
these criteria (Wollin-Giering et al., 2023), they still lack ease of use and
rely on command-line interfaces. This can be a barrier for researchers without
programming knowledge, especially when installed locally and operated on their
own devices. They also lack output formats that integrate into common QDA
software such as MAXQDA or ATLAS.ti. In addition to open-source solutions such
as Whisper, there are several paid subscription tools from commercial
providers such as Trint, Descript, or Sonix (see e.g., Liyanagunawardena,
2019; Wollin-Giering et al., 2023) that provide sufficient ease of use for
researchers. The problem with some of these existing tools is their cloud-
native nature, which requires users to upload their interviews to external
servers for transcription. Especially in the context of the EU’s data
protection legislation (GDPR), this practice of uploading potentially
sensitive personal information to cloud services requires explicit consent
from interviewees (European Parliament and Council of the European Union,
2016), which they may not be willing to give. Also from the perspective of
ethics committees approving qualitative studies, privacy preserving
technologies are demanded.

To help researchers use high-quality open source tools for local interview
transcription, we developed a tool providing accessible transcription of
interviews: aTrain.

Figure 1: aTrain user interface

We developed aTrain as a free, open-source, encapsulated, self-installing, and
completely offline alternative to existing solutions. The following article
presents the main features and details of the technical implementation and
benchmarks of the transcription time on different computer configurations. We
alleviate three main issues in using free open-source automated transcription
on local computers: long runtime, difficulty in setup and use, and integration
in the qualitative data analysis software workflow.

To reduce runtime issues and improve on other open source solutions, we make
use of the faster-whisper framework, which reduces runtime on CPUs by the
factor of 4 to 5 (Klein, 2023), thus making local transcription on typical
business notebooks feasible. In our benchmarks, we found that mid-range
business notebooks, as are common in university settings, allow one to
transcribe one hour of audio with the highest accuracy whisper models in only
about 2 and a half to four hours (depending on the hardware). Additionally,
aTrain also supports CUDA-enabled NVIDIA graphics cards, drastically reducing
the time required to transcribe interviews.

Regarding ease of use, we offer a Microsoft Store deployment for Windows users
and a simple graphical user interface (Figure 1, thus eliminating the barriers
of command line setup and use of Whisper. Furthermore, we also offer an export
format that allows integration in MAXQDA and ATLAS.ti to allow researchers to
continue directly working in established software tools after transcription
with aTrain (see Figure 2).111To the best of our knowledge, one other open
source tool has been developed to alleviate difficulty in setup and use:
noScribe. However, the issues of long runtime on CPUs and lack of GPU support
have been pointed out as a trade-off that must be made when using this
solution (Wollin-Giering et al., 2023), and that we hope to alleviate.

Finally, addressing integration in QDA software workflow, we provide an output
format for syncing audio with transcript passages in the widely used software
tools MAXQDA and ATLAS.ti.

##  2 Main Features

This section provides an overview of the features that the current version of
aTrain offers its users. For an in-depth description of the technical
implementation of these features, refer to later chapters.

###  2.1 Local installation

To simplify the installation process of aTrain as much as possible, we
developed a MSIX package of the application and provide it through the
Microsoft Store (see e.g., Microsoft, 2021). The installation does not require
administrator rights on the local machine and should even work on systems that
are centrally administrated, for example, by university IT departments. aTrain
is available through the following link to the Microsoft Store:
apps.microsoft.com/store/detail/atrain/9N15Q44SZNS2.

The source code can be found in the GitHub repository under
github.com/BANDAS-Center/aTrain.

###  2.2 Automated transcription

The central feature of aTrain is its ability to automatically transcribe audio
and video files (e.g., mp3, mp4, wav and most others) containing speech.222For
a full list of supported formats, see ffmpeg.org/ffmpeg-formats.html After
selecting the input file from the local hard drive, aTrain will use its
internal ML models to transcribe the contained speech and output the text,
including time stamps. There is also an option to specify the ML models which
should be used for transcription, which vary in accuracy and speed of
transcription.

It should be noted that in qualitative research, several distinct forms of
transcriptions are differentiated, with varying level of detail and the in-
and exclusion of phonetic components. A systematic discussion of the different
systems and how AI-based translation relates to them is presented by Wollin-
Giering et al. (2023) who conclude that currently all automated systems omit
phonetic information. This excludes certain types of qualitative analysis
methods that focus on or include phonetic information. Instead, they provide a
reproduction of the text approaching a verbatim transcription of what was
said, but omitting information like pauses or filler words. However, while the
system on which we built our tool, Whisper, was found to be the most accurate
and detailed (Wollin-Giering et al., 2023), it is essential that the
researcher reviews the transcript and compares it with the original recording.
We highly recommend that researchers using aTrain consult Wollin-Giering et
al. (2023) for a systematic review and discussion of current limitations,
capabilities, and advantages of automated transcription tools.

###  2.3 Speaker detection

aTrain additionally offers speaker detection based on Pyannote.Audio (see
Bredin et al., 2020), assigning text passages to the corresponding speakers.
Users can either set the number of speakers in a recording, or let aTrain
automatically detect the number of speakers. We recommend specifying the
speaker count, as we found that it improves the accuracy of the clustering
algorithm used for speaker detection.

###  2.4 Multi-language Support

The inputted recordings can contain speech in any of the 57 languages
available in the underlying ML models used by aTrain (Radford et al., 2023).
While aTrain can generally transcribe speech in any of these languages, the
quality of the transcriptions is generally better for the languages that were
predominant in the underlying training data sets(Radford et al., 2023).
Additionally, currently only a single language can be processed for the
complete interview.333One workaround would be to split your audio file into
segments based on language manually, using an audio or video editing tool.
This would only be feasible, of course, for larger segments with the same
language.

In addition to transcription, translation of audio material into English is
also possible by setting the language option to English for non-English source
material. Translation to other languages is not supported.

###  2.5 Output Formats

The transcribed text will be exported from aTrain as a text file that contains
timestamps for each text segment and optionally information about the
corresponding speakers. The following files are provided:

  * 1.

A plain text file containing the transcripts including timestamps and, if
selected, speaker information.

  * 2.

A plain text file containing the transcripts without timestamps and, if
selected, speaker information.

  * 3.

A version of the plain text file formatted for QDA software import.

  * 4.

A json file containing the complete raw-transcript information, allowing users
to construct their own output formats.

To ensure easy integration into existing research workflows, we also provide
an output format that was designed for seamless import into the commonly used
QDA software tools ATLAS.ti and MAXQDA. This then allows to sync the audio or
video file to the corresponding points in the transcript, thus playing the
corresponding audio or video file by clicking the corresponding text in the
transcript.

Figure 2: Example of a transcript with synced audio in MAXQDA 2022.

###  2.6 Offline Execution and Data Privacy

aTrain is designed to run completely offline without the need for an internet
connection. Thus, both the transcription model and the data set operate in
memory and do not leave the local machine. The same is true for the
transcribed text. This means that no data are sent to any server, which
ensures data privacy and supports GDPR compliance.

The version of aTrain referenced in this article and its components have been
reviewed with regard to them sharing data online by one of the authors with a
programming background. To ensure that updates to the used components do not
introduce the transfer of GDPR-related data in updates, we include all
components in the provided installer. This results in a large installer size
of around 12.8 GB.

##  3 Technology and Programming

###  3.1 Hard- and Software Requirements

We tested aTrain on various computer configurations to ensure that it runs on
different hardware as well as on more powerful systems with a CUDA-enabled
Nvidia GPU.444For a list of CUDA-enabled GPUs see
https://developer.nvidia.com/cuda-gpus, accessed on 02 October 2023. While we
expect aTrain to also work on similar and less powerful hardware, we do not
have data to definitively support this claim.

aTrain can run on a CPU or on a CUDA-enabled NVIDIA GPU, the latter requiring
the installation of the CUDA toolkit (NVIDIA, 2023). While a CUDA-enabled
NVIDIA GPU is not required to run aTrain, it significantly improves the speed
of transcriptions and speaker detection, as reported in Section 4. Currently,
aTrain is only available for computers running Windows 11. There are no
additional software dependencies or requirements.

###  3.2 Software Packages and Architecture

The general architecture of the aTrain desktop application is based on web
technologies using the Python programming language and the flask web framework
(Pallets Projects, 2021). The user interface was built with HTML, CSS and
JavaScript utilizing libraries such as tailwindCSS (Tailwind Labs, 2023),
daisyUI (Saadeghi, 2023), alpine (Porzio, 2023) and htmx (Big Sky Software,
2023).

The main pipeline used for transcription and speaker detection includes
several ML models and data processing steps. An important goal in composing
this pipeline was to reduce the time needed for transcriptions and speaker
detection and to limit the number of ML models included in the final MSIX
installer. The final pipeline is depicted in Figure 3.

Figure 3: Transcription pipeline with corresponding inputs, tools and outputs

The pipeline is initiated once the user provides a speech recording to the
system, which is first converted to the WAV format using a Python binding for
the ffmpeg library (Kroenig, 2019). Subsequently, the resulting WAV file is
used as input for the Whisper transcription model (Radford et al., 2023)
specified by the user. aTrain uses the faster-whisper implementation of
Whisper, which is optimized for speed and memory efficiency, which reduces the
transcription time on CPUs by a factor of 4 to 5 (Klein, 2023). The output of
the Whisper model contains the transcribed text segments of the speech
recording, including time stamps. To enable additional speaker detection,
aTrain utilizes a modified version of the Pyannote.Audio speaker detection
pipeline as described in Bredin et al. (2020). An alignment function developed
by Bain et al. (2023) is ultimately used to combine the outputs of the
transcription and speaker detection models into the final output.

###  3.3 Source code, collaboration and license

We developed aTrain using Git as the version control system and share the
source code through GitHub under github.com/BANDAS-Center/aTrain. With this,
we also want to encourage collaboration with other developers. For future
versions of aTrain, we would greatly appreciate the input of other developers
and qualitative researchers to expand its features.

aTrain is published under an adaptation of the MIT license, where we ask users
to cite this paper when using aTrain for academic or other publications.

##  4 Transcription Duration on Different Whisper Models

To enable a comparison of aTrain against existing implementations, we want to
provide crucial performance metrics. While we did not develop any ML models
ourselves in this project, we did use and combine them in a novel way.
Important performance indicators for automatic transcription tools include
factors such as transcription accuracy and processing time (see e.g., Wollin-
Giering et al., 2023). The accuracy of aTrain’s transcriptions and speaker
detection depends entirely on the underlying models developed by Radford et
al. (2023) and Bredin et al. (2020). Accuracy benchmarks and error rates for
those models were already documented in the respective papers and are
therefore not duplicated in this publication. In general, however, a recent
comparison between various commercial and open source solutions found Whisper
to be the most accurate tool available, for both a German and an English
sample interview (Wollin-Giering et al., 2023).

A main performance indicator is the total processing time that results from
aTrain’s transcription and speaker detection pipeline. To test this processing
time, we used aTrain to transcribe an audio book which was chosen from the
freely available Librispeech corpus (Panayotov et al., 2015). To emulate the
typical length of interviews in qualitative research, we opted to use a
recording of Hans Christian Andersen’s fairy tail ’The Snow Queen’, which was
narrated as an audiobook in 1 hour, 13 minutes, and 38 seconds (see LibriVox,
2006; Panayotov et al., 2015).

This audiobook was transcribed with all available Whisper models from the
faster-whisper implementation (Klein, 2023) and with speaker detection
activated (Bredin et al., 2020). Additionally, we ran the transcriptions on
three different computers to test the processing time on different CPUs and
also on a CUDA-enabled NVIDIA GPU. The exact specifications of the hardware
used during testing and benchmarking are listed in Table 1.

Machine | Year | CPU | RAM | GPU
---|---|---|---|---
Dell Latitude 5530 | 2023 |  Intel i5-1245U | 16GB |  CPU integrated
Lenovo Thinkpad P14s | 2022 |  AMD Ryzen 7 PRO 6850U | 32GB |  CPU Integrated
Lenovo Legion Y740 | 2019 |  Intel i7-8750H | 16GB |  Nvidia RTX 2070 MaxQ 8GB

Table 1: Hardware specifications of our experiments. Please note that the
computing device used in the benchmarks is highlighted in bold.

The processing time is recorded by aTrain using timestamps generated at the
beginning and end of every transcription and displayed as the total duration
after the transcription is complete.

Figure 4: Duration of transcription on different hardware

Figure 4 shows the processing time of each transcription relative to the
length of the speech recording. In this relative processing time (RPT), a
transcription is considered ’real time’ when the recording length and the
processing time are equal. Subsequently, faster transcriptions lead to an RPT
below 1 and slower transcriptions to an RPT time above 1.

In alignment with previous expectations, the RPT increases significantly with
the size of the Whisper model used. Up to the small Whisper model, the RPT was
consistently below 1 for every computer tested. Larger models resulted in RPTs
ranging between 1 and 3.5 when running on CPU, while the test machine using a
CUDA-enabled NVIDIA GPU (comparable to a 2023 entry-level GPU) could run
transcriptions with an RPT around 0.2 even on the largest models. Using
standard hardware without GPU (12th gen Intel, Ryzen 7) is nevertheless
feasible when running on the medium Whisper model, in which we found a good
balance between processing time and transcription quality. However, also the
use of large models without a GPU is possible; especially on potent multicore
CPUs, where we observed an RPT of around 2 for the large Whisper models. Note
that buying an entry-level gaming notebook with a dedicated GPU to transcribe
interviews with five times and more real-time speed would cost around the same
as a paid automated transcription single-user license starting at around 600€
(e.g., https://app.trint.com/plans, accessed on 3 October 2023).

##  5 Conclusion

In the paper on hand, we introduce the open-source tool aTrain, which enables
the transcription (including speaker recognition) of audio recordings. The
software components in aTrain were reviewed by one of the authors with a
Computer Science background to ensure that it runs completely on the local
computer on which it is installed and does not send any data to online
servers, thus helping researchers maintain data privacy requirements arising
from ethical guidelines or to comply with legal requirements such as the GDPR.

It comes with an easy-to-use graphical interface, runs on both CPUs and CUDA-
enabled NVIDIA GPUs, and is available through the Microsoft Store on Windows
computers. aTrain accepts most common audio and video formats as input and
produces a transcript for analysis using the Whisper model introduced by
OpenAI, providing best-in-class accuracy among current commercial and open-
source automated audio transcription solutions. It also provides an output
format for import into the QDA software solutions MAXQDA and ATLAS.ti,
allowing researchers to play the audio recording corresponding to text
passages of the transcript with a single click.

We hope that this tool enables researchers who use audio recordings of various
forms of interviews, either as their main data source or as a supplement to
more quantitative oriented studies like laboratory experiments, to save both
time and monetary resources in transcribing their audio recordings.

## References

  * Bain et al. (2023) Bain, M., Huh, J., Han, T., Zisserman, A., 2023.  WhisperX: Time-Accurate Speech Transcription of Long-Form Audio.  INTERSPEECH 2023 URL: https://arxiv.org/abs/2303.00747.
  * Bell et al. (2018) Bell, E., Bryman, A., Harley, B., 2018.  Business Research Methods.  Oxford University Press, Oxford.
  * Big Sky Software (2023) Big Sky Software, 2023.  Htmx: Version 1.9.3.  URL: https://github.com/bigskysoftware/htmx/releases/tag/v1.9.3.
  * Bredin et al. (2020) Bredin, H., Yin, R., Coria, J.M., Gelly, G., Korshunov, P., Lavechin, M., Fustes, D., Titeux, H., Bouaziz, W., Gill, M.P., 2020.  Pyannote.audio: Neural building blocks for speaker diarization, in: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 7124–7128.  doi:10.1109/ICASSP40776.2020.9052974.
  * European Parliament and Council of the European Union (2016) European Parliament, Council of the European Union, 2016.  Regulation (EU) 2016/679 of the European Parliament and of the Council.  URL: https://data.europa.eu/eli/reg/2016/679/oj.
  * Klein (2023) Klein, G., 2023.  faster-whipser: Version 0.7.1.  URL: https://github.com/guillaumekln/faster-whisper/releases/tag/v0.7.1.
  * Kroenig (2019) Kroenig, K., 2019.  ffmpeg-python: Version 0.2.0.  URL: https://github.com/kkroening/ffmpeg-python/releases/tag/0.2.0.
  * LibriVox (2006) LibriVox, 2006.  Andersen’s fairy tales: The snow queen.  URL: https://archive.org/details/andersens_fairytales_librivox.
  * Liyanagunawardena (2019) Liyanagunawardena, T.R., 2019.  Automatic transcription software: good enough for accessibility? a case study from built environment education, in: European Distance and E-Learning Network (EDEN) Conference Proceedings, European Distance and E-Learning Network. pp. 388–396.
  * Microsoft (2021) Microsoft, 2021.  What is msix?  URL: https://learn.microsoft.com/en-us/windows/msix/overview.
  * NVIDIA (2023) NVIDIA, 2023.  Cuda toolkit documentation 12.2 update 2.  URL: https://docs.nvidia.com/cuda/.
  * Pallets Projects (2021) Pallets Projects, 2021.  Flask: Version 2.3.3.  URL: https://github.com/pallets/flask/releases/tag/2.3.3.
  * Panayotov et al. (2015) Panayotov, V., Chen, G., Povey, D., Khudanpur, S., 2015.  Librispeech: An asr corpus based on public domain audio books, in: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE. pp. 5206–5210.  doi:10.1109/ICASSP.2015.7178964.
  * Porzio (2023) Porzio, C., 2023.  Alpine: Version 3.13.0.  URL: https://github.com/alpinejs/alpine/releases/tag/v3.13.0.
  * Radford et al. (2023) Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I., 2023.  Robust speech recognition via large-scale weak supervision, in: International Conference on Machine Learning, PMLR. pp. 28492–28518.
  * Saadeghi (2023) Saadeghi, P., 2023.  Daisyui: Version 3.5.1.  URL: https://github.com/saadeghi/daisyui/releases/tag/v3.5.1.
  * Seidman (2013) Seidman, I., 2013.  Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences.  Teachers College Press, New York and London.
  * Tailwind Labs (2023) Tailwind Labs, 2023.  Tailwindcss: Version 3.3.3.  URL: https://github.com/tailwindlabs/tailwindcss/releases/tag/v3.3.3.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017.  Attention is all you need.  Advances in Neural Information Processing Systems 30.
  * Wollin-Giering et al. (2023) Wollin-Giering, S., Hoffmann, M., Höfting, J., Ventzke, C., 2023.  Automatic transcription of qualitative interviews.  Sociology of Science Discussion Papers URL: https://www.static.tu.berlin/fileadmin/www/10005401/Publikationen_sos/Wollin-Giering_et_al_2023_Automatic_transcription.pdf.


# On open quantum systems, effective Hamiltonians and device characterization

S.N.A. Duffus  Quantum Systems Engineering Research Group, Loughborough
University, Loughborough, Leicestershire LE11 3TU, United Kingdom  Department
of Physics, Loughborough University  V.M. Dwyer<EMAIL_ADDRESS>Quantum
Systems Engineering Research Group, Loughborough University, Loughborough,
Leicestershire LE11 3TU, United Kingdom  The Wolfson School of Mechanical,
Electriical and Manufacturing Engineering, Loughborough University  M.J.
Everitt  Quantum Systems Engineering Research Group, Loughborough University,
Loughborough, Leicestershire LE11 3TU, United Kingdom  Department of Physics,
Loughborough University

###### Abstract

High fidelity models, which support accurate device characterization and
correctly account for environmental effects, are crucial to the engineering of
scalable quantum technologies. As it ensures positivity of the density matrix,
one preferred model for open systems describes the dynamics with a master
equation in Lindblad form. The Linblad operators are rarely derived from first
principles, resulting in dynamical models which miss those additional terms
that must generally be added to bring the master equation into Lindblad form,
together with concomitant other terms that must be assimilated into an
effective Hamiltonian. In first principles derivations such additional terms
are often canceled (countered), frequently in an ad hoc manner. In the case of
a Superconducting Quantum Interference Device (SQUID) coupled to an Ohmic
bath, the resulting master equation implies the environment has a significant
impact on the system’s energy. We discuss the prospect of keeping or canceling
this impact; and note that, for the SQUID, measuring the magnetic
susceptibility under control of the capacitive coupling strength and the
externally applied flux, results in experimentally measurable differences
between models. If this is not done correctly, device characterization will be
prone to systemic errors.

For any complex system, if device characterization and design engineering are
to be meaningful notions, it is necessary to have access to device models
which faithfully describe behavior and performance when subjected to a wide
range of environmental conditions. The ability to integrate billions of CMOS
transistors onto a single chip relies on careful device characterization as
well as knowledge of likely environmental conditions. A chip meant for liquid
Helium applications will be characterized by lower junction capacitances, as a
result of carrier freeze-out Yoshikawa  _et al._ (2005), whilst if intended
for space applications, it must be designed with radiation hardness in mind
Pouiklis  _et al._ (2013). Successful Quantum Technologies will be no
different in this regard. In the quest to build a scalable quantum processor
from superconducting circuits containing Josephson junctions, significant
developments have been made recently; including enhanced coherence times Houck
_et al._ (2008), high fidelity state-preparation and measurement Jeffrey  _et
al._ (2014), and the demonstration of two-qubit gates Plantenberg  _et al._
(2007). To facilitate the fabrication of functional circuits, future qubits
must be capable of long and short term storage; be frequency tunable; be
capable of communication over both short and long ranges, and possess tunable
couplings Steffen  _et al._ (2009). Decoherence presents a formidable obstacle
to achieving this for superconducting qubits. As a result, the maintenance of
a long coherence time has become a key problem in superconductor qubit
research; making the need for a better understanding of different types of
environment a similarly key concern.

Until relatively recently it had been assumed that the connection to ground of
floating flux-qubits was poor and that such qubits were immune to capacitive
coupling. However a recent study Steffen  _et al._ (2009) has demonstrated the
reactance to ground can become sizeable, and that not protecting against this
decoherence channel may explain some of the short coherence times obtained
previously. The equivalent design paradigm for a classical semiconductor
(CMOS) computer allows a very detailed simulation of critical-path devices in
a variety of environments (at fast/typical/slow mobility process corners;
under mechanical stress, flicker or thermal noise, etc.) as well in a range of
operational conditions (IR-drop on power lines) and including a number of
reliability considerations (metal migration) before it is committed to silicon
Morshed  _et al._ (2011). In the particular case of capacitive coupling, the
contrast of the superconducting qubit to silicon CMOS is stark; indeed CMOS is
the chosen platform because of its excellent noise immunity. Understanding the
impact of an environment is necessary for accurate characterization of a given
quantum device; as devices which are not characterized under the same
environmental conditions may not be easily compared, and a device
characterized in a test environment, may behave differently in use.

A common means of modeling an open quantum system is with a reduced master
equation Arrigoni  _et al._ (2013); Breuer and Petruccione (2007); Dyre
(1987); Giovannetti and Palma (2012); Wu  _et al._ (2009); Ferialdi (2016);
Hall  _et al._ (2014); Maldonado-Mundo  _et al._ (2012); Joshi  _et al._
(2014); Vacchini (2016); Diósi and Ferialdi (2014), as these approximate the
dynamics of the system of interest without the need to model the environment
itself, and, because it guarantees certain nice properties (such as
conservation of probability and complete positivity of the reduced (system)
density matrix), it is usual to cast the master equation into Lindblad form
Lindblad (1976). In this model Lindblad operators represent the action of the
environment on the system. Exactly which operators to use usually involves a
careful construction analysis of the system + environment, followed by a
tracing out of the environmental degrees of freedom. This process requires a
number of approximations, as well as some decisions about the form of the
final equation. Most common is the Born-Markov (BM) approximation which
assumes a weak coupling between a quantum system and its environment, assumed
to have no long-term memory. However the BM approximation alone will not
generally yield a quantum master equation in Lindblad form. As a consequence,
it is common to manipulate the resulting BM master equation by adding those
terms necessary to complete the Lindblad structure and by either absorbing any
terms which present as free evolution into an effective system Hamiltonian, or
by canceling them with heuristic counter terms.

The simplest example is that of Quantum Brownian Motion (QBM), with a Harmonic
Oscillator (HO) potential, coupled to an Ohmic bath through the system’s
position operator $\hat{x}$, e.g. Breuer and Petruccione (2007). Here a
Lindblad master equation can be created from Born-Markov development by: (i)
adding a term of the form $D_{pp}[\hat{p},[\hat{p},\hat{\rho}]]$, where
$\hat{p}$ is the canonical conjugate momentum to $\hat{x}$
($\left[{\hat{x}},{\hat{p}}\right]=\mathrm{i}\hbar$), and $D_{pp}$ is
inversely proportional to the bath temperature $T$; and (ii) by moving terms
proportional to $[\hat{x}^{2},\hat{\rho}]$ (the Lamb shift term) and
$\mu[\hat{x}\hat{p}+\hat{x}\hat{p},\hat{\rho}]$ (a squeezing term, where $\mu$
depends on system-bath coupling strength) into an effective system
Hamiltonian. The justification for the addition of the
$D_{pp}[\hat{p},[\hat{p},\hat{\rho}]]$ term is often that, in the high $T$
limit, it will is small and so amounts to a minimally invasive means of
ensuring Lindblad form. The Lamb shift term $[\hat{x}^{2},\hat{\rho}]$ is
generally canceled by the inclusion of an identical (counter) term in the
system or interaction Hamiltonian on the grounds that it otherwise constitutes
an unphysical frequency renormalization.

Up to this point what has been described is the standard high-temperature
(Lindblad) version of the Caldiera-Leggett equation Caldeira and Leggett
(1983), who argued that, for many systems, a counter term arises to cancel the
Lamb shift in a truly detailed analysis of microscopic dynamics. The final
(squeezing term) $\mu[\hat{x}\hat{p}+\hat{p}\hat{x},\hat{\rho}]$, has been
discussed in the literature on a number of occasions. On the one hand its
presence (uncountered) in an effective Hamiltonian is necessary to ensure:
translational invariance Gao (1997); that Erhenfest’s theorem is satisfied
Lindblad (1976); Sǎndulescu and Scutaru (1987); Gao (1997); Wiseman and Munro
(1998); Lampo  _et al._ (2016); and a correct quantum to classical transition
Schack  _et al._ (1995); Everitt (2009); Everitt  _et al._ (2010, 2011, 2009).

On the other hand, the inclusion of a counter term
$-\lambda[\hat{x}\hat{p}+\hat{p}\hat{x},\hat{\rho}]$ in the system Hamiltonian
has occasionally been part of an ansatz which assumes the most general second
order Lindblad master equation possible, using a single (annihilator)
Lindblad, first order in system variables $\hat{x}$ and $\hat{p}$, together
with a general second order Hamiltonian. Its properties have been investigated
as a function of $\lambda$. Setting $\lambda=\mu$ cancels the squeezing term
completely, while $\lambda\neq 0$ gives additional flexibility to add
desirable properties to the master equation, such as reasonable low $T$
behavior Gao (1997). This does however sacrifice translational invariance
Lindblad (1976), and Ehrenfest Wiseman and Munro (1998).

The difficulties with the derivation of the Lindblads appropriate to a
particular system-environment interaction are clearer still when considering a
particle in an anharmonic potential, such as is the case for a Superconducting
Quantum Interference Device (SQUID). Here the heuristic adding/canceling of
terms becomes more involved Duffus  _et al._ (2016), which necessarily raises
difficult issues, as high-precision control of such a system inevitably
requires precise device characterization Cole  _et al._ (2005).

We consider the following standard model of a SQUID (charge $\hat{Q}$ and flux
$\hat{\Phi}$, $[\hat{\Phi},\hat{Q}]=\mathrm{i}\hbar$), inductively and
capacitively coupled to an Ohmic bath environment, modeled as an infinite set
of harmonic oscillators, indexed by $n$, with charge $\hat{Q}_{n}$ and flux
$\hat{\Phi}_{n}$ ($[\hat{\Phi}_{n},\hat{Q}_{m}]=\mathrm{i}\hbar\delta_{nm}$).
The dimensionless total Hamiltonian $\hat{H}/\hbar\omega_{0}$ may be written
as a sum of the Hamiltonians of the SQUID ($\hat{\mathcal{H}}_{S}$), the bath
($\hat{\mathcal{H}}_{B}$), the coupling between them $(\hat{\mathcal{H}}_{I})$

$\displaystyle\hat{\mathcal{H}}_{S}$ $\displaystyle=$
$\displaystyle\frac{\hat{X}^{2}}{2}+\frac{\hat{P}^{2}}{2}-\frac{\nu}{\omega_{0}}\cos\left(\sqrt{\frac{\beta\nu}{\omega_{0}}}\hat{X}+\frac{2\pi\Phi_{x}}{\Phi_{0}}\right)+\hat{\mathcal{H}}_{LS}$
$\displaystyle\hat{\mathcal{H}}_{B}$ $\displaystyle=$
$\displaystyle\sum_{n}\frac{\hat{X}^{2}_{n}}{2}+\frac{\hat{P}_{n}^{2}}{2},\ \
\hat{\mathcal{H}}_{I}=-\sum_{n}\kappa_{n}\left(\hat{X}\hat{X}_{n}+g\hat{P}\hat{P}_{n}\right)$
(1)

where $L$ and $C$ are the system’s inductance and capacitance,
$\omega_{0}=1/\sqrt{LC}$, $\hat{P}=\hat{Q}(\hbar C\omega_{0})^{-1/2}$,
$\beta\nu=4\pi^{2}\hbar/\Phi_{0}^{2}C$, $\hat{P}_{n}=\hat{Q}_{n}(\hbar
C_{n}\omega_{0})^{-1/2}$, $\hat{X}=\hat{\Phi}(\hbar L\omega_{0})^{-1/2}$, and
$\hat{X}_{n}=\hat{\Phi}_{n}(\hbar L_{n}\omega_{0})^{-1/2}$, while $\Phi_{0}$
denotes the flux quantum. The externally applied flux, $\Phi_{x}$, controls
the phase of the cosine in Eq. (On open quantum systems, effective
Hamiltonians and device characterization) which describes the coupling across
the Josephson junction (energy $=\hbar\nu$) Devitt  _et al._ (2013). The
strengths of the environmental couplings are determined by $\kappa_{n}$ and
$g$, the ratio of inductive to capacitive coupling. $\hat{\mathcal{H}}_{LS}$
is the Lamb Shift Hamiltonian which will be canceled in the Lindbald process
Caldeira and Leggett (1983).

In the interaction picture, the BM master equation is

$\begin{split}&\frac{\mathrm{d}{\hat{\rho}}_{S}(t^{\prime})}{\mathrm{d}t^{\prime}}=\mathcal{L}[{\hat{\rho}}]+\mathcal{K}[{\hat{\rho}}]\\\
&=-\mathrm{i}[\hat{\mathcal{H}}_{S},{\hat{\rho}}_{S}(t^{\prime})]-\int_{0}^{\infty}\mathrm{d}\tau\operatorname{Tr}_{B}\left[\hat{\mathcal{H}}_{I},\left[\hat{\mathcal{H}}_{I}(-\tau),{\hat{\rho}}_{S}(t^{\prime})\otimes{\hat{\rho}}_{B}\right]\right]\end{split}$

Breuer and Petruccione (2007).The first term describes free evolution and the
second, the dissipator, describes non-unitary loss. We assume an Ohmic
spectral function
$J(\omega)=2C\gamma\omega\Omega^{2}/\pi(\Omega^{2}+\omega^{2})$ with damping
rate $\gamma$ and bath cut-off frequency $\Omega$. Substituting
$\hat{\mathcal{H}}_{I}$, from Eq. (On open quantum systems, effective
Hamiltonians and device characterization), into the dissipator integral and
evaluating the bath correlation functions yields Breuer and Petruccione (2007)

$\displaystyle\mathcal{K}[{\hat{\rho}}]$ $\displaystyle=$
$\displaystyle\frac{\gamma\Omega}{2\omega_{0}}\int_{0}^{\infty}\mathrm{d}\tau
e^{-\Omega\tau}\Bigg{(}\frac{2\Omega}{\omega_{0}}\bigg{(}\mathrm{i}\left[{\hat{X}},{\left\\{{\hat{X}(-\tau)},{{\hat{\rho}}_{S}}\right\\}}\right]+\mathrm{i}g^{2}\left[{\hat{P}},{\left\\{{\hat{P}(-\tau)},{{\hat{\rho}}_{S}}\right\\}}\right]+g\left[{\hat{X}},{\left[{\hat{P}(-\tau)},{{\hat{\rho}}_{S}}\right]}\right]-g\left[{\hat{P}},{\left[{\hat{X}(-\tau)},{{\hat{\rho}}_{S}}\right]}\right]\bigg{)}$
(2)
$\displaystyle-\bigg{(}\left[{\hat{X}},{\left[{\hat{X}(-\tau)},{{\hat{\rho}}_{S}}\right]}\right]+g^{2}\left[{\hat{P}},{\left[{\hat{P}(-\tau)},{{\hat{\rho}}_{S}}\right]}\right]-\mathrm{i}g\left[{\hat{X}},{\left\\{{\hat{P}(-\tau)},{{\hat{\rho}}_{S}}\right\\}}\right]+\mathrm{i}g\left[{\hat{P}},{\left\\{{\hat{X}(-\tau)},{{\hat{\rho}}_{S}}\right\\}}\right]\bigg{)}\Bigg{)}$

where $\left\\{{\cdot},{\cdot}\right\\}$ denotes anti-commutation. Then,
evaluating the integrals, and expanding the correlation time-dependent
observables $\hat{X}(-\tau)$ and $\hat{P}(-\tau)$ to first order, yields the
master equation:

$\displaystyle m\frac{\mathrm{d}{\hat{\rho}}_{S}}{\mathrm{d}t^{\prime}}$
$\displaystyle=$
$\displaystyle-\mathrm{i}\left[{\hat{\mathcal{H}}_{S}},{{\hat{\rho}}_{S}}\right]-\frac{\mathrm{i}\gamma}{\omega_{0}}\left(1+g^{2}-g\right)\left[{\hat{X}},{\left\\{{\hat{P}},{{\hat{\rho}}_{S}}\right\\}}\right]-\frac{\mathrm{i}\gamma}{\omega_{0}}g\left(g-\frac{1}{2}\right)\left[{\left\\{{\hat{X}},{\hat{P}}\right\\}},{{\hat{\rho}}_{S}}\right]$
$\displaystyle-\frac{\gamma}{\omega_{0}}\left(g+\frac{1}{2}\right)\left[{\hat{X}},{\left[{\hat{X}},{{\hat{\rho}}_{S}}\right]}\right]+\frac{\gamma}{2\Omega}\left(1-g^{2}\right)\left[{\hat{X}},{\left[{\hat{P}},{{\hat{\rho}}_{S}}\right]}\right]-\frac{\gamma}{\omega_{0}}g\left(1+\frac{g}{2}\right)\left[{\hat{P}},{\left[{\hat{P}},{{\hat{\rho}}_{S}}\right]}\right]$
$\displaystyle+\frac{\mathrm{i}\gamma
g}{\omega_{0}}\sqrt{\frac{\beta\nu}{\omega_{0}}}\left[{\frac{\omega_{0}}{2\Omega}\hat{X}+g\hat{P}},{\left\\{{\sin{\left(\sqrt{\frac{\beta\omega_{0}}{\nu}}\hat{X}+2\pi\frac{\Phi_{x}}{\Phi_{0}}\right)}},{{\hat{\rho}}_{S}}\right\\}}\right]-\frac{\gamma
g}{\omega_{0}}\sqrt{\frac{\beta\nu}{\omega_{0}}}\left[{\hat{X}+\frac{g\omega_{0}}{2\Omega}\hat{P}},{\left[{\sin{\left(\sqrt{\frac{\beta\omega_{0}}{\nu}}\hat{X}+2\pi\frac{\Phi_{x}}{\Phi_{0}}\right)}},{{\hat{\rho}}_{S}}\right]}\right]$

In arriving at Eq. (On open quantum systems, effective Hamiltonians and device
characterization) two quadratic frequency renormalization terms (one
proportional to $[\hat{X}^{2},{\hat{\rho}}]$ and one proportional to
$[\hat{P}^{2},{\hat{\rho}}]$) have been removed, canceled by the Lamb Shift
term $\hat{\mathcal{H}}_{LS}$ Diggins  _et al._ (1994); Spiller  _et al._
(1991). Note also that the external flux control $\Phi_{x}$ appears in the
dissipator in this first order model (as opposed to second order for the case
of flux-only coupling Devitt  _et al._ (2013)). Eq. (On open quantum systems,
effective Hamiltonians and device characterization) is a non-rotating wave
(NRW) equation, and so requires additional terms to assume Lindblad form Munro
and Gardiner (1996)

$\frac{\mathrm{d}{\hat{\rho}}}{\mathrm{d}t}=-\mathrm{i}[\hat{\mathcal{H}}^{\prime},{\hat{\rho}}]+\frac{1}{2}\sum_{j}\bigg{\\{}[\hat{L}_{j},{\hat{\rho}}\hat{L}_{j}^{\dagger}]+[\hat{L}_{j}{\hat{\rho}},\hat{L}_{j}^{\dagger}]\bigg{\\}}$
(4)

where $\hat{L}_{j}$ are Lindblad operators. From Eq. (On open quantum systems,
effective Hamiltonians and device characterization) it is clear that these
will be linear combinations of $\hat{X}$, $\hat{P}$, and
$\hat{S}=\sin{\left(\sqrt{\frac{\beta\omega_{0}}{\nu}}\hat{X}+2\pi\frac{\Phi_{x}}{\Phi_{0}}\right)}$,
whose weights may be obtained by diagonalizing the coefficient matrix Breuer
and Petruccione (2007)

$a_{ij}=\frac{\gamma}{\omega_{0}}\begin{pmatrix}2g+1&-\mathrm{i}(1+g^{2}-g)-\xi(1-g^{2})&g\sqrt{\frac{\beta\nu}{\omega_{0}}}\left(1+\xi\right)\\\
\mathrm{i}(1+g^{2}-g)-\xi(1-g^{2})&2g+g^{2}&g^{2}\sqrt{\frac{\beta\nu}{\omega_{0}}}\left(\xi+\mathrm{i}\right)\\\
g\sqrt{\frac{\beta\nu}{\omega_{0}}}\left(1-\mathrm{i}\xi\right)&g^{2}\sqrt{\frac{\beta\nu}{\omega_{0}}}\left(\xi-\mathrm{i}\right)&0\end{pmatrix}$
(5)

for $i,j\in\\{X,P,S\\}$ and $\xi=\omega_{0}/2\Omega$. As is often found with
NRW BM equations, the coefficient matrix is not positive semidefinite, and so
does not conserve probability. To ensure physicality we alter $a_{SS}$ just
sufficient to make the $a$ matrix positive semi-definite, and its eigenvalues
non-negative. Effectively this is accomplished in a minimally invasive Breuer
and Petruccione (2007) manner by setting $\det(a)=0$, i.e.

$a_{SS}=\frac{\beta\nu}{\omega_{0}}\frac{-\xi^{2}g^{5}+4g^{4}+8\xi^{2}g^{3}}{(-g^{4}+2g^{2}-1)(1+\xi^{2})+4g(g^{2}+1)}$
(6)

together with the constraint that $0.227\leq g\leq 4.40$. This procedure
ensures the two non-zero eigenvalues are positive and, whilst the range of $g$
values may be extended, it does come at the cost of a more invasive change to
the matrix $a$. For our present purposes the details of the Lindblads are not
particularly important; though we note that the dominant one is close to an
annihilator, of $c_{1}\hat{X}+c_{2}\hat{P}$. Instead, here we are interested
in the question of what one might do with the terms that are leftover in the
process. As in the case of the QBM with a harmonic potential, all these terms
present in the form of a unitary evolution (i.e. as
$\mathrm{i}[\hat{A},\hat{\rho}]$), however there appears to be no clear means
of assessing whether they should be kept, to produce an effective Hamiltonian,
or canceled (countered) - either partially or completely. Retaining all such
terms for now, we obtain an effective Hamiltonian
$\hat{\mathcal{H}}^{\prime}=\hat{\mathcal{H}}_{S}+\hat{\mathcal{H}}_{XP}+\hat{\mathcal{H}}_{PS}+\hat{\mathcal{H}}_{XS}$
where

$\begin{split}\mathcal{\hat{H}}_{XP}&=\frac{\gamma}{2\omega_{0}}\left(\frac{3g^{2}}{2}-g+\frac{1}{2}\right)(\hat{X}\hat{P}+\hat{P}\hat{X}),\\\
\mathcal{\hat{H}}_{XS}&=-\frac{\gamma
g}{2\Omega}\sqrt{\frac{\beta\nu}{\omega_{0}}}\hat{X}\sin{\left(\sqrt{\frac{\beta\omega_{0}}{\nu}}\hat{X}+2\pi\frac{\Phi_{x}}{\Phi_{0}}\right)},\\\
\mathcal{\hat{H}}_{PS}&=-\frac{\gamma
g^{2}}{2\omega_{0}}\sqrt{\frac{\beta\nu}{\omega_{0}}}\left\\{{\hat{P}},{\sin{\left(\sqrt{\frac{\beta\omega_{0}}{\nu}}\hat{X}2\pi\frac{\Phi_{x}}{\Phi_{0}}\right)}}\right\\}\\\
&-\frac{\gamma
g^{2}\beta}{4\Omega}\cos{\left(\sqrt{\frac{\beta\omega_{0}}{\nu}}\hat{X}+2\pi\frac{\Phi_{x}}{\Phi_{0}}\right)}.\end{split}$
(7)

The impact of $\hat{\mathcal{H}}_{XP}$ has previously been discussed in terms
of frequency shifts and squeezing Devitt  _et al._ (2013); Duffus  _et al._
(2016) and, as previously noted, in other contexts too Gao (1997); Lindblad
(1976); Sǎndulescu and Scutaru (1987); Gao (1997); Wiseman and Munro (1998);
Lampo  _et al._ (2016); Schack  _et al._ (1995); Everitt (2009); Everitt  _et
al._ (2009), whilst a term similar in nature to $\hat{\mathcal{H}}_{XS}$
appears in the higher order (in $\omega_{0}/\Omega$) master equation for a
purely inductively coupled SQUID Devitt  _et al._ (2013). The final term
$\hat{\mathcal{H}}_{PS}$ includes a slight renormalization of the Josephson
junction energy.

Figure 1: The energy eigenvalues of $\hat{\mathcal{H}}^{\prime}$ as a function
of external flux $\Phi_{x}$ for various coupling ratios $g$. Values obtained
with Josephson energy $\hbar\nu=6.693\times 10^{-22}$J,capacitance
$C_{0}=5\times 10^{-15}$F, inductance $L_{0}=3\times 10^{-10}$H, damping rate
$\gamma=0.05\omega_{0}$ and bath frequency $\Omega=10*\omega_{0}$.

Fig. 1 demonstrates the impact of these terms on the lowest five energy
eigenvalues of $\hat{\mathcal{H}}^{\prime}$ as a function of the external flux
$\Phi_{x}$, and for a variety of $g$ values.

$\hat{\mathcal{H}}^{\prime}=\hat{\mathcal{H}}_{S}+\hat{\mathcal{H}}_{XP}+\hat{\mathcal{H}}_{PS}+\hat{\mathcal{H}}_{XS}$$\hat{\mathcal{H}}_{0}$$\hat{\mathcal{H}}_{0}+\hat{\mathcal{H}}_{XS}$$\hat{\mathcal{H}}_{0}+\hat{\mathcal{H}}_{XP}$$\hat{\mathcal{H}}_{0}+\hat{\mathcal{H}}_{PS}$$\hat{\mathcal{H}}_{0}+\hat{\mathcal{H}}_{XP}+\hat{\mathcal{H}}_{XS}$$\hat{\mathcal{H}}_{0}+\hat{\mathcal{H}}_{XP}+\hat{\mathcal{H}}_{PS}$$\hat{\mathcal{H}}_{0}+\hat{\mathcal{H}}_{PS}+\hat{\mathcal{H}}_{XS}$

Figure 2: Spiderweb Diagram. The solid line shows the lowest eigenvalue for
the SQUID Hamiltonian plus combinations of additional terms as labeled. Dashed
and dotted lines show, for comparison, the lowest eigenvalues of
$\hat{\mathcal{H}}_{0}$ and $\hat{\mathcal{H}}^{\prime}$ respectively. Note
that the contributions of $\hat{\mathcal{H}}_{XP}$,$\hat{\mathcal{H}}_{XS}$
and $\hat{\mathcal{H}}_{PS}$ are not independent, and the combination
$\hat{\mathcal{H}}_{XP}+\hat{\mathcal{H}}_{XS}$ accounts for the majority of
the alteration. The origin is $E=0$, the outer radius is
$E=6.0\hbar\omega_{0}$. Here $\Phi_{x}=0.5\Phi_{0}$, $g=1.8$.

Fig. 2 shows the contribution of
$\hat{\mathcal{H}}_{XP},\hat{\mathcal{H}}_{PS}$ and $\hat{\mathcal{H}}_{XP}$
to the lowest energy level, both individually and in pairs, and illustrates
how the cancellation, or not, of such terms will alter the system energy
eigenstates. It is clear that these effects are non-negligible, and that any
form of ad hoc process to deal with their contribution is likely to lead to
erroneous system models. That one of the terms is known to be important in the
quantum to classical transition Schack  _et al._ (1995); Everitt (2009);
Everitt  _et al._ (2010, 2011, 2009) implies that there are physically
important implications here which probably should not be ignored. The magnetic
susceptibility for a SQUID ring

$\chi_{0}(\Phi_{x};g)=-L\frac{\partial^{2}E_{0}(\Phi_{x};g)}{\partial\Phi_{x}^{2}},$
(8)

in its ground state, provides a useful mechanism to probe the lowest energy
eigenvalue’s dependence on the external flux $\Phi_{x}$ and coupling ratio $g$
Prance  _et al._ (1985). The ground state magnetic susceptibility (in the form
$\chi_{0}(\Phi_{x};g)/L$) is shown in Fig. 3 where it is seen to vary
significantly with both $\Phi_{x}$ and $g$. An important conclusion is that,
by arranging an adjustable capacitive coupling, and varying the external flux,
a measurement of $\chi_{0}$ provides an empirical test of the presence of
these terms; and hence whether they are indeed physical and need to be kept,
or merely an artifact of the process needed to shoehorn Born-Markov master
equations into Lindblad form, and need to be canceled.

Figure 3: Ground state magnetic susceptibility as a function of the external
flux $\Phi_{x}$, and capacitive coupling strengths $g$.

Making new Quantum Technologies scalable will rely on the ability to perform
careful modeling and simulation. Any simulation framework will require high
fidelity models and accurate characterization methods to allow precision
control and enable the necessary error correction. However it seems clear
that, beyond the simple case of a harmonic potential, deriving Lindblad master
equations from the Born-Markov approximation becomes rather ad hoc, as the
decision to include (as being physically real) or to exclude (through the use
of counter terms) is not supported well enough by our knowledge of the systems
involved. Even in that simplest case (QBM) there is still some disagreement
about whether a squeezing term of the form $\hat{X}\hat{P}+\hat{P}\hat{X}$
should be kept or canceled Sǎndulescu and Scutaru (1987); Gao (1997); Wiseman
and Munro (1998); Lampo  _et al._ (2016). The SQUID system considered here has
simple, adjustable control parameters: the external flux $\Phi_{x}$ and the
relative strength $g$ of capacitive coupling. Together they appear to provide
a sufficient means of interrogating the energy level structure through the
magnetic susceptibility. In the model considered, the additional terms will
make a measurable difference. Experimentally, the ability to create
superconducting devices coupled to an artificially constructed bath of
harmonic oscillators is well within the state-of-the-art. Hence, it will be
possible to experimentally verify whether, for example the squeezing term
$\hat{X}\hat{P}+\hat{P}\hat{X}$ should be canceled by a counter term.

## References

  * Yoshikawa  _et al._ (2005) N. Yoshikawa, T. Tomida, M. Tokuda, Q. Liu, X. Meng, S. R. Whiteley, and T. V. Duzer, IEEE Transactions on Applied Superconductivity 15, 267 (2005).
  * Pouiklis  _et al._ (2013) G. Pouiklis, G. Kottaras, A. Psomoulis, and E. Sarris, International Journal of Electronics 100, 913 (2013).
  * Houck  _et al._ (2008) A. A. Houck, J. A. Schreier, B. R. Johnson, J. M. Chow, J. Koch, J. M. Gambetta, D. I. Schuster, L. Frunzio, M. H. Devoret, S. M. Girvin, and R. J. Schoelkopf, Phys. Rev. Lett. 101, 080502 (2008).
  * Jeffrey  _et al._ (2014) E. Jeffrey, D. Sank, J. Y. Mutus, T. C. White, J. Kelly, R. Barends, Y. Chen, Z. Chen, B. Chiaro, A. Dunsworth, A. Megrant, P. J. J. O’Malley, C. Neill, P. Roushan, A. Vainsencher, J. Wenner, A. N. Cleland, and J. M. Martinis, Phys. Rev. Lett. 112, 190504 (2014).
  * Plantenberg  _et al._ (2007) J. Plantenberg, P. de Groot, C. Harmans, and J. Mooij, Nature 447, 836 (2007).
  * Steffen  _et al._ (2009) M. Steffen, F. Brito, D. DiVincenzo, S. Kumar, and M. Ketchen, New Journal of Physics 11, 033030 (2009).
  * Morshed  _et al._ (2011) T. H. Morshed _et al._ , BSIM4v4.7 MOSFET Model-User’s Manual, University of California, Berkley (2011).
  * Arrigoni  _et al._ (2013) E. Arrigoni, M. Knap, and W. von der Linden, Phys. Rev. Lett. 110, 086403 (2013).
  * Breuer and Petruccione (2007) H. Breuer and F. Petruccione,  _The Theory of Open Quantum Systems_ (OUP Oxford, 2007).
  * Dyre (1987) J. C. Dyre, Phys. Rev. Lett. 58, 792 (1987).
  * Giovannetti and Palma (2012) V. Giovannetti and G. M. Palma, Phys. Rev. Lett. 108, 040401 (2012).
  * Wu  _et al._ (2009) L.-A. Wu, G. Kurizki, and P. Brumer, Phys. Rev. Lett. 102, 080405 (2009).
  * Ferialdi (2016) L. Ferialdi, Phys. Rev. Lett. 116, 120402 (2016).
  * Hall  _et al._ (2014) M. J. W. Hall, J. D. Cresser, L. Li, and E. Andersson, Phys. Rev. A 89, 042120 (2014).
  * Maldonado-Mundo  _et al._ (2012) D. Maldonado-Mundo, P. Öhberg, B. W. Lovett, and E. Andersson, Phys. Rev. A 86, 042107 (2012).
  * Joshi  _et al._ (2014) C. Joshi, P. Öhberg, J. D. Cresser, and E. Andersson, Phys. Rev. A 90, 063815 (2014).
  * Vacchini (2016) B. Vacchini, Phys. Rev. Lett. 117, 230401 (2016).
  * Diósi and Ferialdi (2014) L. Diósi and L. Ferialdi, Phys. Rev. Lett. 113, 200403 (2014).
  * Lindblad (1976) G. Lindblad, Reports on Mathematical Physics 10, 393 (1976).
  * Caldeira and Leggett (1983) A. Caldeira and A. Leggett, Annals of Physics 149, 374 (1983).
  * Gao (1997) S. Gao, Phys. Rev. Lett. 79, 3101 (1997).
  * Sǎndulescu and Scutaru (1987) A. Sǎndulescu and H. Scutaru, Annals of Physics 173, 277 (1987).
  * Wiseman and Munro (1998) H. M. Wiseman and W. J. Munro, Phys. Rev. Lett. 80, 5702 (1998).
  * Lampo  _et al._ (2016) A. Lampo, S. H. Lim, J. Wehr, P. Massignan, and M. Lewenstein, Phys. Rev. A 94, 042123 (2016).
  * Schack  _et al._ (1995) R. Schack, T. A. Brun, and I. C. Percival, Journal of Physics A: Mathematical and General 28, 5401 (1995).
  * Everitt (2009) M. J. Everitt, New Journal of Physics 11, 013014 (2009).
  * Everitt  _et al._ (2010) M. Everitt, W. Munro, and T. Spiller, Physics Letters A 374, 2809 (2010).
  * Everitt  _et al._ (2011) M. J. Everitt, W. J. Munro, and T. P. Spiller, Journal of Physics: Conference Series 306, 012045 (2011).
  * Everitt  _et al._ (2009) M. J. Everitt, W. J. Munro, and T. P. Spiller, Phys. Rev. A 79, 032328 (2009).
  * Duffus  _et al._ (2016) S. N. A. Duffus, K. N. Bjergstrom, V. M. Dwyer, J. H. Samson, T. P. Spiller, A. M. Zagoskin, W. J. Munro, K. Nemoto, and M. J. Everitt, Phys. Rev. B 94, 064518 (2016).
  * Cole  _et al._ (2005) J. H. Cole, S. G. Schirmer, A. D. Greentree, C. J. Wellard, D. K. L. Oi, and L. C. L. Hollenberg, Phys. Rev. A 71, 062312 (2005).
  * Devitt  _et al._ (2013) S. J. Devitt, W. J. Munro, and K. Nemoto, Reports on Progress in Physics 76, 076001 (2013).
  * Diggins  _et al._ (1994) J. Diggins, J. F. Ralph, T. P. Spiller, T. D. Clark, H. Prance, and R. J. Prance, Phys. Rev. E 49, 1854 (1994).
  * Spiller  _et al._ (1991) T. P. Spiller, D. A. Poulton, T. D. Clark, R. J. Prance, and H. Prance, International Journal of Modern Physics B 5, 1437 (1991), arXiv:arXiv:1011.1669v3 .
  * Munro and Gardiner (1996) W. Munro and C. Gardiner, Physical Review A 53, 2633 (1996).
  * Prance  _et al._ (1985) R. J. Prance, J. E. Mutton, E. P. Shephard, T. D. Clark, H. Prance, and T. P. Spiller,  _SQUID ’85 Superconducting Quantum Interference Devices and their Applications_ , edited by H. D. Hahlbohm and H. Lubbig (1985).