# Third-Party Language Model Performance Prediction from Instruction

Rahul Nadkarni∗ Yizhong Wang∗† Noah A. Smith∗†
∗Paul G. Allen School of Computer Science & Engineering, University of
Washington
†Allen Institute for Artificial Intelligence
<EMAIL_ADDRESS>
###### Abstract

Language model-based instruction-following systems have lately shown
increasing performance on many benchmark tasks, demonstrating the capability
of adapting to a broad variety of instructions. However, such systems are
often not designed to be transparent about their limitations; a user may
easily prompt a model with an instruction without any idea of whether the
responses should be expected to be accurate, or if the system is even capable
of performing the task. We propose a third party performance prediction
framework, where a separate model is trained to predict the metric resulting
from evaluating an instruction-following system on a task while assuming
access only to its inputs and outputs at inference time. We perform this
analysis with a variety of both open and closed instruction-following models
as well as multiple performance predictors, and examine the effect of various
factors such as model size, number of training tasks, and prompt format. Our
findings indicate that third-party performance prediction is very challenging,
and much work remains in developing predictors that can automatically reveal
the limitations of modern instruction-following natural language processing
systems. Our code can be found at https://github.com/rahuln/instr-perf-pred.

##  1 Introduction

Despite much-discussed advances in the capabilities of language model-based
systems that follow instructions (Mishra et al., 2022; Sanh et al., 2022; Wei
et al., 2022; Ouyang et al., 2022; OpenAI, 2022, 2023), the research community
lacks an understanding of the limits of these capabilities. Ideally, purveyors
of a technological product would clearly explain to users the limitations of
what the system can be used for.111While documentation such as that of OpenAI
(2023) breaks down performance by factors such as task categories, languages,
or benchmarks (each benchmark being a collection of many tasks), there is
little transparency at any finer granularity (e.g., instruction-level) and no
publicly-available tool or mechanism to give users a sense of performance for
tasks that do not fit neatly into the documented categories. At present, the
best a user can do is explore: try out a prompt and see whether the language
model can correctly complete the task. We find this state of affairs
concerning, because the cost of such tests will fall on the users. Without
coordination and information-sharing, different users will make the same
explorations and incur unnecessary costs while simultaneously running the risk
of relying on systems for tasks which they are incapable of performing
adequately.222A second issue, not addressed here, is that users may not
realize that they need to check system output for correctness, and may simply
assume that any confident answer from a model can be trusted. We suspect that
this problem will worsen as users explore more and more use cases not
anticipated by the builders of the systems and therefore unaddressed by so-
called refusal training.

In this work, we take a step toward empowering users of language-model based
systems by proposing a third party approach to predicting model performance at
the task level. Consider a user with a particular task in mind. Our proposed
task performance predictor takes as input the same prompt the user intends for
the language model, and _without querying the language model itself_ , offers
an estimate of the model’s performance on the task. We instantiate such models
by regressing quantitative model performance metrics on natural language task
instructions. If successful, such predictors could help users decide among
commercial systems, or even opt out of delegating a task to a language model
at all.

Our experiments examine how well existing instruction-tuned LMs’ performance
can be predicted as a function of model size, choice of evaluation metric,
amount of training data, and other factors. We find overall that the task is
challenging, with the various factors we explore providing little improvement
to predictability. Our results underscore how much progress still needs to be
made in designing instruction-following natural language systems whose
performance can be accurately predicted and made known for the sake of
transparency and user safety.

##  2 Related Work

###  2.1 Instruction Tuning

Our work focuses on analyzing the behavior of models that have been trained to
follow task instructions. This includes models trained on human-generated
instructions and instances (Mishra et al., 2022; Sanh et al., 2022; Wei et
al., 2022; Ouyang et al., 2022; Wang et al., 2022) as well as model-generated
data (Wang et al., 2023b; Honovich et al., 2023; Taori et al., 2023; Chiang et
al., 2023). We primarily use models trained by Wang et al. (2023a) on a
variety of instruction-following datasets and initialized with the publicly
available LLaMA family of language models (Touvron et al., 2023). We also
explore using the closed models GPT-3.5 (OpenAI, 2022) and GPT-4 (OpenAI,
2023). For more details on our choice of instruction-tuned models, please
refer to §3.1.

###  2.2 Predicting Model Behavior

Initial work on performance prediction involved training simpler models to
predict the performance of larger models as a function of various features,
such as model family, model size, task, language, and training procedure (Xia
et al., 2020; Ye et al., 2021, 2023). The primary motivation of that work was
to address computational and data constraints – training separate performance
prediction models could alleviate issues where the computational cost of
finetuning on all datasets was prohibitive or where scarce data in a
particular language or domain prevented finetuning altogether. While much of
that work predicted performance at the task level as in our study, its
motivations were different and its methods were often implemented by
predicting from hand-crafted features related to properties of each dataset
(e.g., model parameter count or language features) rather than the text prompt
input to the model itself.

More recent efforts to better understand model behavior have analyzed the
ability to predict whether or not a model will perform well on a given input,
either as determined by the language model itself or a separately-trained
model. Kadavath et al. (2022) analyze models to determine whether they can
identify examples for which they can generate the correct response, either by
prompting for a “True/False” label with an instance and one or more
generations from the model itself or by finetuning the model as a binary
classifier using a dataset of inputs and correctness labels from previous
model outputs. Yao and Koller (2023) similarly finetune models as binary
classifiers to predict whether another model’s generated response is correct
for tagging, parsing, and semantic parsing tasks. Other work explores models’
ability to generate calibrated uncertainty about their responses either
through logits, multiple generations, or verbalized expressions of
uncertainty, applied to domains such as solving arithmetic problems (Lin et
al., 2022) or question answering (QA; Si et al., 2023; Zhou et al., 2023;
Cheng et al., 2023). Models’ ability to verbalize their uncertainty has also
been explored for models trained with reinforcement learning with human
feedback (RLHF), again using QA datasets (Tian et al., 2023). Notably, all of
these efforts examine being able to predict model performance at the instance
level, whereas we are primarily interested in predicting performance at the
task level given a task instruction.

The most similar work to ours in spirit is that of Fu et al. (2023), which
also attempts to predict how well a model will perform on a given dataset by
training a separate predictor. There are a number of key differences from our
efforts: their work focuses on the in-context learning setting using purely
pretrained models rather than models specifically trained to follow
instructions; they assume access to a set of unlabeled examples for each
dataset, rather than just a task instruction; their analysis requires access
to model internals by using model output logits to construct a “confidence
profile” feature vector for each dataset; and they restrict their analysis to
a small set of question answering datasets. A modern trend has seen public
releases of models whose internal states are inaccessible, but that can be
prompted with a wide variety of user-defined instructions. This situation
challenges many of the assumptions made by Fu et al. (2023) and motivates our
own work. However, their explorations inspire our efforts, and we attempt to
extend their analysis while focusing on task instructions and models trained
to follow them.

##  3 Methods

Figure 1: A diagram illustrating our complete analysis pipeline. We begin with
a pretrained LM that is instruction-tuned using training tasks from chosen
instruction data, resulting in an instruction-tuned model (IM). The IM is
evaluated using the test tasks of the instruction data (not necessarily from
the same dataset as the training tasks) and a choice of evaluation metric.
Each pair of test task instruction ($\boldsymbol{x}$) and evaluation
performance metric value ($y$) is used to construct the performance data,
which itself is split into train, validation, and test sets. The train and
validation sets are used to train _another_ (“third party”) pretrained LM to
predict the performance of the IM as a regression model, resulting in the
performance predictor (PP). Finally, the PP is evaluated on the test set of
the performance data to determine how well it can predict the performance of
the IM on unseen tasks. The sections of the diagram highlighted in  blue
indicate the components of the pipeline that we vary to determine their effect
on performance prediction: the size of the IM, the choice of instruction data,
the choice of evaluation metric, and the size and type of PP model.

We begin by describing our complete analysis pipeline, also illustrated as a
diagram in Figure 1. Each of our experiments involves two finetuned language
models: one that is trained to follow instructions to perform tasks, which we
term the _instruction-tuned model_ (IM), and another that is trained to map
from an instruction to some measure of the IM’s performance on that task,
which we call the _performance predictor_ (PP). The IMs considered here are
drawn from past work. To tune a PP, we evaluate a single IM on instructions it
was not exposed to during any of its training (including instruction-tuning).
The resulting pairs – each an instruction $\boldsymbol{x}$ paired with the
model’s performance score $y$ – are divided into training, validation, and
test sets for the PP. We train the PP to predict the IM’s performance $y$ on
instructions $\boldsymbol{x}$ unseen to the IM, and evaluate those predictions
on a different set of instructions unseen to either the IM or the PP. We
explore a variety of choices for IMs, PPs, and evaluation metrics, which are
detailed below.

###  3.1 Instruction-Tuned Models (IMs)

Our experiments use a range of pretrained LMs that have been finetuned to
follow instructions. Part of our goal is to assess the effect of IM size
(i.e., parameter count) and choice of instruction-tuning dataset on how well
the model’s performance can be predicted. To this end, we primarily use LLaMA
models (Touvron et al., 2023) of various sizes finetuned on a range of
instruction-tuning datasets (Wang et al., 2023a). The LLaMA family of models
is among the best performing open models, and they have already been used as
the basis for a variety of models trained to follow instructions (Taori et
al., 2023; Chiang et al., 2023; Wang et al., 2023a). All models were trained
by maximizing the per-token likelihood of the gold output given an instruction
and possibly an input, with no additional training procedures such as RLHF
(Christiano et al., 2017). While adapting our analysis to models trained using
RLHF would be an interesting direction of future work, such models and the
data to train them are currently limited relative to models trained with
supervised finetuning alone, and our intention is to perform an initial
analysis on the least-complicated systems that still demonstrate an ability to
follow instructions. For the sake of completeness, we additionally include the
closed, API-based models GPT-3.5 (`gpt-3.5-turbo`) and GPT-4, as our goal is
to implement a pipeline that is applicable even when access is restricted to
model inputs and outputs.

###  3.2 Evaluation Metrics

For each instruction-tuned model, we perform inference on a separate
evaluation set of instructions in order to generate a dataset of the model’s
behavior ($y$) on unseen instructions ($\boldsymbol{x}$) for training
performance prediction models. For each instruction and output pair, we
calculate a quantitative evaluation metric that compares the model-generated
output to the gold output, in most cases averaging this metric across
instances for tasks that have multiple instances. We treat this final metric
as the instruction-tuned model’s performance for that instruction. We explore
two quantitative metrics commonly used when evaluating instruction-tuned
models on a broad range of tasks: ROUGE-L (Lin, 2004) and Exact Match score.
Again, since these automated metrics can be computed by comparing a model-
generated output to a gold output, we can apply them to models where we only
have access to model generations. Additionally, we briefly compare to using
model loss as the metric to predict in §4.3, as an exploration of what we can
achieve with performance prediction when additional information (in this case,
the LM’s output distribution) is available.

###  3.3 Performance Predictors (PP)

Once we’ve evaluated an IM on unseen instructions, we use the performance data
($\langle\boldsymbol{x},y\rangle$ pairs) to build models that predict the IM’s
performance, and evaluate those predictions on the test subset of the IM
performance data. We primarily finetune RoBERTa (both base and large sizes;
Liu et al., 2019) as the PP, motivated by the goal of having a lightweight
separate model that can predict how well an instruction-tuned model will
perform without incurring the inference costs of a much larger model. RoBERTa
models are trained as regression models by adding a linear layer to the
`[CLS]` token at the output layer and training to minimize mean-squared error
between the predicted and true evaluation metric for each instruction. In some
cases, we also train the base LLaMA model used to build the IM by similarly
adding a linear layer to the EOS token at the output layer and adding and
updating LoRA adapters (Hu et al., 2022) rather than updating all model
weights. This is done in an effort to establish an “upper bound” of
performance prediction that can be achieved while incurring the prohibitive
computational cost of using a much larger model. We additionally include a
simple baseline of predicting the mean metric value across all training
instructions, as a “lower bound” to establish whether training a separate
predictor model offers any benefit at all.

Unless otherwise specified, PPs are trained and evaluated on the results of
evaluating IMs on the Super-NaturalInstructions (SuperNI) test set tasks Wang
et al. (2022). For each experiment, we perform 10 random 80%/10%/10%
train/validation/test splits of these tasks and report mean and standard
deviation of the performance predictor’s root mean squared error (RMSE) in
predicting the true evaluation metric from the task instruction. Validation
data is used to tune hyperparameters for the PPs, namely batch size and
learning rate; for full experiment details, please refer to Appendix A.

##  4 Results

###  4.1 Performance Prediction is Challenging

IM: | Alpaca-13B | Vicuna-13B | Tülu-13B | GPT-3.5 | GPT-4
---|---|---|---|---|---
avg. performance (ROUGE-L): | 45.9 | 44.6 | 61.7 | 53.6 | 63.5
Exact Match | mean | $25.7_{3.4}$ | $26.4_{3.2}$ | $32.4_{2.3}$ | $30.8_{3.9}$ | $36.4_{3.8}$
| RoBERTa-base | $26.2_{4.3}$ | $26.1_{4.9}$ | $33.6_{3.3}$ | $32.5_{5.4}$ | $38.2_{4.1}$
| RoBERTa-large | $27.4_{5.4}$ | $26.9_{5.4}$ | $34.4_{4.9}$ | $33.0_{5.7}$ | $37.2_{4.5}$
| LLaMA-13B | $19.1_{5.3}$ | $18.9_{5.8}$ | $21.4_{3.8}$ | $21.7_{5.6}$ | $22.4_{8.7}$
ROUGE-L | mean | $21.3_{3.5}$ | $20.4_{3.3}$ | $21.4_{1.8}$ | $22.4_{2.9}$ | $24.2_{4.2}$
| RoBERTa-base | $22.5_{4.1}$ | $21.2_{3.6}$ | $22.1_{3.8}$ | $23.4_{3.4}$ | $25.6_{4.0}$
| RoBERTa-large | $22.6_{4.2}$ | $21.2_{3.6}$ | $22.0_{3.7}$ | $23.5_{3.4}$ | $25.7_{3.9}$
| LLaMA-13B | $22.2_{4.7}$ | $21.2_{3.8}$ | $21.5_{3.4}$ | $21.6_{3.5}$ | $22.1_{5.1}$
Table 1: Test set RMSE of mean baseline and various PP models finetuned to
predict performance from task instruction for various IMs (columns). Subscript
shows standard deviation across 10 splits of the SuperNI test tasks.  Figure
2: Predicted vs. true metric value when using RoBERTa-large to map from task
instruction to performance – either ROUGE-L (top row) or Exact Match (bottom
row) – for various instruction-following models (columns).

Table 1 shows our main results of predicting performance on SuperNI test set
instructions, with mean and standard deviation (subscript) RMSE values across
all train-test splits of the SuperNI test tasks. We explore using both Exact
Match and ROUGE-L as the target metrics to be predicted for each task, with
base and large RoBERTa models as the PP models as well as LLaMA-13B (upper
bound) and the simple mean baseline (lower bound). For the IMs, we use the
13B-parameter versions of the Alpaca, Vicuna333We use the 13B-parameter
version of LLaMA finetuned on the ShareGPT dataset by Wang et al. (2023a) as
an attempt to replicate the training of Vicuna., and Tülu models, as well as
GPT-3.5 (`gpt-3.5-turbo`) and GPT-4.

Results demonstrate that performance prediction from task instruction is
incredibly difficult, with RMSE values generally around 20 or higher (for
metrics in the 0–100 range) across all experimental conditions. Both RoBERTa-
base and RoBERTa-large perform comparably to the simple mean baseline, likely
indicating that there is little learnable signal within the set of
instruction-metric pairs. The lack of a meaningful difference in performance
between RoBERTa-base and RoBERTa-large suggests that size of the PP model also
makes little difference in performance prediction at the RoBERTa scale. In
general, ROUGE-L is more predictable than Exact Match, with consistently lower
RMSE values across choices of PPs and IMs. The values below the name of each
model indicating performance in ROUGE-L on the SuperNI test tasks show that
models which tend to perform better on SuperNI are _less_ predictable,
exhibiting the worst RMSE values in general.

The results for LLaMA-13B as PP are more promising, with RMSE values that
outperform the RoBERTa models as well as the mean baseline when predicting the
Exact Match score across all IMs. However, results are still poor, with RMSE
values near 20 for both evaluation metrics and all IMs, so even a much larger
PP with a more modern base model is unable to predict performance values that
are near ground truth. It is additionally worth noting that finetuning
LLaMA-13B is a much more prohibitive method of performance prediction, as it
involves using a model with around 36$\times$ the number of parameters of
RoBERTa-large and incurring comparable inference costs to many of the IMs
themselves. As such, we treat the LLaMA-13B performance prediction results as
demonstration of a hopeful upper bound while simultaneously underscoring the
large room for improvement in developing better-performing and lightweight PP
models.

Figure 2 shows a more detailed view of a subset of the same results, with
scatter plots of the predicted versus true metric value (ROUGE-L or Exact
Match) when using RoBERTa-large as the PP for both metrics and all IMs.
Results are shown for all train-test splits of the SuperNI test tasks with 12
held-out instructions per random split (10% of the SuperNI test tasks),
resulting in a total of 120 predicted and true performance values for each
combination of evaluation metric and IM. This qualitative view of the
predictions highlights the fact that RoBERTa-large generally learns to predict
roughly the same mean performance value across all tasks within each train-
test split of the SuperNI test tasks. These results explain the similarity in
performance to the simple mean baseline and demonstrate that the PP model does
not learn a meaningful association between instruction and performance.

###  4.2 Effect of Various Factors on Predictability

We perform more detailed analyses by altering various factors that may affect
the behavior of a PP when used to predict the performance of an IM. These
factors include the size of the IM, the number of tasks used to train the PP,
and the choice of prompt (instruction-only or instruction + 2 positive
demonstrations).

####  4.2.1 Size of Instruction-tuned Model

PP: | mean | RoBERTa-large
---|---|---
LLaMA-7B (35.8) | $21.3_{3.4}$ | $20.7_{4.6}$
LLaMA-13B (44.6) | $26.4_{3.2}$ | $27.0_{5.4}$
LLaMA-30B (44.2) | $27.5_{2.8}$ | $27.9_{5.3}$
LLaMA-65B (48.9) | $30.0_{3.3}$ | $31.7_{5.3}$

Table 2: Test set RMSE of predicting Exact Match given task instruction, for
various sizes of LLaMA models instruction-tuned on ShareGPT. Model performance
in ROUGE-L on the SuperNI test set is given in parentheses next to the name of
each model.

We examine the effect of IM size by predicting the performance of various-
sized LLaMA models instruction-tuned by Wang et al. (2023a) on the ShareGPT
dataset, ranging in scale from 7B to 65B parameters. Results are shown in
Table 2, focusing on the mean baseline and RoBERTa-large PP model as well as
the Exact Match metric (results for RoBERTa-base and ROUGE-L are similar). The
RMSE values indicate that performance prediction worsens for increasing model
size, while model performance on the SuperNI test set improves (ROUGE-L values
in parentheses next to each model name). However, the mean baseline also
exhibits increasing RMSE values with model scale. This likely suggests that
larger, better-performing models are less predictable not because they exhibit
more dissimilar behavior on tasks with similar instructions, but because their
performance metric values cover a broader range. In any case, we can conclude
that scale of the IM alone does not improve performance prediction.

####  4.2.2 Number of Training Tasks

PP training: | SuperNI | \+ BIG-bench
---|---|---
Alpaca-13B | $27.4_{5.4}$ | $25.4_{6.2}$
Vicuna-13B | $26.9_{5.4}$ | $26.1_{5.5}$
Tülu-13B | $34.4_{4.9}$ | $34.6_{3.9}$
GPT-3.5 | $33.0_{5.7}$ | $32.3_{6.7}$
GPT-4 | $37.2_{4.5}$ | $37.7_{4.5}$
Table 3: Test set RMSE of using RoBERTa-large to predict Exact Match given
task instruction, using either just SuperNI instructions or SuperNI and BIG-
bench instructions for training the PP model.

We examine the effect of increasing the number of tasks / instructions used to
train the PP model by including tasks from BIG-bench (BIG-bench authors,
2023). Since BIG-bench tasks do not come with instructions, we manually
annotate tasks either by converting the task description (if provided) into a
SuperNI-style declarative instruction or writing an instruction from scratch
when necessary. We additionally filter out a number of tasks, leaving a total
of 156 tasks to use as additional training tasks (a full list of included
tasks can be found in Appendix B). We perform the same splits of the SuperNI
test tasks as before, only adding the BIG-bench tasks to the training split in
each case to increase the number of instructions used to train the PP model in
an effort to improve its performance.

The results can be found in Table 3. While the inclusion of BIG-bench task
instructions represents a nearly 2.5$\times$ increase in the number of
training instructions for the PP model, the RMSE values demonstrate that the
performance difference is negligible. There are a number of plausible
explanations for why this is the case – in general, BIG-bench tasks are
substantially different from SuperNI tasks, including those which deviate from
standard language understanding such as decoding encrypted text, performing
arithmetic operations, and generating chess moves. The domain shift between
the two datasets may explain why the inclusion of the additional instructions
did not provide enough meaningful signal for the PP models.

####  4.2.3 Prompt Format

format: | instruction | \+ 2 demonstrations
---|---|---
Alpaca-13B | $27.4_{5.4}$ | $27.3_{3.7}$
Vicuna-13B | $26.9_{5.4}$ | $29.0_{2.6}$
Tülu-13B | $34.4_{4.9}$ | $33.6_{2.4}$
GPT-3.5 | $33.0_{5.7}$ | $31.6_{3.3}$
GPT-4 | $37.2_{4.5}$ | $37.8_{3.8}$

Table 4: Test set RMSE of using RoBERTa-large to predict Exact Match given
task instruction, using SuperNI tasks with an instruction-only prompt or the
instruction with 2 positive demonstrations.

We primarily focus on instruction-tuned models evaluated in a zero-shot manner
with only an instruction and instance input, as this has become standard
practice and reflects how a generic user might interact with an instruction-
following system. However, previous work exploring the effect of the prompt
format has shown notable improvements in IM performance when including
additional information, such as positive demonstrations of the task (Wang et
al., 2022). We explore whether this improvement in performance also leads to
an improvement in predictability by additionally evaluating all IMs with a
prompt format that uses a task instruction with two positive demonstrations,
using the positive examples provided with each SuperNI task. The RMSE values
resulting from training RoBERTa-large to predict the Exact Match score in this
setting, as well as the original instruction-only RMSE values, can be seen in
Table 4. While the inclusion of demonstrations improves the performance of the
IMs themselves by roughly 2–6 points in Exact Match score, the evaluation
results don’t lead to the models being more predictable as there are no
meaningful differences in values between prompt formats for any model.

###  4.3 Predicting Loss

Figure 3: Predicted vs. true loss value when using RoBERTa-large to map from
task instruction to loss for various instruction-following models.

PP: | mean | RoBERTa-large
---|---|---
Alpaca-13B | $1.70_{0.27}$ | $1.51_{0.43}$
Vicuna-13B | $1.97_{0.46}$ | $2.10_{0.53}$
Tülu-13B | $1.25_{0.32}$ | $1.10_{0.36}$

Table 5: Test set RMSE of predicting cross-entropy loss given task
instruction, for various LLaMA-based instruction-following models.

Our core analysis above relied on the use of automated evaluation metrics such
as Exact Match and ROUGE-L that can be formulated as functions applied to a
pair of generated and gold outputs. This is based on the assumption that we do
not have access to model internals or outputs other than generated text, a
practical assumption given the increasing use of modern systems that only
allow for limited access such as through API requests. However, while such
automated metrics can be shown to correlate with other variables such as
accuracy on classification tasks (Wang et al., 2022), they come with their own
limitations. For instance, for tasks that require more creative or free-form
generations from the model, it is possible for the model output and gold label
to be semantically equivalent and equally valid responses to the instruction
while differing in surface form such that metrics like Exact Match and ROUGE-L
are inappropriate (Holtzman et al., 2021). To address this, we perform an
additional experiment training PP models to predict the model’s _loss_ (on the
gold output) instead, avoiding token-based comparison between two pieces of
text entirely. We do this by evaluating the average per-token cross-entropy
loss of the gold label given a prompt (task instruction and input) for each
instance in each dataset, averaging across instances to get a single average
loss value per task / instruction. We then follow the same regression training
procedure to build PP models that predict this value.

Scatter plots similar to Figure 2 can be seen in Figure 3 (where this time the
true and predicted values are this average per-task loss), and the
quantitative RMSE values corresponding to these results are in Table 5. As
this analysis requires access to model output distributions to compute the
loss, we are limited to the LLaMA-based open models. The plots show the
predictions made by RoBERTa-large finetuned to predict average per-task loss,
using the same 10 splits of the SuperNI test tasks as in previous results.
Qualitatively, these results appear to show greater correlation between the
true and predicted values as compared to the metrics considered earlier,
indicating that there may be more of a learnable signal for predicting loss.
However, the mean RMSE values show that the finetuned RoBERTa-large model
still does not outperform the simple mean baseline on average, indicating that
performance prediction remains challenging even with access to a quantitative
metric not based on token-level comparison between generated and gold outputs.
Taken together, our results motivate further work in identifying quantitative
task-level metrics that can be accurately predicted, and perhaps in designing
IMs themselves to be more predictable.

##  5 Discussion

Our experiments varied a number of factors that could impact the
predictability of an instruction-tuned model’s performance on unseen
instructions. We broadly summarize our findings below:

Performance prediction remains incredibly challenging regardless of setup.
RMSE values remain at 20 or higher for metrics in the range of 0–100,
indicating that PP models fail to predict values that are even somewhat close
to true performance. This remains true across a variety of open and closed
instruction-following models, for multiple automated evaluation metrics. There
is also little performance difference between RoBERTa PP models of different
scales, and none of them outperform a simple mean baseline. One optimistic
result occurs when using LLaMA-13B as the PP model, but performance is still
relatively poor and this comes at the cost of scaling up the PP model to the
size of the IM.

Increasing instruction-tuned model scale, increasing number of training tasks,
and adding demonstrations to prompts all fail to improve performance
prediction. The behavior of larger models does not seem to be any more
predictable, nor is the behavior of models with access to additional
information in the prompt. The typical strategy of increasing the amount of
training data available for the finetuned PP model to better learn a
prediction signal is also insufficient. The number of instructions remains
small, so future work could ascertain whether the problem setup is still
limited by the amount of instructions or if there really is no learnable
pattern in IM behavior.

Predicting cross-entropy loss does not improve performance. Despite avoiding
the issues inherent in using metrics based on token-level comparison between
generated and gold outputs, training PP models to predict loss still does not
lead to better results than the simple mean baseline.

##  6 Conclusion

Recent NLP systems seem to be able to perform arbitrary tasks given an
instruction. Yet we are still not able to understand or explain to users the
limitations of these systems such as by reliably predicting their success or
failure on new, previously unseen instructions. We take a first step toward
this goal by training a separate predictor model to map from a task
instruction to the quantified performance of a given instruction-tuned model
on that task. Our results show that performance prediction is challenging,
with numerous factors like choice of evaluation metric, predictor model size,
instruction-following model size, number of training tasks, and prompt format
all showing negligible effect on the predictability of instruction-tuned model
behavior. Much work remains to be done in designing systems whose limitations
can be well-predicted and revealed transparently and freely to their users.

##  7 Limitations & Ethical Considerations

While we explore the third party performance prediction problem across a
variety of factors, there are several constraints that limit our analysis and
could be explored in future work. Likely the largest limitation is data – few
datasets exist in the SuperNI style (with multiple tasks each having a
declarative instruction and multiple instances for evaluation) that existing
models have not already been trained on. Even with the addition of BIG-bench
instructions, the resulting dataset is around 250 training instructions, which
is still small by most standards and may not provide enough data in general to
learn to predict performance. Building more datasets in this format, perhaps
by scaling up dataset generation in an automated fashion (Wang et al., 2023b;
Honovich et al., 2023), could expand this analysis to help overcome the data
limitation.

Additionally, using a quantitative, automatic evaluation metric may itself not
be appropriate when considering arbitrary tasks, including ones that are
creative or based on open-ended generation. Reliable quantitative evaluation
in the general instruction-tuned setting is an open challenge. While we
attempt to address this with our results predicting cross-entropy loss, the
choice of evaluation metric for arbitrary instruction-following tasks remains
an open question.

Instruction-tuned model performance can also depend on how the instruction is
phrased, and previous work has demonstrated model sensitivity to perturbed or
paraphrased instructions (Zhao et al., 2021; Webson and Pavlick, 2022). Driven
by assumptions that most users will not “engineer” instructions extensively,
our experiments only consider a single instruction per task, and redefining
performance based on multiple instructions per task (i.e., more “task-
specific” rather than “instruction-specific” behavior) may lead to other
interesting results.

## Acknowledgements

We would like to thank the members of Noah’s ARK research group for their
helpful feedback on this work. This research was supported in part by the
Office of Naval Research under MURI grant N00014-18-1-2670.

## References

  * BIG-bench authors (2023) BIG-bench authors. 2023.  Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research_.
  * Cheng et al. (2023) Silei Cheng, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. 2023.  Prompting GPT-3 To Be Reliable.  In _International Conference on Learning Representations_.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality.
  * Christiano et al. (2017) Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep Reinforcement Learning from Human Preferences.  In _Conference on Neural Information Processing Systems_.
  * Fu et al. (2023) Harvey Fu, Qinyuan Ye, Albert Xu, Xiang Ren, and Robin Jia. 2023.  Estimating Large Language Model Capabilities without Labeled Test Data.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 9530–9546.
  * Holtzman et al. (2021) Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021\.  Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7038–7051.
  * Honovich et al. (2023) Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023.  Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 14409–14428.
  * Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, T. J. Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zachary Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, John Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom B. Brown, Jack Clark, Nicholas Joseph, Benjamin Mann, Sam McCandlish, Christopher Olah, and Jared Kaplan. 2022.  Language Models (Mostly) Know What They Know.  _ArXiv_ , abs/2207.05221.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A Package for Automatic Evaluation of Summaries.  In _Association for Computational Linguistics_.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  Teaching Models to Express Their Uncertainty in Words.  _Transactions on Machine Learning Research_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _ArXiv_ , abs/1907.11692.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.  Cross-Task Generalization via Natural Language Crowdsourcing Instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * OpenAI (2022) OpenAI. 2022.  Introducing ChatGPT.  https://openai.com/blog/chatgpt.  Accessed: 2024-01-15.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Conference on Neural Information Processing Systems_.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022.  Multitask Prompted Training Enables Zero-Shot Task Generalization.  In _International Conference on Learning Representations_.
  * Si et al. (2023) Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang. 2023.  Prompting GPT-3 To Be Reliable.  In _International Conference on Learning Representations_.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford Alpaca: An Instruction-following LLaMA model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Tian et al. (2023) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023.  Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5433–5442.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  LLaMA: Open and Efficient Foundation Language Models.  _ArXiv_ , abs/2302.13971.
  * Wang et al. (2023a) Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023a.  How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources.  In _Conference on Neural Information Processing Systems_.
  * Wang et al. (2023b) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023b.  Self-Instruct: Aligning Language Models with Self-Generated Instructions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patel, Kuntal Kumar Pal, M. Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddharth Deepak Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hanna Hajishirzi, and Daniel Khashabi. 2022.  Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_.
  * Webson and Pavlick (2022) Albert Webson and Ellie Pavlick. 2022.  Do Prompt-Based Models Really Understand the Meaning of Their Prompts? In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022.  Finetuned Language Models are Zero-Shot Learners.  In _International Conference on Learning Representations_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_.
  * Xia et al. (2020) Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. 2020.  Predicting Performance for Natural Language Processing Tasks.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Yao and Koller (2023) Yuekun Yao and Alexander Koller. 2023.  Predicting generalization performance with correctness discriminators.  _ArXiv_ , abs/2311.09422.
  * Ye et al. (2023) Qinyuan Ye, Harvey Fu, Xiang Ren, and Robin Jia. 2023.  How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_.
  * Ye et al. (2021) Zihuiwen Ye, Pengfei Liu, Jinlan Fu, and Graham Neubig. 2021.  Towards More Fine-grained and Reliable NLP Performance Prediction.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate Before Use: Improving Few-shot Performance of Language Models.  In _Proceedings of the 38th International Conference on Machine Learning_.
  * Zhou et al. (2023) Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023.  Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5506–5524.

## Appendix

##  Appendix A Finetuning Details

We use the Huggingface `transformers` library to run all experiments (Wolf et
al., 2020). Details for finetuning RoBERTa-base and RoBERTa-large performance
predictors can be found in Table 6, while details for the LLaMA-13B
performance predictors can be found in Table 7. We perform a hyperparameter
search over batch size and learning rate for all finetuning experiments.
RoBERTa models are finetuned by updating all parameters, while LLaMA models
are finetuned by adding and updating LoRA adapters Hu et al. (2022). All
performance predictors use an additional linear layer applied to either the
`[CLS]` token (for RoBERTa) or the EOS token (for LLaMA) at the last layer to
make the final prediction. All models were evaluated on the validation set of
instructions at every epoch, with early stopping performed based on validation
set RMSE.

Hyperparameter | Assignment
---|---
number of epochs | 20
batch size | {4, 8, 16}
maximum learning rate | {1e-5, 5e-5, 1e-4, 5e-4}
optimizer | AdamW
epsilon | 1e-8
betas | (0.9, 0.999)
learning rate schedule | constant
weight decay | 0
warmup proportion | none
learning rate decay | none

Table 6: Experiment settings for finetuning RoBERTa performance predictor
models.

Hyperparameter | Assignment
---|---
number of epochs | 20
batch size | {8, 16, 32}
maximum learning rate | {1e-5, 2e-5, 5e-5, 1e-4}
optimizer | AdamW
epsilon | 1e-8
betas | (0.9, 0.999)
learning rate schedule | linear warmup
weight decay | 0
warmup proportion | 0.03
learning rate decay | linear
LoRA rank | 256
LoRA alpha | 256
LoRA dropout | 0.05

Table 7: Experiment settings for finetuning LLaMA performance predictor
models.

##  Appendix B Details on BIG-bench

The full list of tasks included from BIG-bench can be found in Table 8.

Tasks
---
abstract_narrative_understanding, anachronisms, analogical_similarity,
analytic_entailment, arithmetic, ascii_word_recognition,
authorship_verification, auto_categorization, auto_debugging, bbq_lite_json,
bridging_anaphora_resolution_barqa, causal_judgment, cause_and_effect,
checkmate_in_one, chess_state_tracking, chinese_remainder_theorem,
cifar10_classification, code_line_description, codenames, color,
common_morpheme, conceptual_combinations, conlang_translation,
contextual_parametric_knowledge_conflicts, crash_blossom, crass_ai,
cryobiology_spanish, cryptonite, cs_algorithms, dark_humor_detection,
date_understanding, disambiguation_qa, discourse_marker_prediction, disfl_qa,
dyck_languages, elementary_math_qa, emoji_movie, emojis_emotion_prediction,
empirical_judgments, english_proverbs, english_russian_proverbs,
entailed_polarity, entailed_polarity_hindi, epistemic_reasoning,
evaluating_information_essentiality, fact_checker, fantasy_reasoning,
few_shot_nlg, figure_of_speech_detection,
formal_fallacies_syllogisms_negation, gem, gender_inclusive_sentences_german,
general_knowledge, geometric_shapes, goal_step_wikihow,
gre_reading_comprehension, hhh_alignment, hindi_question_answering,
hindu_knowledge, hinglish_toxicity, human_organs_senses, hyperbaton,
identify_math_theorems, identify_odd_metaphor, implicatures,
implicit_relations, intent_recognition, international_phonetic_alphabet_nli,
international_phonetic_alphabet_transliterate, intersect_geometry,
irony_identification, kanji_ascii, kannada, key_value_maps, known_unknowns,
language_games, language_identification, linguistics_puzzles,
logic_grid_puzzle, logical_args, logical_fallacy_detection, logical_sequence,
mathematical_induction, matrixshapes, metaphor_boolean,
metaphor_understanding, minute_mysteries_qa, misconceptions, mnist_ascii,
modified_arithmetic, moral_permissibility, movie_dialog_same_or_different,
movie_recommendation, mult_data_wrangling, navigate, nonsense_words_grammar,
novel_concepts, object_counting, odd_one_out, operators,
paragraph_segmentation, parsinlu_qa, parsinlu_reading_comprehension,
penguins_in_a_table, periodic_elements, persian_idioms, phrase_relatedness,
physical_intuition, physics, physics_questions, play_dialog_same_or_different,
polish_sequence_labeling, presuppositions_as_nli, qa_wikidata,
question_selection, real_or_fake_text, reasoning_about_colored_objects,
repeat_copy_logic, rephrase, riddle_sense, ruin_names,
salient_translation_error_detection, scientific_press_release,
semantic_parsing_in_context_sparc, semantic_parsing_spider,
sentence_ambiguity, similarities_abstraction, simp_turing_concept,
simple_ethical_questions, simple_text_editing, snarks, social_iqa,
social_support, sports_understanding, strange_stories, strategyqa,
sufficient_information, suicide_risk, swahili_english_proverbs,
swedish_to_german_proverbs, symbol_interpretation, temporal_sequences, tense,
timedial, topical_chat, tracking_shuffled_objects, understanding_fables,
undo_permutation, unit_conversion, unit_interpretation,
vitaminc_fact_verification, what_is_the_tao, which_wiki_edit, winowhy,
word_sorting, word_unscrambling
Table 8: List of BIG-bench tasks included when training performance predictors
on additional tasks.


EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCH (CERN)

 CERN-PH-EP-2012-214 LHCb-PAPER-2012-014 September 20, 2012

Measurement of the $B^{0}_{s}\rightarrow J/\psi\kern
4.14793pt\overline{\kern-4.14793ptK}{}^{*0}$ branching fraction and angular
amplitudes

The LHCb collaboration†††Authors are listed on the following pages.

A sample of $114\pm 11$
$B^{0}_{s}\rightarrow{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}K^{-}\pi^{+}$
signal events obtained with $0.37$ fb-1 of $pp$ collisions at $\sqrt{s}$ = 7
TeV collected by the LHCb experiment is used to measure the branching fraction
and polarization amplitudes of the $B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0}$ decay, with $\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0}\rightarrow K^{-}\pi^{+}$. The
$K^{-}\pi^{+}$ mass spectrum of the candidates in the $B^{0}_{s}$ peak is
dominated by the $\kern 1.99997pt\overline{\kern-1.99997ptK}{}^{*0}$
contribution. Subtracting the non-resonant $K^{-}\pi^{+}$ component, the
branching fraction of $B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0}$ is $\left(4.4_{-0.4}^{+0.5}\pm
0.8\right)\times 10^{-5}$, where the first uncertainty is statistical and the
second is systematic. A fit to the angular distribution of the decay products
yields the $K^{*0}$ polarization fractions $f_{L}=0.50\pm 0.08\pm 0.02$ and
$f_{\parallel}=0.19^{+0.10}_{-0.08}\pm 0.02$.

Submitted to Physical Review D (R)

LHCb collaboration

R. Aaij38, C. Abellan Beteta33,n, A. Adametz11, B. Adeva34, M. Adinolfi43, C.
Adrover6, A. Affolder49, Z. Ajaltouni5, J. Albrecht35, F. Alessio35, M.
Alexander48, S. Ali38, G. Alkhazov27, P. Alvarez Cartelle34, A.A. Alves Jr22,
S. Amato2, Y. Amhis36, J. Anderson37, R.B. Appleby51, O. Aquines Gutierrez10,
F. Archilli18,35, A. Artamonov 32, M. Artuso53,35, E. Aslanides6, G.
Auriemma22,m, S. Bachmann11, J.J. Back45, V. Balagura28,35, W. Baldini16, R.J.
Barlow51, C. Barschel35, S. Barsuk7, W. Barter44, A. Bates48, C. Bauer10, Th.
Bauer38, A. Bay36, J. Beddow48, I. Bediaga1, S. Belogurov28, K. Belous32, I.
Belyaev28, E. Ben-Haim8, M. Benayoun8, G. Bencivenni18, S. Benson47, J.
Benton43, R. Bernet37, M.-O. Bettler17, M. van Beuzekom38, A. Bien11, S.
Bifani12, T. Bird51, A. Bizzeti17,h, P.M. Bjørnstad51, T. Blake35, F. Blanc36,
C. Blanks50, J. Blouw11, S. Blusk53, A. Bobrov31, V. Bocci22, A. Bondar31, N.
Bondar27, W. Bonivento15, S. Borghi48,51, A. Borgia53, T.J.V. Bowcock49, C.
Bozzi16, T. Brambach9, J. van den Brand39, J. Bressieux36, D. Brett51, M.
Britsch10, T. Britton53, N.H. Brook43, H. Brown49, A. Büchler-Germann37, I.
Burducea26, A. Bursche37, J. Buytaert35, S. Cadeddu15, O. Callot7, M.
Calvi20,j, M. Calvo Gomez33,n, A. Camboni33, P. Campana18,35, A. Carbone14, G.
Carboni21,k, R. Cardinale19,i,35, A. Cardini15, L. Carson50, K. Carvalho
Akiba2, G. Casse49, M. Cattaneo35, Ch. Cauet9, M. Charles52, Ph.
Charpentier35, P. Chen3,36, N. Chiapolini37, M. Chrzaszcz 23, K. Ciba35, X.
Cid Vidal34, G. Ciezarek50, P.E.L. Clarke47, M. Clemencic35, H.V. Cliff44, J.
Closier35, C. Coca26, V. Coco38, J. Cogan6, E. Cogneras5, P. Collins35, A.
Comerma-Montells33, A. Contu52, A. Cook43, M. Coombes43, G. Corti35, B.
Couturier35, G.A. Cowan36, D. Craik45, R. Currie47, C. D’Ambrosio35, P.
David8, P.N.Y. David38, I. De Bonis4, K. De Bruyn38, S. De Capua21,k, M. De
Cian37, J.M. De Miranda1, L. De Paula2, P. De Simone18, D. Decamp4, M.
Deckenhoff9, H. Degaudenzi36,35, L. Del Buono8, C. Deplano15, D. Derkach14,35,
O. Deschamps5, F. Dettori39, J. Dickens44, H. Dijkstra35, P. Diniz Batista1,
F. Domingo Bonal33,n, S. Donleavy49, F. Dordei11, A. Dosil Suárez34, D.
Dossett45, A. Dovbnya40, F. Dupertuis36, R. Dzhelyadin32, A. Dziurda23, A.
Dzyuba27, S. Easo46, U. Egede50, V. Egorychev28, S. Eidelman31, D. van Eijk38,
F. Eisele11, S. Eisenhardt47, R. Ekelhof9, L. Eklund48, I. El Rifai5, Ch.
Elsasser37, D. Elsby42, D. Esperante Pereira34, A. Falabella16,e,14, C.
Färber11, G. Fardell47, C. Farinelli38, S. Farry12, V. Fave36, V. Fernandez
Albor34, F. Ferreira Rodrigues1, M. Ferro-Luzzi35, S. Filippov30, C.
Fitzpatrick47, M. Fontana10, F. Fontanelli19,i, R. Forty35, O. Francisco2, M.
Frank35, C. Frei35, M. Frosini17,f, S. Furcas20, A. Gallas Torreira34, D.
Galli14,c, M. Gandelman2, P. Gandini52, Y. Gao3, J-C. Garnier35, J.
Garofoli53, J. Garra Tico44, L. Garrido33, D. Gascon33, C. Gaspar35, R.
Gauld52, N. Gauvin36, E. Gersabeck11, M. Gersabeck35, T. Gershon45,35, Ph.
Ghez4, V. Gibson44, V.V. Gligorov35, C. Göbel54, D. Golubkov28, A.
Golutvin50,28,35, A. Gomes2, H. Gordon52, M. Grabalosa Gándara33, R. Graciani
Diaz33, L.A. Granado Cardoso35, E. Graugés33, G. Graziani17, A. Grecu26, E.
Greening52, S. Gregson44, O. Grünberg55, B. Gui53, E. Gushchin30, Yu. Guz32,
T. Gys35, C. Hadjivasiliou53, G. Haefeli36, C. Haen35, S.C. Haines44, T.
Hampson43, S. Hansmann-Menzemer11, N. Harnew52, S.T. Harnew43, J. Harrison51,
P.F. Harrison45, T. Hartmann55, J. He7, V. Heijne38, K. Hennessy49, P.
Henrard5, J.A. Hernando Morata34, E. van Herwijnen35, E. Hicks49, M.
Hoballah5, P. Hopchev4, W. Hulsbergen38, P. Hunt52, T. Huse49, R.S. Huston12,
D. Hutchcroft49, D. Hynds48, V. Iakovenko41, P. Ilten12, J. Imong43, R.
Jacobsson35, A. Jaeger11, M. Jahjah Hussein5, E. Jans38, F. Jansen38, P.
Jaton36, B. Jean-Marie7, F. Jing3, M. John52, D. Johnson52, C.R. Jones44, B.
Jost35, M. Kaballo9, S. Kandybei40, M. Karacson35, T.M. Karbach9, J.
Keaveney12, I.R. Kenyon42, U. Kerzel35, T. Ketel39, A. Keune36, B. Khanji6,
Y.M. Kim47, M. Knecht36, O. Kochebina7, I. Komarov29, R.F. Koopman39, P.
Koppenburg38, M. Korolev29, A. Kozlinskiy38, L. Kravchuk30, K. Kreplin11, M.
Kreps45, G. Krocker11, P. Krokovny31, F. Kruse9, M. Kucharczyk20,23,35,j, V.
Kudryavtsev31, T. Kvaratskheliya28,35, V.N. La Thi36, D. Lacarrere35, G.
Lafferty51, A. Lai15, D. Lambert47, R.W. Lambert39, E. Lanciotti35, G.
Lanfranchi18, C. Langenbruch35, T. Latham45, C. Lazzeroni42, R. Le Gac6, J.
van Leerdam38, J.-P. Lees4, R. Lefèvre5, A. Leflat29,35, J. Lefrançois7, O.
Leroy6, T. Lesiak23, L. Li3, Y. Li3, L. Li Gioi5, M. Lieng9, M. Liles49, R.
Lindner35, C. Linn11, B. Liu3, G. Liu35, J. von Loeben20, J.H. Lopes2, E.
Lopez Asamar33, N. Lopez-March36, H. Lu3, J. Luisier36, A. Mac Raighne48, F.
Machefert7, I.V. Machikhiliyan4,28, F. Maciuc10, O. Maev27,35, J. Magnin1, S.
Malde52, R.M.D. Mamunur35, G. Manca15,d, G. Mancinelli6, N. Mangiafave44, U.
Marconi14, R. Märki36, J. Marks11, G. Martellotti22, A. Martens8, L. Martin52,
A. Martín Sánchez7, M. Martinelli38, D. Martinez Santos35, A. Massafferri1, Z.
Mathe12, C. Matteuzzi20, M. Matveev27, E. Maurice6, A. Mazurov16,30,35, J.
McCarthy42, G. McGregor51, R. McNulty12, M. Meissner11, M. Merk38, J. Merkel9,
D.A. Milanes13, M.-N. Minard4, J. Molina Rodriguez54, S. Monteil5, D. Moran12,
P. Morawski23, R. Mountain53, I. Mous38, F. Muheim47, K. Müller37, R.
Muresan26, B. Muryn24, B. Muster36, J. Mylroie-Smith49, P. Naik43, T.
Nakada36, R. Nandakumar46, I. Nasteva1, M. Needham47, N. Neufeld35, A.D.
Nguyen36, C. Nguyen-Mau36,o, M. Nicol7, V. Niess5, N. Nikitin29, T. Nikodem11,
A. Nomerotski52,35, A. Novoselov32, A. Oblakowska-Mucha24, V. Obraztsov32, S.
Oggero38, S. Ogilvy48, O. Okhrimenko41, R. Oldeman15,d,35, M. Orlandea26, J.M.
Otalora Goicochea2, P. Owen50, B.K. Pal53, A. Palano13,b, M. Palutan18, J.
Panman35, A. Papanestis46, M. Pappagallo48, C. Parkes51, C.J. Parkinson50, G.
Passaleva17, G.D. Patel49, M. Patel50, G.N. Patrick46, C. Patrignani19,i, C.
Pavel-Nicorescu26, A. Pazos Alvarez34, A. Pellegrino38, G. Penso22,l, M. Pepe
Altarelli35, S. Perazzini14,c, D.L. Perego20,j, E. Perez Trigo34, A. Pérez-
Calero Yzquierdo33, P. Perret5, M. Perrin-Terrin6, G. Pessina20, A.
Petrolini19,i, A. Phan53, E. Picatoste Olloqui33, B. Pie Valls33, B.
Pietrzyk4, T. Pilař45, D. Pinci22, S. Playfer47, M. Plo Casasus34, F. Polci8,
G. Polok23, A. Poluektov45,31, E. Polycarpo2, D. Popov10, B. Popovici26, C.
Potterat33, A. Powell52, J. Prisciandaro36, V. Pugatch41, A. Puig Navarro33,
W. Qian53, J.H. Rademacker43, B. Rakotomiaramanana36, M.S. Rangel2, I.
Raniuk40, N. Rauschmayr35, G. Raven39, S. Redford52, M.M. Reid45, A.C. dos
Reis1, S. Ricciardi46, A. Richards50, K. Rinnert49, D.A. Roa Romero5, P.
Robbe7, E. Rodrigues48,51, F. Rodrigues2, P. Rodriguez Perez34, G.J. Rogers44,
S. Roiser35, V. Romanovsky32, A. Romero Vidal34, M. Rosello33,n, J.
Rouvinet36, T. Ruf35, H. Ruiz33, G. Sabatino21,k, J.J. Saborido Silva34, N.
Sagidova27, P. Sail48, B. Saitta15,d, C. Salzmann37, B. Sanmartin Sedes34, M.
Sannino19,i, R. Santacesaria22, C. Santamarina Rios34, R. Santinelli35, E.
Santovetti21,k, M. Sapunov6, A. Sarti18,l, C. Satriano22,m, A. Satta21, M.
Savrie16,e, D. Savrina28, P. Schaack50, M. Schiller39, H. Schindler35, S.
Schleich9, M. Schlupp9, M. Schmelling10, B. Schmidt35, O. Schneider36, A.
Schopper35, M.-H. Schune7, R. Schwemmer35, B. Sciascia18, A. Sciubba18,l, M.
Seco34, A. Semennikov28, K. Senderowska24, I. Sepp50, N. Serra37, J. Serrano6,
P. Seyfert11, M. Shapkin32, I. Shapoval40,35, P. Shatalov28, Y. Shcheglov27,
T. Shears49, L. Shekhtman31, O. Shevchenko40, V. Shevchenko28, A. Shires50, R.
Silva Coutinho45, T. Skwarnicki53, N.A. Smith49, E. Smith52,46, M. Smith51, K.
Sobczak5, F.J.P. Soler48, A. Solomin43, F. Soomro18,35, D. Souza43, B. Souza
De Paula2, B. Spaan9, A. Sparkes47, P. Spradlin48, F. Stagni35, S. Stahl11, O.
Steinkamp37, S. Stoica26, S. Stone53,35, B. Storaci38, M. Straticiuc26, U.
Straumann37, V.K. Subbiah35, S. Swientek9, M. Szczekowski25, P. Szczypka36, T.
Szumlak24, S. T’Jampens4, M. Teklishyn7, E. Teodorescu26, F. Teubert35, C.
Thomas52, E. Thomas35, J. van Tilburg11, V. Tisserand4, M. Tobin37, S. Tolk39,
S. Topp-Joergensen52, N. Torr52, E. Tournefier4,50, S. Tourneur36, M.T.
Tran36, A. Tsaregorodtsev6, N. Tuning38, M. Ubeda Garcia35, A. Ukleja25, U.
Uwer11, V. Vagnoni14, G. Valenti14, R. Vazquez Gomez33, P. Vazquez Regueiro34,
S. Vecchi16, J.J. Velthuis43, M. Veltri17,g, G. Veneziano36, M. Vesterinen35,
B. Viaud7, I. Videau7, D. Vieira2, X. Vilasis-Cardona33,n, J. Visniakov34, A.
Vollhardt37, D. Volyanskyy10, D. Voong43, A. Vorobyev27, V. Vorobyev31, C.
Voß55, H. Voss10, R. Waldi55, R. Wallace12, S. Wandernoth11, J. Wang53, D.R.
Ward44, N.K. Watson42, A.D. Webber51, D. Websdale50, M. Whitehead45, J.
Wicht35, D. Wiedner11, L. Wiggers38, G. Wilkinson52, M.P. Williams45,46, M.
Williams50, F.F. Wilson46, J. Wishahi9, M. Witek23, W. Witzeling35, S.A.
Wotton44, S. Wright44, S. Wu3, K. Wyllie35, Y. Xie47, F. Xing52, Z. Xing53, Z.
Yang3, R. Young47, X. Yuan3, O. Yushchenko32, M. Zangoli14, M. Zavertyaev10,a,
F. Zhang3, L. Zhang53, W.C. Zhang12, Y. Zhang3, A. Zhelezov11, L. Zhong3, A.
Zvyagin35.

1Centro Brasileiro de Pesquisas Físicas (CBPF), Rio de Janeiro, Brazil
2Universidade Federal do Rio de Janeiro (UFRJ), Rio de Janeiro, Brazil
3Center for High Energy Physics, Tsinghua University, Beijing, China
4LAPP, Université de Savoie, CNRS/IN2P3, Annecy-Le-Vieux, France
5Clermont Université, Université Blaise Pascal, CNRS/IN2P3, LPC, Clermont-
Ferrand, France
6CPPM, Aix-Marseille Université, CNRS/IN2P3, Marseille, France
7LAL, Université Paris-Sud, CNRS/IN2P3, Orsay, France
8LPNHE, Université Pierre et Marie Curie, Université Paris Diderot,
CNRS/IN2P3, Paris, France
9Fakultät Physik, Technische Universität Dortmund, Dortmund, Germany
10Max-Planck-Institut für Kernphysik (MPIK), Heidelberg, Germany
11Physikalisches Institut, Ruprecht-Karls-Universität Heidelberg, Heidelberg,
Germany
12School of Physics, University College Dublin, Dublin, Ireland
13Sezione INFN di Bari, Bari, Italy
14Sezione INFN di Bologna, Bologna, Italy
15Sezione INFN di Cagliari, Cagliari, Italy
16Sezione INFN di Ferrara, Ferrara, Italy
17Sezione INFN di Firenze, Firenze, Italy
18Laboratori Nazionali dell’INFN di Frascati, Frascati, Italy
19Sezione INFN di Genova, Genova, Italy
20Sezione INFN di Milano Bicocca, Milano, Italy
21Sezione INFN di Roma Tor Vergata, Roma, Italy
22Sezione INFN di Roma La Sapienza, Roma, Italy
23Henryk Niewodniczanski Institute of Nuclear Physics Polish Academy of
Sciences, Kraków, Poland
24AGH University of Science and Technology, Kraków, Poland
25Soltan Institute for Nuclear Studies, Warsaw, Poland
26Horia Hulubei National Institute of Physics and Nuclear Engineering,
Bucharest-Magurele, Romania
27Petersburg Nuclear Physics Institute (PNPI), Gatchina, Russia
28Institute of Theoretical and Experimental Physics (ITEP), Moscow, Russia
29Institute of Nuclear Physics, Moscow State University (SINP MSU), Moscow,
Russia
30Institute for Nuclear Research of the Russian Academy of Sciences (INR RAN),
Moscow, Russia
31Budker Institute of Nuclear Physics (SB RAS) and Novosibirsk State
University, Novosibirsk, Russia
32Institute for High Energy Physics (IHEP), Protvino, Russia
33Universitat de Barcelona, Barcelona, Spain
34Universidad de Santiago de Compostela, Santiago de Compostela, Spain
35European Organization for Nuclear Research (CERN), Geneva, Switzerland
36Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
37Physik-Institut, Universität Zürich, Zürich, Switzerland
38Nikhef National Institute for Subatomic Physics, Amsterdam, The Netherlands
39Nikhef National Institute for Subatomic Physics and VU University Amsterdam,
Amsterdam, The Netherlands
40NSC Kharkiv Institute of Physics and Technology (NSC KIPT), Kharkiv, Ukraine
41Institute for Nuclear Research of the National Academy of Sciences (KINR),
Kyiv, Ukraine
42University of Birmingham, Birmingham, United Kingdom
43H.H. Wills Physics Laboratory, University of Bristol, Bristol, United
Kingdom
44Cavendish Laboratory, University of Cambridge, Cambridge, United Kingdom
45Department of Physics, University of Warwick, Coventry, United Kingdom
46STFC Rutherford Appleton Laboratory, Didcot, United Kingdom
47School of Physics and Astronomy, University of Edinburgh, Edinburgh, United
Kingdom
48School of Physics and Astronomy, University of Glasgow, Glasgow, United
Kingdom
49Oliver Lodge Laboratory, University of Liverpool, Liverpool, United Kingdom
50Imperial College London, London, United Kingdom
51School of Physics and Astronomy, University of Manchester, Manchester,
United Kingdom
52Department of Physics, University of Oxford, Oxford, United Kingdom
53Syracuse University, Syracuse, NY, United States
54Pontifícia Universidade Católica do Rio de Janeiro (PUC-Rio), Rio de
Janeiro, Brazil, associated to 2
55Institut für Physik, Universität Rostock, Rostock, Germany, associated to 11

aP.N. Lebedev Physical Institute, Russian Academy of Science (LPI RAS),
Moscow, Russia
bUniversità di Bari, Bari, Italy
cUniversità di Bologna, Bologna, Italy
dUniversità di Cagliari, Cagliari, Italy
eUniversità di Ferrara, Ferrara, Italy
fUniversità di Firenze, Firenze, Italy
gUniversità di Urbino, Urbino, Italy
hUniversità di Modena e Reggio Emilia, Modena, Italy
iUniversità di Genova, Genova, Italy
jUniversità di Milano Bicocca, Milano, Italy
kUniversità di Roma Tor Vergata, Roma, Italy
lUniversità di Roma La Sapienza, Roma, Italy
mUniversità della Basilicata, Potenza, Italy
nLIFAELS, La Salle, Universitat Ramon Llull, Barcelona, Spain
oHanoi University of Science, Hanoi, Viet Nam

Interpretations of measurements of time-dependent $C\\!P$ violation in
$B^{0}_{s}\rightarrow{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}\phi$ and
$B^{0}_{s}\rightarrow{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}f_{0}(980)$
decays have thus far assumed the dominance of the colour-suppressed tree-level
process. However, there are contributions from higher order (penguin)
processes (see Fig. 1) that cannot be calculated reliably in QCD and could be
large enough to affect the measured asymmetries. It has been suggested that
the penguin effects can be determined by means of an analysis of the angular
distribution of $B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*}(892)^{0}$, where the penguin
diagram is not suppressed relative to the tree-level one, and $SU(3)$ flavour
symmetry arguments can be used to determine the hadronic parameters entering
the $B^{0}_{s}\rightarrow{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}\phi$
observables [1].

Figure 1: Tree and penguin decay topologies contributing to the decays
$B^{0}_{s}\rightarrow J/\psi\kern 1.79997pt\overline{\kern-1.79997ptK}{}^{*0}$
and $B^{0}_{s}\rightarrow{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}\phi$.
The dashed line indicates a colour singlet exchange.

In this paper the $K^{*}(892)^{0}$ meson will be written as $K^{*0}$, while
for other $K^{*}$ resonances the mass will be given in parentheses.
Furthermore, mention of any specific mode implies the use of the charge
conjugated mode as well, and $K^{-}\pi^{+}$ pairs will be simply written as
$K\pi$. The decay $B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0}$ has already been observed by the
CDF experiment [2], which reported ${\cal B}(B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0})=(8.3\pm 3.8)\times 10^{-5}$.
Under the assumption that the light quark ($s$,$d$) is a spectator of the $b$
quark decay, the branching fraction can be approximated as

${\cal B}(B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0})\sim\frac{|V_{cd}|^{2}}{|V_{cs}|^{2}}\times{\cal
B}(B^{0}\rightarrow J/\psi K^{*0})=(6.5\pm 1.0)\times 10^{-5},$ (1)

with $|V_{cd}|=0.230\pm 0.011$, $|V_{cs}|=1.023\pm 0.036$ [3], and ${\cal
B}(B^{0}\rightarrow J/\psi K^{*0})=(1.29\pm 0.05\pm 0.13)\times 10^{-3}$ [4].
The measurement in Ref. [4], where the $K\pi$ S-wave contribution is
subtracted, is used instead of the PDG average.

In this paper, $0.37\mbox{\,fb}^{-1}$ of data taken in 2011 are used to
determine ${\cal B}(B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0})$, to study the angular properties
of the decay products of the $B^{0}_{s}$ meson, and to measure the resonant
contributions to the $K\pi$ spectrum in the region of the $K^{*0}$ meson. The
measurement of the branching fraction uses the decay $B^{0}\rightarrow J/\psi
K^{*0}$ as a normalization mode.

The LHCb detector [5] is a single-arm forward spectrometer covering the
pseudo-rapidity range $2<\eta<5$. The detector includes a high precision
tracking system consisting of a silicon-strip vertex detector located around
the interaction point, a large-area silicon-strip detector located upstream of
a dipole magnet with a bending power of about $4{\rm\,Tm}$, and three stations
of silicon-strip detectors and straw drift tubes placed downstream. The
combined tracking system has a momentum resolution $\Delta p/p$ that varies
from 0.4 % at 5${\mathrm{\,Ge\kern-1.00006ptV\\!/}c}$ to 0.6 % at
100${\mathrm{\,Ge\kern-1.00006ptV\\!/}c}$. Two ring-imaging Cherenkov
detectors (RICH) are used to determine the identity of charged particles. The
separation of pions and kaons is such that, for efficiencies of $\sim 75\%$
the rejection power is above $99\%$. Photon, electron and hadron candidates
are identified by a calorimeter system consisting of scintillating-pad and
pre-shower detectors, an electromagnetic calorimeter and a hadronic
calorimeter. Muons are identified by alternating layers of iron and multiwire
proportional chambers.

The trigger consists of a hardware stage, based on information from the
calorimeter and muon systems, followed by a software stage called High Level
Trigger (HLT) that applies a full event reconstruction. Events with muon final
states are triggered using two hardware trigger decisions: the single-muon
decision (one muon candidate with transverse momentum $p_{\rm
T}>1.5$${\mathrm{\,Ge\kern-1.00006ptV\\!/}c}$), and the di-muon decision (two
muon candidates with $p_{{\rm T},1}$ and $p_{{\rm T},2}$ such that
$\sqrt{p_{{\rm T},1}\,p_{{\rm
T},2}}>1.3$${\mathrm{\,Ge\kern-1.00006ptV\\!/}c}$). All tracks in the HLT are
required to have a $p_{\rm T}>0.5$${\mathrm{\,Ge\kern-1.00006ptV\\!/}c}$. The
single muon trigger decision in the HLT selects events with at least one muon
track with an impact parameter ${\rm IP}$ $>0.1$ mm with respect to the
primary vertex and $p_{\rm T}>1.0$${\mathrm{\,Ge\kern-1.00006ptV\\!/}c}$. The
di-muon trigger decision, designed to select $J/\psi$ mesons, also requires a
di-muon mass ($M_{\mu\mu}$)
$2970<M_{\mu\mu}<3210$${\mathrm{\,Me\kern-1.00006ptV\\!/}c^{2}}$.

Simulated events are used to compute detection efficiencies and angular
acceptances. For this purpose, $pp$ collisions are generated using Pythia 6.4
[6] with a specific LHCb configuration [7]. Decays of hadronic particles are
described by EvtGen [8] in which final state radiation is generated using
Photos [9]. The interaction of the generated particles with the detector and
its response are implemented using the Geant4 toolkit [10,
*Agostinelli:2002hh] as described in Ref. [12].

The selection of $B^{0}_{(s)}\rightarrow
J/\psi\accentset{\scalebox{0.4}{(}\raisebox{-1.7pt}{\bf{--}}\scalebox{0.4}{)}}{K}^{*0}$
decays first requires the reconstruction of a
${J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}\rightarrow\mu^{+}\mu^{-}$
candidate. The ${J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}$ vertex is
required to be separated from any primary vertex (PV) by a distance-of-flight
significance greater than 13. Subsequently, the muons from the
${J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}$ decay are combined with the $K$
and $\pi$ candidates to form a good vertex, where the di-muon mass is
constrained to the ${J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}$ mass. A
$\mbox{$p_{\rm T}$}>0.5$${\mathrm{\,Ge\kern-1.00006ptV\\!/}c}$ is required for
each of the four daughter tracks. Positive muon identification is required for
the two tracks of the ${J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}$ decay,
and the kaons and pions are selected using the different hadron probabilities
based on combined information given by the RICH detectors. The candidate
$B^{0}_{(s)}$ momentum is required to be compatible with the flight direction
as given by the vector connecting the PV with the candidate vertex. An
explicit veto to remove $B^{+}\rightarrow J/\psi K^{+}$ events is applied, as
they otherwise would pollute the upper sideband of the $B^{0}_{(s)}$ mass
spectrum.

Following this initial selection, several geometrical variables are combined
into a single discriminant geometrical likelihood variable (GL). This
multivariate method is described in Refs. [13, 14]. The geometrical variables
chosen to build the GL are: the $B^{0}_{(s)}$ candidate minimum impact
parameter with respect to any PV in the event, the decay time of the
$B^{0}_{(s)}$ candidate, the minimum impact parameter $\chi^{2}$ of the four
daughter tracks with respect to all PV in the event (defined as the difference
between the $\chi^{2}$ of the PV built with and without the considered track),
the distance of closest approach between the
${J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}$ and $K^{*0}$ trajectories
reconstructed from their decay products, and the $p_{\rm T}$ of the
$B^{0}_{(s)}$ candidate. The GL was tuned using simulated $B^{0}\rightarrow
J/\psi K^{*0}$ signal passing the selection criteria, and background from data
in the $B^{0}_{(s)}$ mass sidebands with a value for the kaon particle
identification variable in a range which does not overlap with the one used to
select the data sample for the final analysis.

The $K\pi$ mass spectrum in the $B^{0}\rightarrow J/\psi K\pi$ channel is
dominated by the $K^{*0}$ resonance but contains a non-negligible S-wave
contribution, originating from $K^{*}_{0}(1430)^{0}$ and non-resonant $K\pi$
pairs [15]. To determine ${\cal B}(B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0})$ it is therefore important to
measure the S-wave magnitude in both
$B^{0}_{(s)}\rightarrow{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}K\pi$
channels. The $K\pi$ spectrum is analyzed in terms of a non-resonant S-wave
and several $K\pi$ resonances parameterized using relativistic Breit-Wigner
distributions with mass-dependent widths, following closely [15]. The
considered waves are: a non-resonant S-wave amplitude interfering with the
$K^{*}_{0}(1430)^{0}$ resonance, $K^{*0}$ for the P-wave and
$K^{*}_{2}(1430)^{0}$ for the D-wave. F-wave and G-wave components are found
to be negligible in the $B^{0}$ fit. In bins of the $K\pi$ mass, a fit is made
to the $B^{0}_{(s)}$ candidate mass distribution to determine the yield. As
shown in Fig. 2, a fit is then made to the $B^{0}$ and $B^{0}_{s}$ yields as a
function of the $K\pi$ mass without any efficiency correction. The S and
P-wave components dominate in the $\pm
40\,{\mathrm{\,Me\kern-1.00006ptV\\!/}c^{2}}$ window around the $K^{*0}$ mass,
where the $K^{*0}$ contribution is above 90%. A more exact determination of
this contribution using this method would require $K\pi$ mass-dependent
angular acceptance corrections. For the branching fraction calculation, the
fraction of $K^{*0}$ candidates is determined from a different full angular
and mass fit, which is described next.

The angular and mass analysis is based on an unbinned maximum likelihood fit
which handles simultaneously the mass
($M_{{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}K\pi}$) and the angular
parameters of the $B^{0}_{(s)}$ decays and the background. Each of these three
components is is modelled as a product of probability density functions ($\rm
PDF$), ${\cal P}(M_{{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip
2.0mu}K\pi},\psi,\theta,\varphi)={\cal
P}(M_{{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}K\pi})\,{\cal
P}(\psi,\theta,\varphi),$ with $\psi$ the angle between the kaon momentum in
the rest frame of the $K^{*0}$ and the direction of motion of the $K^{*0}$ in
the rest frame of the $B$. The polar and azimuthal angles ($\theta$,
$\varphi$) describe the direction of the $\mu^{+}$ in the coordinate system
defined in the ${J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}$ rest frame,
where the $x$ axis is the direction of motion of the $B^{0}_{(s)}$ meson, the
$z$ axis is normal to the plane formed by the $x$ axis and the kaon momentum,
and the $y$ axis is chosen so that the $y$ component of the kaon momentum is
positive.

The function describing the mass distribution of both $B^{0}_{(s)}$ signal
peaks is the sum of two Crystal Ball (CB) functions [17], which are a
combination of a Gaussian and a power law function to describe the radiative
tail at low masses,

${\cal P}(M_{{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}K\pi})=f\,{\rm
CB}(M_{{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip
2.0mu}K\pi},\mu_{B},\sigma_{1},\alpha_{1})+(1-f)\,\operatorname{CB}(M_{{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip
2.0mu}K\pi},\mu_{B},\sigma_{2},\alpha_{2}).$ (2)

The starting point of the radiative tail is governed by a transition point
parameter $\alpha_{(1,2)}$. The mean and width of the Gaussian component are
$\mu_{B}$ and $\sigma_{(1,2)}$. The values of the $f$, $\sigma_{1}$,
$\sigma_{2}$, $\alpha_{1}$ and $\alpha_{2}$ parameters are constrained to be
the same for the $B^{0}_{s}$ and $B^{0}$ peaks. The difference in the means
between the $B^{0}_{s}$ and the $B^{0}$ distributions,
($\mu_{B^{0}_{s}}-\mu_{B^{0}}$), is fixed to the value taken from Ref. [18].
The mass $\rm PDF$ of the background is described by an exponential function.

Figure 2: Fit to the $K\pi$ mass spectrum for (a) $B^{0}\rightarrow J/\psi
K\pi$ events, and (b) $B^{0}_{s}\rightarrow J/\psi K\pi$ events. The
$B^{0}_{(s)}\rightarrow J/\psi K\pi$ yields in each bin of $K\pi$ mass are
determined from a fit to the ${J\mskip-3.0mu/\mskip-2.0mu\psi\mskip
2.0mu}K\pi$ mass spectrum. The pink dashed-dotted line represents the
$K^{*0}$, the red short-dashed line is the S-wave and the black dotted line is
the $K^{*}_{2}(1430)$. The black solid line is their sum.

Assuming that direct $C\\!P$ violation and the $B^{0}_{(s)}-\kern
1.79993pt\overline{\kern-1.79993ptB}{}^{0}_{(s)}$ production asymmetry are
insignificant, the differential decay rate is [1, 16]

$\displaystyle\frac{{\rm d}^{3}\Gamma}{{\rm d}\Omega}$ $\displaystyle\propto$
$\displaystyle 2|A_{0}|^{2}\cos^{2}\psi(1-\sin^{2}\theta\cos^{2}\varphi)$
$\displaystyle+$
$\displaystyle|A_{\parallel}|^{2}\sin^{2}\psi(1-\sin^{2}\theta\sin^{2}\varphi)$
$\displaystyle+$ $\displaystyle|A_{\perp}|^{2}\sin^{2}\psi\sin^{2}\theta$
$\displaystyle+$
$\displaystyle\frac{1}{\sqrt{2}}|A_{0}||A_{\parallel}|\cos(\delta_{\parallel}-\delta_{0})\sin
2\psi\sin^{2}\theta\sin 2\varphi$ $\displaystyle+$
$\displaystyle\frac{2}{3}|A_{\rm
S}|^{2}\left[1-\sin^{2}\theta\cos^{2}\varphi\right]$ $\displaystyle+$
$\displaystyle\frac{4\sqrt{3}}{3}|A_{0}||A_{\rm S}|\cos(\delta_{\rm
S}-\delta_{0})\cos\psi\left[1-\sin^{2}\theta\cos^{2}\varphi\right]$
$\displaystyle+$ $\displaystyle\frac{\sqrt{6}}{3}|A_{\parallel}||A_{\rm
S}|\cos(\delta_{\parallel}-\delta_{\rm S})\sin\psi\sin^{2}\theta\sin
2\varphi,$

where $A_{0}$, $A_{\parallel}$ and $A_{\perp}$ are the decay amplitudes
corresponding to longitunally and transversely polarized vector mesons.
$A_{\rm S}=|A_{\rm S}|e^{i\delta_{\rm S}}$ is the $K\pi$ S-wave amplitude and
($\delta_{\parallel}-\delta_{0}$) the relative phase between the longitudinal
and parallel amplitudes. The convention $\delta_{0}=0$ is used hereafter. The
$\Omega$ differential is ${\rm d}\Omega\equiv{\rm d}\cos\psi\,{\rm
d}\cos\theta\,{\rm d}\varphi$. The polarization fractions are normalized
according to

$f_{L,\parallel,\perp}=\frac{|A_{0,\parallel,\perp}|^{2}}{|A_{0}|^{2}+|A_{\parallel}|^{2}+|A_{\perp}|^{2}},$
(4)

which satisfy $f_{L}+f_{\parallel}+f_{\perp}=1$.

The parameters $f_{L}$, $f_{\parallel}$ and $\delta_{\parallel}$ describing
the P-wave are left floating in the fit. The $|A_{\rm S}|$ amplitude and the
$\delta_{\rm S}$ phase depend on $M_{K\pi}$, but this dependence is ignored in
the fit, which is performed in a $K\pi$ mass window of $\pm
40{\mathrm{\,Me\kern-1.00006ptV\\!/}c^{2}}$, and they are just treated also as
floating parameters. A systematic uncertainty is later associated with this
assumption. The angular distribution of observed events is parameterized as a
product of the expression in Eq. S0.Ex6 and a detector acceptance function,
${\rm Acc}(\Omega)$, which describes the efficiency to trigger, reconstruct
and select the events. Simulation studies have shown almost no correlation
between the three one-dimensional angular acceptances ${\rm
Acc}_{\psi}(\psi)$, ${\rm Acc}_{\theta}(\theta)$ and ${\rm
Acc}_{\varphi}(\varphi)$. Therefore the global acceptance factorizes as ${\rm
Acc}(\Omega)={\rm Acc}_{\psi}(\psi)\,{\rm Acc}_{\theta}(\theta)\,{\rm
Acc}_{\varphi}(\varphi)$, where ${\rm Acc}_{\psi}(\psi)$ is parameterized as a
fifth degree polynomial, ${\rm Acc}_{\theta}(\theta)$ as a second degree
polynomial and ${\rm Acc}_{\phi}(\phi)$ as a sinusoidal function. A systematic
uncertainty due to this factorization hypothesis is later evaluated. The
angular distribution for the background component is determined using the
upper sideband of the $B^{0}_{s}$ mass spectrum, defined as the interval
$[5417,5779]$ ${\mathrm{\,Me\kern-1.00006ptV\\!/}c^{2}}$.

Figure 3 shows the projection of the fit in the
$M_{{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}K\pi}$ mass axis, together
with the projections in the angular variables in a window of $\pm 25$
${\mathrm{\,Me\kern-1.00006ptV\\!/}c^{2}}$ around the $B^{0}_{s}$ mass. The
number of candidates corresponding to $B^{0}$ and $B^{0}_{s}$ decays is found
to be $13,\\!365\pm 116$ and $114\pm 11$, respectively.

Figure 3: Projections of the fit in $M_{J/\psi K\pi}$ and in the angular variables for the mass range indicated by the two dashed vertical lines in the mass plot. The red dashed, pink long-dashed, and blue dotted lines represent the fitted contributions from $B^{0}\rightarrow J/\psi K^{*0}$, $B^{0}_{s}\rightarrow J/\psi\kern 1.79997pt\overline{\kern-1.79997ptK}{}^{*0}$ and background. The black solid line is their sum. Table 1: Summary of the measured $B^{0}_{s}\rightarrow J/\psi\kern 1.79997pt\overline{\kern-1.79997ptK}{}^{*0}$ angular properties and their statistical and systematic uncertainties. Parameter name | $|A_{\rm S}|^{2}$ | $f_{L}$ | $f_{\parallel}$
---|---|---|---
Value and statistical error | $0.07_{-0.07}^{+0.15}$ | $0.50\pm 0.08$ | $0.19_{-0.08}^{+0.10}$
Systematic uncertainties
Angular acceptance | $0.044$ | $0.011$ | $0.016$
Background angular model | $0.038$ | $0.017$ | $0.013$
Assumption $\delta_{\rm S}(M_{K\pi})=$ constant | $0.026$ | $0.005$ | $0.002$
$B^{0}$ contamination | $0.036$ | $0.004$ | $0.007$
Fit bias | $-$ | $-$ | $0.005$
Total systematic error | $0.073$ | $0.021$ | $0.022$
Table 2: Angular parameters of $B^{0}\rightarrow J/\psi K^{*0}$ needed to compute ${\cal B}(B^{0}_{s}\rightarrow J/\psi\kern 1.79997pt\overline{\kern-1.79997ptK}{}^{*0})$. The systematic uncertainties from background modelling and the mass $\rm PDF$ are found to be negligible in this case. Parameter name | $|A_{\rm S}|^{2}$ | $f_{L}$ | $f_{\parallel}$
---|---|---|---
Value and statistical error | $0.037\pm 0.010$ | $0.569\pm 0.007$ | $0.240\pm 0.009$
Systematic uncertainties
Angular acceptance | $0.044$ | $0.011$ | $0.016$
Assumption $\delta_{\rm S}(M_{K\pi})=$ constant | $0.026$ | $0.005$ | $0.002$
Total systematic error | $0.051$ | $0.012$ | $0.016$

Tables 1 and 2 summarize the measurements of the $B^{0}_{(s)}\rightarrow
J/\psi\accentset{\scalebox{0.4}{(}\raisebox{-1.7pt}{\bf{--}}\scalebox{0.4}{)}}{K}^{*0}$
angular parameters, together with their statistical and systematic
uncertainties. The correlation coefficient given by the fit between $f_{L}$
and $f_{\parallel}$ is $\rho=-0.44$ for $B^{0}_{s}$ decays. The results for
the $B^{0}\rightarrow J/\psi K^{*0}$ decay are in good agreement with previous
measurements [16, 4, 19, 20]. Based on this agreement, the systematic
uncertainties caused by the modelling of the angular acceptance were evaluated
by summing in quadrature the statistical error on the measured
$B^{0}\rightarrow J/\psi K^{*0}$ parameters with the uncertainties on the
world averages ($f_{L}=0.570\pm 0.008$ and $f_{\perp}=0.219\pm 0.010$) [3].
The angular analysis was repeated with two additional acceptance descriptions,
one which uses a three-dimensional histogram to describe the efficiency
avoiding any factorization hypothesis, and another one based on a method of
normalization weights described in Ref. [21]. A very good agreement was found
in the values of the polarization fractions computed with all the three
methods. For the parameter $|A_{\rm S}|^{2}$, uncertainties caused by the
finite size of the simulation sample used for the acceptance description, as
well as from the studies with several acceptance models, are included. The
systematic uncertainty caused by the choice of the angular $\rm PDF$ for the
background is shown for the $B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0}$ decay but it was found to be
negligible for $B^{0}\rightarrow J/\psi K^{*0}$.

Also included in Tables 1 and 2 is the uncertainty from the assumption of a
constant $\delta_{\rm S}$ as a function of $M_{K\pi}$. This assumption can be
relaxed by adding an extra free parameter to the angular $\rm PDF$. This
addition makes the fit unstable for the small size of the $B^{0}_{s}$ sample,
but can be used in the control channel $B^{0}\rightarrow J/\psi K^{*0}$. The
differences found in the $B^{0}$ parameters with the two alternate
parameterizations are used as systematic uncertainties. The parameters
$\delta_{\parallel}$ fit to
$\cos(\delta_{\parallel})=-0.960^{+0.021}_{-0.017}$ for the $B^{0}$ and to
$\cos(\delta_{\parallel})=-0.93\pm 0.31$ (where the error corresponds to the
positive one, being symmetrized) for the $B^{0}_{s}$. These parameters could
in principle affect the efficiency corrections, but it was found that the
effect of different values of $\delta_{\parallel}$ on the overall efficiency
is negligible. A simulation study of the fit pulls has shown that the errors
on $f_{L}$ and $f_{\parallel}$ of the $B^{0}_{s}$ decays are overestimated by
a small amount ($\sim 10\%$) since they do not follow exactly a Gaussian
distribution, therefore the decision was taken to quote an uncertainty which
corresponds to an interval containing $68\%$ of the generated experiments,
rather than giving an error corresponding to a log-likelihood interval of
$0.5$. A slight bias observed in the pulls of $f_{\parallel}$ in $B^{0}_{s}$
decays was accounted for by adding a systematic uncertainty corresponding to
6% of the statistical error.

The ratio of the two branching fractions is obtained from

$\frac{{\cal B}(B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0})}{{\cal B}(B^{0}\rightarrow J/\psi
K^{*0})}=\frac{f_{d}}{f_{s}}\,\frac{\varepsilon^{\rm
tot}_{B^{0}}}{\varepsilon^{\rm
tot}_{B^{0}_{s}}}\,\frac{\lambda_{B^{0}}}{\lambda_{B^{0}_{s}}}\,\frac{f^{(d)}_{K^{*0}}}{f^{(s)}_{K^{*0}}}\,\frac{N_{B^{0}_{s}}}{N_{B^{0}}},$
(5)

where $f_{d}$ $(f_{s})$ is the probability of the $b$ quark to hadronize to
$B^{0}$ $(B^{0}_{s})$ mesons, $\varepsilon^{\rm tot}_{B^{0}}/\varepsilon^{\rm
tot}_{B^{0}_{s}}$ is the efficiency ratio,
$\lambda_{B^{0}}/\lambda_{B^{0}_{s}}$ is the ratio of angular corrections,
$f^{(s)}_{K^{*0}}/f^{(d)}_{K^{*0}}$ is the ratio of $K^{*0}$ fractions and
$N_{B^{0}_{s}}/N_{B^{0}}$ is the ratio of signal yields. The value of
$f_{d}/f_{s}$ has been taken from Ref. [22]. The efficiencies in the ratio
$\varepsilon_{B^{0}}^{\rm tot}/\varepsilon_{B^{0}_{s}}^{\rm tot}$ are computed
using simulation and receive two contributions: the efficiency of the offline
reconstruction (including geometrical acceptance) and selection cuts, and the
trigger efficiency on events that satisfy the analysis offline selection
criteria. The systematic uncertainty in the efficiency ratio is negligible due
to the similarity of the final states. Effects due to possible differences in
the decay time acceptance between data and simulation were found to affect the
efficiency ratio by less than 1 per mille. On the other hand, since the
efficiency depends on the angular distribution of the decay products,
correction factors $\lambda_{B^{0}}$ and $\lambda_{B^{0}_{s}}$ are applied to
account for the difference between the angular amplitudes used in simulation
and those measured in the data. The observed numbers of $B^{0}$ and
$B^{0}_{s}$ decays, denoted by $N_{B^{0}}$ and $N_{B^{0}_{s}}$, correspond to
the number of $B^{0}_{s}\rightarrow J/\psi K\pi$ and $B^{0}\rightarrow J/\psi
K\pi$ decays with a $K\pi$ mass in a $\pm 40$
${\mathrm{\,Me\kern-1.00006ptV\\!/}c^{2}}$ window around the nominal $K^{*0}$
mass. This includes mostly the $K^{*0}$ meson, but also an S-wave component
and the interference between the S-wave and P-wave components. The fraction of
candidates with a $K^{*0}$ meson present is then

$f_{K^{*0}}=\frac{\displaystyle\int_{\Omega}{\rm
Acc}(\Omega)\;\left.\frac{{\rm d}^{3}\Gamma}{{\rm d}\Omega}\right|_{|A_{\rm
S}|=0}\;{\rm d}\Omega}{\displaystyle\int_{\Omega}{\rm
Acc}(\Omega)\;\;\frac{{\rm d}^{3}\Gamma}{{\rm d}\Omega}\;\;{\rm d}\Omega},$
(6)

from which the ratio $f^{(s)}_{K^{*0}}/f^{(d)}_{K^{*0}}=1.09\pm 0.08$ follows.
Table 3 summarizes all the numbers needed to compute the ratio of branching
fractions

$\frac{{\cal B}(B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0})}{{\cal B}(B^{0}\rightarrow J/\psi
K^{*0})}=\big{(}3.43_{-0.36}^{+0.34}\pm 0.50\big{)}\%.$

The contributions to the systematic uncertainty are also listed in Table 3 and
their relative magnitudes are: $1.2\%$ for the error in the efficiency ratio;
$2.5\%$ for the uncertainty on the transition point ($\alpha$) of the Crystal
Ball function; $8.6\%$ for the parameterization of the upper tail of the
$B^{0}$ peak; $3.9\%$ for the angular correction of the efficiencies; $7.3\%$
for the uncertainty on the ratio $f^{(s)}_{K^{*0}}/f^{(d)}_{K^{*0}}$ and
$7.7\%$ for the uncertainty on $f_{d}/f_{s}$. The errors are added in
quadrature.

Table 3: Parameter values and errors for $\frac{{\cal B}(B^{0}_{s}\rightarrow J/\psi\kern 1.25998pt\overline{\kern-1.25998ptK}{}^{*0})}{{\cal B}(B^{0}\rightarrow J/\psi K^{*0})}$. Parameter | Name | Value
---|---|---
Hadronization fractions | $f_{d}/f_{s}$ | $3.75\pm 0.29$
Efficiency ratio | $\varepsilon^{\rm tot}_{B^{0}}/\varepsilon^{\rm tot}_{B^{0}_{s}}$ | $0.97\pm 0.01$
Angular corrections | $\lambda_{B^{0}}/\lambda_{B^{0}_{s}}$ | $1.01\pm 0.04$
Ratio of $K^{*0}$ fractions | $f^{(s)}_{K^{*0}}/f^{(d)}_{K^{*0}}$ | $1.09\pm 0.08$
$B$ signal yields | $N_{B^{0}_{s}}/N_{B^{0}}$ | $\left(8.5^{+0.9}_{-0.8}\pm 0.8\right)\times 10^{-3}$

Taking the value ${\cal B}(B^{0}\rightarrow J/\psi K^{*0})=(1.29\pm 0.05\pm
0.13)\times 10^{-3}$ from Ref. [4] the following branching fraction is
obtained,

${\cal B}(B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0})=\big{(}4.4_{-0.4}^{+0.5}\,\pm
0.8\,\big{)}\times 10^{-5}.$

This value is compatible with the CDF measurement [2] and is similar to the
naive quark spectator model prediction of Eq. (1), although it is closer to
the estimation in Ref. [1], ${\cal B}(B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0})\sim 2\times{\cal
B}(B^{0}_{d}\rightarrow J/\psi\rho^{0})=(4.6\pm 0.4)\times 10^{-5}$. The
branching fraction measured here is in fact the average of the
$B^{0}_{s}\rightarrow J/\psi\kern 1.99997pt\overline{\kern-1.99997ptK}{}^{*0}$
and $\kern
1.79993pt\overline{\kern-1.79993ptB}{}^{0}_{s}\rightarrow{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip
2.0mu}K^{*0}$ branching fractions and corresponds to the time integrated
quantity, while theory predictions usually refer to the branching fraction at
$t=0$ [23]. In the case of $B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0}$, the two differ by
$(\Delta\Gamma_{s}/2\Gamma_{s})^{2}=(0.77\pm 0.25)\%$, where
$\Delta\Gamma_{s}=\Gamma_{\rm L}-\Gamma_{\rm H}$, $\Gamma_{s}=(\Gamma_{\rm
L}+\Gamma_{\rm H})/2$, and $\Gamma_{\rm L(H)}$ is the decay width of the light
(heavy) $B^{0}_{s}$-mass eigenstate.

In conclusion, using $0.37$ fb-1 of $pp$ collisions collected by the LHCb
detector at $\sqrt{s}$ = 7 TeV, a measurement of the $B^{0}_{s}\rightarrow
J/\psi\kern 1.99997pt\overline{\kern-1.99997ptK}{}^{*0}$ branching fraction
yields ${\cal B}(B^{0}_{s}\rightarrow J/\psi\kern
1.99997pt\overline{\kern-1.99997ptK}{}^{*0})=\big{(}4.4_{-0.4}^{+0.5}\,\pm
0.8\,\big{)}\times 10^{-5}$. In addition, an angular analysis of the decay
products is presented, which provides the first measurement of the $K^{*0}$
polarization fractions in this decay, giving $f_{L}=0.50\pm 0.08\pm 0.02$,
$f_{\parallel}=0.19^{+0.10}_{-0.08}\pm 0.02$, and an S-wave contribution of
$|A_{\rm S}|^{2}=0.07^{+0.15}_{-0.07}$ in a $\pm
40\,{\mathrm{\,Me\kern-1.00006ptV\\!/}c^{2}}$ window around the $K^{*0}$ mass.

We express our gratitude to our colleagues in the CERN accelerator departments
for the excellent performance of the LHC. We thank the technical and
administrative staff at CERN and at the LHCb institutes, and acknowledge
support from the National Agencies: CAPES, CNPq, FAPERJ and FINEP (Brazil);
CERN; NSFC (China); CNRS/IN2P3 (France); BMBF, DFG, HGF and MPG (Germany); SFI
(Ireland); INFN (Italy); FOM and NWO (The Netherlands); SCSR (Poland); ANCS
(Romania); MinES of Russia and Rosatom (Russia); MICINN, XuntaGal and GENCAT
(Spain); SNSF and SER (Switzerland); NAS Ukraine (Ukraine); STFC (United
Kingdom); NSF (USA). We also acknowledge the support received from the ERC
under FP7 and the Region Auvergne.

## References

  * [1] S. Faller, R. Fleischer, and T. Mannel, Precision physics with $B^{0}_{s}\rightarrow J/\psi\phi$ at the LHC: the quest for new physics, Phys. Rev. D79 (2009) 014005, arXiv:0810.4248
  * [2] CDF collaboration, T. Aaltonen et al., Observation of $B^{0}_{s}\rightarrow J/\psi K^{*}(892)^{0}$ and $B^{0}_{s}\rightarrow J/\psi K^{0}_{S}$ decays, Phys. Rev. D83 (2011) 052012, arXiv:1102.1961
  * [3] Particle Data Group, K. Nakamura et al., Review of particle physics, J. Phys. G37 (2010) 075021
  * [4] Belle collaboration, K. Abe et al., Measurements of branching fractions and decay amplitudes in $B\rightarrow J/\psi K^{*}$ decays, Phys. Lett. B538 (2002) 11, arXiv:hep-ex/0205021
  * [5] LHCb collaboration, A. A. Alves Jr. et al., The LHCb detector at the LHC, JINST 3 (2008) S08005
  * [6] T. Sjöstrand, S. Mrenna, and P. Skands, PYTHIA 6.4 Physics and manual, JHEP 05 (2006) 026, arXiv:hep-ph/0603175
  * [7] I. Belyaev et al., Handling of the generation of primary events in Gauss, the LHCb simulation framework, Nuclear Science Symposium Conference Record (NSS/MIC) IEEE (2010) 1155
  * [8] D. J. Lange, The EvtGen particle decay simulation package, Nucl. Instrum. Meth. A462 (2001) 152
  * [9] P. Golonka and Z. Was, PHOTOS Monte Carlo: a precision tool for QED corrections in $Z$ and $W$ decays, Eur. Phys. J. C45 (2006) 97, arXiv:hep-ph/0506026
  * [10] GEANT4 collaboration, J. Allison et al., Geant4 developments and applications, IEEE Trans. Nucl. Sci. 53 (2006) 270
  * [11] GEANT4 collaboration, S. Agostinelli et al., GEANT4: a simulation toolkit, Nucl. Instrum. Meth. A506 (2003) 250
  * [12] M. Clemencic et al., The LHCb simulation application, Gauss: design, evolution and experience, J. of Phys: Conf. Ser. 331 (2011) 032023
  * [13] D. Martínez Santos, Study of the very rare decay $B_{s}\rightarrow\mu^{+}\mu^{-}$ in LHCb, PhD thesis, University of Santiago de Compostela, 2010, CERN-THESIS-2010-068
  * [14] D. Karlen, Using projections and correlations to approximate probability distributions, Comput. Phys. 12 (1998) 380, arXiv:physics/9805018
  * [15] BABAR collaboration, B. Aubert et al., Search for the $Z(4430)^{-}$ at BABAR, Phys. Rev. D79 (2009) 112001, arXiv:0811.0564
  * [16] BABAR collaboration, B. Aubert et al., Time-integrated and time-dependent angular analyses of $B\rightarrow J/\psi K\pi$: a measurement of $\cos 2\beta$ with no sign ambiguity from strong phases, Phys. Rev. D71 (2005) 032005, arXiv:hep-ex/0411016
  * [17] T. Skwarnicki, A study of the radiative cascade transitions between the Upsilon-prime and Upsilon resonances, PhD thesis, Institute of Nuclear Physics, Krakow, 1986, DESY-F31-86-02
  * [18] LHCb collaboration, R. Aaij et al., Measurement of $b$-hadron masses, Phys. Lett. B708 (2012) 241, arXiv:1112.4896
  * [19] D0 collaboration, V. Abazov et al., Measurement of the angular and lifetime parameters of the decays $B^{0}_{d}\rightarrow J/\psi K^{*0}$ and $B^{0}_{s}\rightarrow J/\psi\phi$, Phys. Rev. Lett. 102 (2009) 032001, arXiv:0810.0037
  * [20] CDF collaboration, T. Aaltonen et al., Angular analysis of $B^{0}_{s}\rightarrow{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}\phi$ and $B\rightarrow{J\mskip-3.0mu/\mskip-2.0mu\psi\mskip 2.0mu}K^{*}$ decays and measurement of $\Delta\Gamma_{s}$ and $\phi_{s}$, CDF public note (2011), CDF public note 8950
  * [21] T. du Pree, Search for a strange phase in beautiful oscillations, PhD thesis, Vrije Universiteit (Amsterdam), 2010, CERN-THESIS-2010-124
  * [22] LHCb collaboration, R. Aaij et al., Measurement of $b$ hadron production fractions in 7 TeV $pp$ collisions, Phys. Rev. D85 (2012) 032008, arXiv:1111.2357
  * [23] K. De Bruyn et al., Branching ratio measurements of $B_{s}$ decays, Phys. Rev. D86 (2012) 014027, arXiv:1204.1735