marginparsep has been altered.
topmargin has been altered.
marginparpush has been altered.
The page layout violates the ICML style. Please do not change the page layout,
or include packages like geometry, savetrees, or fullpage, which change it for
you. We’re not able to reliably undo arbitrary changes to the style. Please
remove the offending package(s), or layout-changing commands and try again.

A Meta-Analysis of Distributionally Robust Models

Anonymous Authors1

††footnotetext:  1Anonymous Institution, Anonymous City, Anonymous Region,
Anonymous Country. Correspondence to: Anonymous Author
<EMAIL_ADDRESS>
Preliminary work. Under review at ICML Workshop on Principles of Distribution
Shift 2022. Do not distribute.

###### Abstract

State-of-the-art image classifiers trained on massive datasets (such as
ImageNet) have been shown to be vulnerable to a range of both intentional and
incidental distribution shifts. On the other hand, several recent classifiers
with favorable out-of-distribution (OOD) robustness properties have emerged,
achieving high accuracy on their target tasks while maintaining their in-
distribution accuracy on challenging benchmarks. We present a meta-analysis on
a wide range of publicly released models, most of which have been published
over the last twelve months. Through this meta-analysis, we empirically
identify four main commonalities for all the best-performing OOD-robust
models, all of which illuminate the considerable promise of vision-language
pre-training.

##  1 Introduction

This meta-analysis catalogs the out-of-distribution (OOD) robustness of a
variety of image classifiers, with emphasis on models that have been publicly
released over the last several months. The goal of our meta-analysis is to
address the following question:

> _What makes an OOD-robust classifier?_

By parsing published results (as well as generating our own test results on
published models), we uncover several interesting commonalities between the
best OOD-robust models. See Figure 1 for a summary.

Figure 1: Large vision-language models are consistently more OOD-robust under
natural distribution shifts. Robustness of several ImageNet classifiers
discussed in our results. Robustness correlates with vision-language
contrastive pretraining with high-capacity image backbones trained on massive
datasets; robustness reliably fails to develop when any of these properties
are absent. Large batch sizes also appear to play a role, but the effect is
less consistent.

###  1.1 Setup

#### Defining OOD-robustness.

The scope of our analysis corresponds to models capable of inference on the
1000-class ImageNet validation set Deng et al. (2009).

There are numerous ways to formally define robustness to out-of-distribution
shifts. Some emphasize the difficulty of the novel dataset as a component of
the loss Ben-David et al. (2010); Lu et al. (2020), and others Recht et al.
(2019) focus on pure gaps in accuracy. This topic has also been studied under
related sub-fields such as domain generalization Wang et al. (2021), and has
found applications in several areas including robotics Körber et al. (2021).

In our study, for simplicity, we define an “OOD-robust classifier” as one that
fulfills the following properties:

  1. 1.

Attains a high-accuracy regime on the target task, and

  2. 2.

Maintains $\geq 90\%$ of its in-distribution accuracy under distribution shift
using, either zero-shot classification or linear probing (without feature re-
training).

In our study, we fix inference on ImageNet as the in-distribution task, and we
use a suite of distribution shift datasets designed for ImageNet as OOD
benchmarks.

#### Metrics of interest.

For inference on the ImageNet validation dataset, we define three broad
regimes: the _low_ regime covers models with accuracy $\leq 50\%$, the
_medium_ regime covers models with $50-75\%$ accuracy, and the _high_ accuracy
regime covers models with accuracy $\geq 75\%$. Note that Point #1 above in
our definition of OOD-robustness above automatically excludes the possibility
of a low- or medium-accuracy model being labeled OOD-robust. In any case, as
anticipated we found that no low- or medium-accuracy classifier achieves out-
of-distribution accuracy $\geq 90\%$.

We consider natural distribution shifts to be those generated by processes
such as occlusion, weather, a physical transformation of the object, or varied
representations of an object (such as sketches, clip-art, or silhouettes).
Similarly, we consider artificial distribution shifts to be those generated by
a machine or a mechanical process such as jpeg compression artifacts, blur,
noise, and algorithmically generated stylization effects. We do not consider
other distribution shifts such as adversarial robustness in this paper,
leaving this aspect to future work.

###  1.2 Core Findings

Our most important findings are shown in Figure 1. We see that OOD-robustness
is undergridded by multiple design and implementation decisions, including
both familiar features (massive dataset sizes, training time, model capacity)
but also more recent approaches such as large-batch contrastive pretraining
with a vision-language training objective.

As we discuss in depth in Sec. 4.5, we find that four key design decisions
reliably co-occur in OOD-robust models, and that the absence of any of these
four reliably causes the model to fail to achieve OOD-robustness. We further
show that one additional design decision, batch size, has a less predictable
(but persistent) effect.

##  2 Related Work

###  2.1 Natural Distribution Shifts

Concerns over methodology and generalization in ImageNet-trained classifiers
have led to the creation of new datasets as supplements to standard ImageNet
validation Beyer et al. (2020) Recht et al. (2019). Recent large-scale studies
Taori et al. (2020); Miller et al. (2021); Yadav & Bottou (2019); Lu et al.
(2020) have suggested that not only do standard ImageNet classifiers
generalize less well than we might expect, but they do so in patterned and
consistent ways. Our goal in this meta-review is to probe similar patterns in
better-performing (distributionally robust) classifiers.

###  2.2 Artificial Distribution Shifts

Training classifiers that are resilient to adversarial perturbations have been
shown to produce models less susceptible to spurious features Ilyas et al.
(2019). Various forms of artificial image perturbations have also been used
for data augmentation Cubuk et al. (2020). We note here that OOD-robust models
exhibit properties that are distinct from adversarial robustness, in that they
attempt to perform well against real-world inputs which are corrupted or
otherwise of low quality Hendrycks & Dietterich (2019).

###  2.3 Vision-Language Contrastive Pretraining

Contrastive pretraining of visual-language models has opened up exciting new
possibilities for designing and scaling robust models. Although precursors of
these models have been around for a while Frome et al. (2013); Li et al.
(2017), several success stories have emerged in the last twelve months.
OpenAI’s CLIP Radford et al. (2021) was the first vision-language model to
reach ImageNet accuracies comparable to SOTA supervised models. Moreover, CLIP
also exhibits a surprising degree of OOD-robustness, unlike standard CNN based
architectures. Subsequent models trained in a similar fashion, including ALIGN
Jia et al. (2021), and BASIC Pham et al. (2021), have also been able to
achieve CLIP-level performance. The cumulative effect has lead some
researchers to refer to them as ’more human’ Geirhos et al. (2021).
Unfortunately, the training details of many of these models, and the
associated datasets, remain private.

More recent work has experimented with CLIP-like objectives with smaller
public datasets, both using the original CLIP objective and variations
thereof. Meta’s SLIP models Mu et al. (2021) employ a multi-task learning
framework for combining self-supervised learning and CLIP pre-training. CLIP,
however, remains the only publicly available model which achieves high
accuracy on its target task, making apples-to-apples comparisons difficult.

##  3 Experimental Covariates

Inspired by previous work Miller et al. (2021), we selected a representative
sample of classifiers for diversity with respect to properties we originally
suspected might play a role in model robustness: number of parameters, noisy
label pretraining, vision-language contrastive pretraining, other forms of
contrastive pretraining, quantity of data, batch size, and choice of vision
backbone.

We group our findings into models for which both artificial and natural
distribution shifts are considered, and models where only natural distribution
shifts are considered. We do this because some of the models we used for
comparison are not publicly available and only report partial results.

Whenever models were publicly available, we re-ran tests using published
weights, rather than citing results from published papers; therefore, our
results sometimes differ from the published results. For instance, in the case
of CLIP-ViT-L-14, we found that certain tasks varied by as much as $6\%$ from
the published figures on OpenAI’s website. These are outliers, and generally
our metrics match the aforementioned published results.

##  4 Results and Discussion

Table 1: OOD-robust models share key properties. Our core findings on
robustness are divided into artificial and natural examples. We order our
results by accuracy on the main task, in descending order. Three models were
robust to natural distribution shifts – all of them used contrastive VL
pretraining on massive datasets but varied in most other respects, including
dataset used, image backbone, and hyperparameters. No model was robust to
artificial distribution shifts. Robust models are in bold. Key: Val denotes
validation accuracy on ImageNet-2012, NR denotes the average natural OOD-
robustness measured on our testbed datasets, AR denotes the artificial OOD-
robustness measured on our testbed datasets, VL denotes vision-language
pretraining.
Acc. Regimes | Model | Val | NR | AR | VL | Data | Backbone
---|---|---|---|---|---|---|---
High | EfficientNet-L2-NS | 88.32 | 80.71 | 81.88 |  ✗ |  300M | EfficientNet
FixResNext101-V2 | 86.36 | 79.48 | 65.59 |  ✗ |  1B | ResNet
SEER | 85.8 | 67.79 | N/A |  ✗ |  1B | RegNet
BASIC | 85.7 |  97.29 | N/A |  ✓ |  6.6B | ViT
ViT-L-16-384 | 85.15 | 66.17 | 70.86 |  ✗ |  300M | ViT
RN-152-SimCLR-FT | 81.04 | 57.63 | 62.8 |  ✗ |  $\leq$300M | ResNet
ALIGN | 76.4 |  98.19 | N/A |  ✓ |  1.8B | EfficientNet
CLIP-ViT-L-14 | 75.53 |  93.83 | 67.6 |  ✓ |  400M | ViT
Medium | Resnet-34 | 73.31 | 42.26 | 47.13 |  ✗ |  $\leq$300M | ResNet-34
CLIP-ViT-B-32 | 63.3 | 76.7 | 56.68 |  ✓ |  400M | ViT
OpenCLIP-ViT-B-32-L400M | 60.3 | 76.33 | 53.33 |  ✓ |  400M | ViT
CLIP-RN50 | 59.8 | 70.72 | 39.91 |  ✓ |  400M | ResNet
| AlexNet | 56.52 | 32.39 | 29.9 |  ✗ |  $\leq$300M | CNN
Low | SLIP-ViT-L-YFCC15M | 47.9 | 52.34 | 54.74 |  ✓ |  $\leq$300M | ViT
RN50-Adv-Smooth | 44.26 | 38.75 | 64.69 |  ✗ |  $\leq$300M | ResNet
SLIP-ViT-B-16-CC12M | 40.7 | 59.28 | 44.4 |  ✓ |  $\leq$300M | ViT
RN-Subsample | 36.7 | 27.25 | 25.86 |  ✗ |  $\leq$300M | ResNet
OpenCLIP-RN50-CC12M | 35.98 | 71.52 | 29.85 |  ✓ |  $\leq$300M | ResNet
| OpenCLIP-RN101-YFCC15M | 34.93 | 59.03 | 31.86 |  ✓ |  $\leq$300M | ResNet

Table 2 provides details of our meta-analysis. We observed several
commonalities among models that are OOD-robust, which we highlight as follows.

###  4.1 Vision-Language Contrastive Pretraining

Our findings corroborate those of Radford et al. (2021): vision-language
models are capable of achieving extremely high natural robustness (above
$90\%$), and in this respect, they appear to differ substantially from other
training objectives Taori et al. (2020). The effect is pronounced and
reliable, originating in the low-accuracy regime and persisting throughout
training. (See Table 2).

Some of the examples of this phenomenon are truly striking. For instance, our
experiments on OpenCLIP reveal that, after only 30 epochs of training,
OpenCLIP, trained on 8 GPUs, exhibits proportionate out-of-distribution
robustness within _ten_ percent of the fully-trained SEER model Goyal et al.
(2022), with $\sim 10B$ parameters and trained with a SwAV Caron et al. (2020)
objective on almost 500 GPUs.

###  4.2 Artificial Distribution Shifts

Reported results on robustness to artificial distribution shifts are somewhat
scarcer than those on natural distribution shifts. To the best of our
knowledge, none of the large-scale private models have reported results on
ImageNet-c or Stylized-ImageNet.

However, our own experiments (Table 3 in the appendix) seem to indicate that
artificial distribution robustness may be a much bigger challenge than natural
distribution robustness, since none of our models were OOD-robust to
artificial distribution shifts. This may have to do with the methodologies
used to generate these particular datasets, which include many samples that
have proven extremely challenging even for humans to classify. Geirhos et al.
(2019)

Our experiments also affirm that vision-language pretraining does not reliably
make a model more robust to artificial distribution shifts as it appears to do
with natural distribution shifts. This, too, follows the scaling laws
demonstrated by Taori et al. (2020) and Miller et al. (2021).

###  4.3 Model Architecture and Dataset Size

The results reported in the CLIP and ALIGN papers demonstrate that ViT-,
ResNet- and EfficientNet backbones can be used to train OOD-robust models.

Model capacity is crucial. CLIP experiments show us that lower-capacity models
such as ResNet-50 and ViT-B-32 fail to improve after the model is saturated.
The training curves of most of the larger vision models suggest that it is
very straightforward to know when such a limit has been reached Radford et al.
(2021). Precisely how large a model architecture is necessary remains an open
question, but the results shown in Radford et al. (2021) strongly suggest that
even larger models than ViT-L may prove beneficial.

A precise cutoff for the number of data samples required to train an OOD-
robust model remains elusive. The effect of dataset size on robustness of non-
VL models, which are more widely available, show robustness benefits for
models trained with at least 300M samples, but shows few marginal gains to
robustness above that threshold.

###  4.4 Other Parameters

Of the remaining covariates, the question of batch size is a matter of
practical concern. Most VL-models examined in our meta-review used a very
large batch size, including all models (of which we are aware) that belong to
the high-accuracy regime.

Our experiments with OpenCLIP (batch size 128) and SLIP (batch size 4096), the
best comparison pair available to us, show comparable OOD robustness for both
models. However, it is likely that neither model was trained to the point of
saturation, and it is possible that the model may learn harder examples more
effectively later in the training process. Tu et al. (2020)

###  4.5 Common Properties of Robust Models

While our meta-review is likely not comprehensive, we observe that the
following qualities consistently recur:

  1. 1.

Vision-language contrastive pretraining objective.

  2. 2.

$\geq$ 400m training samples with noisy labels.

  3. 3.

A high-capacity ViT or ResNet backbone, such as ViT/L or EfficientNet.

  4. 4.

Sufficiently many training epochs to reach the high accuracy regime

Our results show that among more than two hundred models, many evaluated for
the first time in our meta-review, no model achieved robustness without these
properties. Further, we have not encountered any model which fulfilled these
requirements and failed to achieve robustness.

Current best practices, therefore, may involve focusing on these four
properties as prerequisites. We further recommend carefully probing the effect
of batch size $>$ 4096 and dataset sizes between $100$M and $400$M when
designing future OOD-robust classifiers, as the effects of these may be
present but are not as consistently observed.

##  5 Conclusions

In this meta-review we identify several commonalities among distributionally-
robust models. This, combined with results from other recent studies, enables
us to suggest best practices to achieve OOD-robustness of models that are
trained up to the high accuracy regime.

We also make qualitative remark on key outstanding questions, such as the
precise role batch size and internet-scale datasets play in training robust
models.

Overall, we believe that continued attention to these questions will pave the
way towards achieving even superior models than those that are currently
available.

## References

  * Ben-David et al. (2010) Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan, J. W.  A theory of learning from different domains.  _Machine Learning_ , 79(1):151–175, May 2010\.
  * Beyer et al. (2020) Beyer, L., H’enaff, O. J., Kolesnikov, A., Zhai, X., and van den Oord, A.  Are we done with imagenet?  _ArXiv_ , abs/2006.07159, 2020.
  * Borji (2020) Borji, A.  Objectnet dataset: Reanalysis and correction.  _ArXiv_ , abs/2004.02042, 2020.
  * Caron et al. (2020) Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A.  Unsupervised learning of visual features by contrasting cluster assignments.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Cubuk et al. (2020) Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V.  Randaugment: Practical automated data augmentation with a reduced search space.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pp. 3008–3017, 2020.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 248–255, 2009.
  * Frome et al. (2013) Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M. A., and Mikolov, T.  Devise: A deep visual-semantic embedding model.  In Burges, C. J. C., Bottou, L., Welling, M., Ghahramani, Z., and Weinberger, K. Q. (eds.), _NeurIPS_ , volume 26, 2013.
  * Geirhos et al. (2019) Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W.  Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.  In _International Conference on Learning Representations_ , 2019.
  * Geirhos et al. (2021) Geirhos, R., Narayanappa, K., Mitzkus, B., Thieringer, T., Bethge, M., Wichmann, F. A., and Brendel, W.  Partial success in closing the gap between human and machine vision.  In _Advances in Neural Information Processing Systems 34_ , 2021\.
  * Goyal et al. (2022) Goyal, P., Duval, Q., Seessel, I., Caron, M., Misra, I., Sagun, L., Joulin, A., and Bojanowski, P.  Vision models are more robust and fair when pretrained on uncurated images without supervision.  _ArXiv_ , abs/2202.08360, 2022.
  * Hendrycks & Dietterich (2019) Hendrycks, D. and Dietterich, T. G.  Benchmarking neural network robustness to common corruptions and perturbations.  _ArXiv_ , abs/1903.12261, 2019.
  * Hendrycks et al. (2021a) Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  _ICCV_ , 2021a.
  * Hendrycks et al. (2021b) Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D.  Natural adversarial examples.  _CVPR_ , 2021b.
  * Ilharco et al. (2021) Ilharco, G., Wortsman, M., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L.  Openclip, July 2021.  URL https://doi.org/10.5281/zenodo.5143773.
  * Ilyas et al. (2019) Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A.  Adversarial examples are not bugs, they are features.  _ArXiv_ , abs/1905.02175, 2019.
  * Jia et al. (2021) Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y.-H., Li, Z., and Duerig, T.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _ICML_ , 2021.
  * Körber et al. (2021) Körber, M., Lange, J., Rediske, S., Steinmann, S., and Glück, R.  Comparing popular simulation environments in the scope of robotics and reinforcement learning.  _ArXiv_ , abs/2103.04616, 2021.
  * Li et al. (2017) Li, A., Jabri, A., Joulin, A., and van der Maaten, L.  Learning visual n-grams from web data.  _ICCV_ , pp. 4193–4202, 2017.
  * Lu et al. (2020) Lu, S., Nott, B., Olson, A., Todeschini, A., Vahabi, H., Carmon, Y., and Schmidt, L.  Harder or different? a closer look at distribution shift in dataset reproduction.  _ICML Workshop on Uncertainty and Robustness in Deep Learning_ , 2020\.
  * Miller et al. (2021) Miller, J., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., and Schmidt, L.  Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization.  _ArXiv_ , abs/2107.04649, 2021.
  * Mu et al. (2021) Mu, N., Kirillov, A., Wagner, D., and Xie, S.  Slip: Self-supervision meets language-image pre-training.  _arXiv preprint arXiv:2112.12750_ , 2021.
  * Pham et al. (2021) Pham, H., Dai, Z., Ghiasi, G., Liu, H., Yu, A. W., Luong, M.-T., Tan, M., and Le, Q. V.  Combined scaling for zero-shot transfer learning.  _ArXiv_ , abs/2111.10050, 2021.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Recht et al. (2019) Recht, B., Roelofs, R., Schmidt, L., and Shankar, V.  Do imagenet classifiers generalize to imagenet?  _ArXiv_ , abs/1902.10811, 2019.
  * Taori et al. (2020) Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L.  Measuring robustness to natural distribution shifts in image classification.  _ArXiv_ , abs/2007.00644, 2020.
  * Tu et al. (2020) Tu, L., Lalwani, G., Gella, S., and He, H.  An empirical study on robustness to spurious correlations using pre-trained language models.  _Transactions Assoc. Comp. Linguistics_ , 8:621–633, 2020\.
  * Wang et al. (2019) Wang, H., Ge, S., Xing, E. P., and Lipton, Z. C.  Learning robust global representations by penalizing local predictive power.  In _NeurIPS_ , 2019.
  * Wang et al. (2021) Wang, J., Lan, C., Liu, C., Ouyang, Y., and Qin, T.  Generalizing to unseen domains: A survey on domain generalization.  In _IJCAI_ , 2021.
  * Xie et al. (2019) Xie, Q., Hovy, E. H., Luong, M., and Le, Q. V.  Self-training with noisy student improves imagenet classification.  _Arxiv_ , abs/1911.04252, 2019.
  * Yadav & Bottou (2019) Yadav, C. and Bottou, L.  Cold case: The lost MNIST digits.  _CoRR_ , abs/1905.10498, 2019.

##  Appendix A Appendix

###  A.1 Experimental Design

Because many potential points of comparison exist in a large-scale study, our
methodology focuses on a relatively narrow set of variables. For this reason,
our study largely follows an existing methodology of Taori _et al._ and Miller
_et al._ Taori et al. (2020); Miller et al. (2021). In these papers, over two
hundred models are examined under very similar conditions, including CNNs,
visual transformers, self-supervised models, logistic regression, nearest
neighbors, and kernel machines, making it an ideal starting point for
investigating questions of robustness.

We select for our experiments an extended subset of the models examined in the
above papers, extending it in three important ways. First, we evaluate the
robustness of several new models which were not tested in the paper. Second,
we present an expanded set of artificial distribution shift results for models
which were in the original study, but for which artificial distribution
robustness results were previously unavailable. Third, we collect and compare
robustness results which have been published, with a special attention to
contrastive models.

A complete list of the representative sample of models considered in this
paper, along with a breakdown of the key properties we compare, can be seen in
1.

The pretrained SLIP weights used in this paper can be can be accessed via
their repository. The OpenCLIP weights and the framework used to train them
can be found in Ilharco et al. (2021).

The provenance of the remaining models, including links to pretrained weights,
can be referenced in Taori et al. (2020).

We fully acknowledge that opinions on what constitutes a ’representative
sample’ of models may differ, and we encourage the curious reader to explore
the interactive testbed in Taori et al. (2020) and Miller et al. (2021), where
hundreds of additional model results can be examined. Furthermore, since both
the models and the testbed are publicly available, the reader is also free to
evaluate and compare additional models beyond what we present here.

Wherever possible, we use the ImageNet-testbed benchmark suite from that same
paper, in order to provide a consistent grounds for comparison between them
models. Where this is not possible, we run inference using the same label set
as provided in that benchmark.

We quantify robustness to distribution shift as the ratio of average OOD-
robustness to in-distribution robustness:

$\frac{\text{avg}(OOD)}{ID}$

For more information on the datasets tested here, please refer to A.2.

Wherever possible, we consider classifier accuracy as a relative, rather than
absolute, measurement – we are interested in consistency above all. However,
we also provide raw accuracy scores in the appendix for reference.

###  A.2 Additional detail on datasets

Our selection of natural distribution datasets is ImageNet-a, Hendrycks et al.
(2021b) ImageNet-r, Hendrycks et al. (2021a) ImageNet-s, Wang et al. (2019)
ImageNet-V2, Recht et al. (2019) and ObjectNet Borji (2020). We use the
matched-frequency-format-val version of ImageNet-V2, and the 1.0-beta version
of ObjectNet. We also provide a small sample of our comparison datasets for
randomly selected classes in ImageNet.

Figure 2: Examples from Datasets used for our Robustness Tests.Two classes
from the ImageNet validation set (upper left) and their distribution-shifted
counterparts. Blank spaces indicate missing classes. Artificial distribution
shifts show the model perturbations of images it has already classified,
whereas natural distribution shifts show the model un-perturbed but novel
images.

We select ImageNet-C-in-memory-avg Hendrycks & Dietterich (2019), ImageNet-C-
on-disk-avg, and Stylized-ImageNet Geirhos et al. (2019) as test-beds for our
artificial distribution robustness experiments. ImageNet-C is a set of common
visual corruptions (blur, noise) applied to the ImageNet test-set. The former
is computed by the GPU at test time, the latter evaluates a fixed set of
JPEGs. Our experiments show that 10-percent ImageNet-C subset results are very
good approximations of the full dataset – we use them on all models except for
the SLIP models, which are incompatible with CLIP’s framework, and therefore
with the testbed. Following Taori et al. (2020); Miller et al. (2021), for all
models except SLIP, we evaluate ImageNet-C both in-memory and on-disk,
sometimes finding significant discrepancies between the two. Our averaging
method treats all ImageNet-C in-memory corruptions as a single dataset, and
all ImageNet-C on-disk corruptions as another single dataset.

Stylized-ImageNet was created by applying AdaIN style transfer to ImageNet
images. These generated images range from highly recognizable to extremely
abstract. It is the most challenging dataset we evaluate in this study, with
the best classifier achieving a little over $55\%$ accuracy.

We observe that there are important differences in the two distribution shifts
that may partly account for our findings on them; artificial distribution
shifts show the model perturbations of images it has already classified,
whereas natural distribution shifts show the model un-perturbed but novel
images.

###  A.3 OOD robustness scales disproportionately with in-distribution
robustness

Table 2: Robustness trends emerge in the low-accuracy regime. Comparing VL and
non-VL models in each accuracy regime, we can see that on average, OOD-
robustness trends persist as the model is trained to higher accuracy. We also
see a stark contrast in natural and artificial distribution robustness – under
artificial shifts, VL models perform no better than non-VL models.
Model Class | Acc Regime | Nat. Range | Artif. Range
---|---|---|---
Non-VL | Low | 27-39 | 26-65
VL | Low | 52-72 | 30-51
Non-VL | Medium | 32-42 | 30-47
VL | Medium | 71-77 | 40-57
Non-VL | High | 57-81 | 63-79
VL | High | 95-97 | 68

Recent studies have suggested that models in the low and medium accuracy
regimes appear to follow the scaling laws observed by Taori et al Taori et al.
(2020) and Miller et al Miller et al. (2021). Our experiments confirm these
observations on a wider range of vision-language models and over a wider range
of datasets. This is important, given that one of the aims of our work is to
better predict the robustness of unseen classifiers. The above-cited scaling
law estimates allow us to make a better guess at an early checkpoint whether a
model has the potential to achieve robustness.

Our experiments confirm that models which are eventually trained to robustness
perform significantly better on the metric in the low and medium accuracy
regimes. We also note that natural OOD-robustness scales reliably with model
accuracy in vision-language models, and that it occupies a narrower range,
compared to non-vision-language classifiers. See Table 1 for detailed results.

This appearance of uniformity, however, conceals another interesting
observation – in the low and medium accuracy regimes, vision-language models
are more robust on certain datasets (ImageNet-V2, ImageNet-r) than others
(ImageNet-a, ImageNet-s). This effect is less pronounced in non-VL models, and
may be caused by informative contrastive counterexamples the model has yet to
encounter. Tu et al. (2020)

###  A.4 Noisy Student Training

One near-exception to the domination of vision-language contrastive models can
be found in the area of knowledge distillation. Noisy Student Training Xie et
al. (2019) extends the idea of self-training and distillation with the use of
equal-or-larger student models and noise added to the student during learning.
An EfficientNet teacher generates pseudo-labels on 300M unlabeled images. A
larger EfficientNet student model then trains on the combination of labeled
and pseudo labeled images.

The effects on robustness are similar, but less pronounced and less
consistent, to those seen in VL training. Specifically, this approach is the
most robust, both in proportionate and in absolute terms, at handling
artificial distribution shifts.

In fact, there appears to be a curious counterbalance between the two training
approaches, as can be seen in Table 3. For all of the tasks we evaluated,
either CLIP ViT-L or Noisy Student were the most robust, but with the
exception of ImageNet-a, there is a marked performance discrepancy between the
models on every single task.

Although speculative explanations immediately suggest themselves, we leave the
explanation for this curious inverse relationship to future work.

We do note that data augmentation may have helped Noisy Student’s robustness
to artificial distribution shifts, and it is possible that a VL model trained
in a similar manner to CLIP, but with data augmentation added to the training
procedure, would show improvements in this area.

Table 3: Raw accuracy scores on the datasets. The above table collects the raw
accuracy scores for each model on each dataset in the testbed, including only
the models we were able to test directly. We also report the proportional 95%
confidence interval average for all models on each dataset, rounded up to the
nearest .005. CLIP and Noisy Student, the most OOD-robust VL and non-VL models
we tested directly, exhibit largely complementary behavior on per-task
accuracy comparison. CLIP performs better on natural distribution shifts,
while Noisy Student performs better on artificial ones. COLUMNS FROM LEFT TO
RIGHT: ImageNet-2012, ImageNet-a, ImageNet-r, ImageNet-sketch, ImageNet-V2,
ObjectNet, ImageNet-C (in memory, 10% subset), ImageNet-C (on disk, 10%
subset), Stylized ImageNet
Model | IN-VAL | IN-A | IN-R | IN-S | IN-V2 | ObjNet | IN-c-mem | IN-c-disk | Style-IN
---|---|---|---|---|---|---|---|---|---
EfficientNet-L2-NS | 88.32 | 84.85 | 74.67 | 47.64 | 80.85 | 68.45 | 83.92 | 76.05 | 56.99
FixResNext101-V2 | 86.36 | 68.41 | 79.9 | 59.14 | 77.94 | 57.85 | 72.63 | 59.63 | 37.66
ViT-L-16-384 | 85.15 | 53.85 | 54.75 | 43.28 | 75.37 | 54.49 | 75.27 | 65.74 | 40.03
RN-152-SimCLR-FT | 81.04 | 33.85 | 47.3 | 35.06 | 70.35 | 46.94 | 63.8 | 57.42 | 31.49
CLIP-ViT-L-14 | 76.2 | 71.15 | 87.74 | 59.57 | 75.87 | 67.7 | 60.98 | 58.1 | 35.46
ResNet-34 | 73.31 | 1.87 | 36.07 | 23.47 | 60.87 | 32.62 | 42.37 | 39.6 | 21.69
CLIP-ViT-B-32 | 63.3 | 31.38 | 69.25 | 42.37 | 55.77 | 43.96 | 44.26 | 42.13 | 21.25
OpenCLIP-ViT-B-32-L400M | 60.3 | 19.63 | 70.73 | 46.42 | 52.2 | 41.15 | 39.45 | 36.97 | 20.08
CLIP-RN50 | 59.8 | 22.8 | 60.46 | 35.38 | 52.53 | 40.28 | 31.48 | 27.43 | 12.71
Alexnet | 56.52 | 1.77 | 21.77 | 10.71 | 43.43 | 13.86 | 23.1 | 20.18 | 7.41
SLIP-ViT-L-YFCC15M | 47.9 | 23.17 | 33.05 | 19.01 | 42.09 | 24.05 | 33.41 | 31.88 | 14.9
RN50-Adv-Smooth | 44.26 | 2.06 | 27.07 | 11.53 | 33.23 | 11.86 | 39.45 | 35.91 | 10.54
SLIP-ViT-B-16-CC12M | 40.7 | 5.5 | 37.68 | 29.2 | 34.58 | 20.17 | 21.91 | 21.76 | 10.7
RN-Subsample | 36.7 | 1.41 | 11.32 | 3.17 | 26.94 | 7.17 | 12.75 | 12.72 | 2.98
OpenCLIP-RN50-CC12M | 35.98 | 7.57 | 44.64 | 23.46 | 30.44 | 22.55 | 13.54 | 11.56 | 7.11
OpenCLIP-RN101-YFCC15M | 34.93 | 15.68 | 26.28 | 8.89 | 33.52 | 18.73 | 14.63 | 12.91 | 5.85
U.B. Confidence Intvl (+/-) | 0.005 | 0.015 | 0.005 | 0.005 | 0.01 | 0.005 | 0.005 | 0.005 | 0.005


# Use of Bayesian Nonparametric methods for Estimating the Measurements in
High Clutter.

Bahman Moraffah†, Christ Richmond†, Raha Moraffah‡, and Antonia Papandreou-
Suppappola†
†Department of Electrical, Computer, and Energy, Arizona State University,
Tempe AZ
‡ School of Computing, Informatics, and Decision Systems Engineering, Arizona
State University, Tempe AZ
{bahman.moraffah, christ.richmond, raha.moraffah<EMAIL_ADDRESS>This
work was supported in part by Grant AFOSR FA9550-17-1-0100.

###### Abstract

Robust tracking of a target in a clutter environment is an important and
challenging task. In recent years, the nearest neighbor methods and
probabilistic data association filters were proposed. However, the performance
of these methods diminishes as number of measurements increases. In this
paper, we propose a robust generative approach to effectively model multiple
sensor measurements for tracking a moving target in an environment with high
clutter. We assume a time-dependent number of measurements that include sensor
observations with unknown origin, some of which may only contain clutter with
no additional information. We robustly and accurately estimate the trajectory
of the moving target in high clutter environment with unknown number of
clutters by employing Bayesian nonparametric modeling. In particular, we
employ a class of joint Bayesian nonparametric models to construct the joint
prior distribution of target and clutter measurements such that the
conditional distributions follow a Dirichlet process. The marginalized
Dirichlet process prior of the target measurements is then used in a Bayesian
tracker to estimate the dynamically-varying target state. We show through
experiments that the tracking performance and effectiveness of our proposed
framework are increased by suppressing high clutter measurements. In addition,
we show that our proposed method outperforms existing methods such as nearest
neighbor and probability data association filters.

##  I Introduction

Target tracking in a high clutter environment is very challenging as sensor
measurements , more often than not, contain detections from false targets [1].
Time-dependent number of measurements that include both clutter and true
sensor observations with unknown origin makes the tracking in the clutter
environment difficult. To solve this problem a set of assumptions is made: 1)
true measurements from the target are present with some probability of
detection, and 2) number and location of clutter measurements are unknown and
random.

For accurate target parameter estimates, tracking algorithms must only
incorporate target generated measurements. To this end, various methods have
been considered to solve the problem of tracking in clutter. One of the widely
used methods is the strongest-neighbor and nearest-neighbor (NN) filters [2].
In this method, a Gaussian object motion considered and measurements that are
statistically closest to the predicted measurements are from the object and
the rest are considered clutters. It is shown that these filters only process
one measurement, the one with the highest probability that is generated by the
target [3]; however, the performance of these methods diminish with increase
in the probability of false alarm rate.

In [4, 1], the probabilistic data association (PDA) method was used to
validate multiple measurements according to their probability of target
origin. Similar to nearest-neighbor filters, PDA filters assume that the
object motion obeys linear Gaussian statistics. The PDA filter assumes that
all non-object originated measurements are clutter which is uniformly
distributed in the space and Poisson distributed in time. Furthermore, several
variations of the PDA method are proposed, such as the filtered gate structure
method, the interactive-multiple model PDA [5], as well as other data
association approaches, such as the Viterbi and fuzzy data association, were
also considered [6]. Note, however, that PDA type methods can become
computationally intensive as the number of measurements increases.

Recently, Bayesian nonparametric methods [7, 8] have been used in target
tracking to solve problems such as multiple models [9], maneuvering [10],
dependent measurements [11, 12, 13], waveform design [14], and measurement-to-
target associations in multiple target tracking [15, 16, 17]. In this paper,
we propose the measurement estimation for tracking in high clutter (METRIC)
Bayesian nonparametric approach to track a target in high clutter. Our
proposed model does not suffer from the issues that aforementioned methods
face and is computationally inexpensive. In particular, the joint prior
distribution of measurements with different origins is modeled using Dirichlet
processes (DPs). The marginal DP prior of the target measurements is then
integrated into Bayesian filtering methods to track the target.

The rest of the paper is organized as follows. In Section II, we briefly
review Bayesian nonprahmteric methods. In particular, we study the Dirichlet
process which is used in the rest of this paper. Section III discusses the
joint modeling of the measurements through a joint Dirichlet process. We
integrate the proposed model into a Bayesian tracker to robustly and
accurately track the target trajectory. Section IV demonstrates the advantage
of the proposed model over the existing methods.

##  II Background: Bayesian Nonparametric Modeling

With the surge of Monte Carlo (MC) methods, Bayesian nonparametric models have
been ubiquitous in probabilistic modeling of the observations to avoid the
limitations in parametric modeling such as model misfit. The family of
infinite dimension of random measures has been used as a substitute of finite
mixture model estimate the density. In this section, we briefly describe one
of the main stochastic processes that play an integral role in our
investigation.

###  II-A Dirichlet Process

Dirichlet process (DP) is the most well-studied nonparametric model that
defines a prior over the space of probability distributions [18]. Dirichlet
process can be considered as the infinite generalization of the Dirichlet
distribution, that is

$\lim\limits_{K\to\infty}\sum\limits_{k=1}^{K}\pi_{k}\delta_{\theta_{k}}$ (1)

for mixing distribution $\pi=(\pi_{1},\dots,\pi_{K})$ and $\delta_{\theta}$ is
a point mass centered at the parameter $\theta\in\Theta$. It can be shown
equation (1) results in a well-defined Dirichlet process [19, 20]. Dirichlet
process is discrete with probability one and thus is mainly used as a prior on
the parameters in the space of parameters $\Theta$. A Dirichlet process with
concentration parameter $\alpha$ and base distribution $H$ is denoted by
$\text{DP}(\alpha,H)$. A draw $G\sim\text{DP}(\alpha,H)$ is a probability
random measure and is represented as [19]

$G(\cdot)=\sum\limits_{k=1}^{\infty}\pi_{k}\delta_{\theta_{k}}(\cdot)$ (2)

for mixing probability $\pi\sim\text{GEM}(\alpha)$ and the sequence of
$\theta_{k}$ is drawn independently and identically from H, i.e.,
$\theta_{k}\sim H$. Note that since $G$ provides a distribution on $\Theta$,
$\theta_{k}$ takes on values in $\Theta$. The GEM($\alpha$) is defined as

$\displaystyle\pi\sim$ $\displaystyle\text{ GEM}(\alpha)\Longleftrightarrow$
(3) $\displaystyle v_{k}\sim\text{Beta}(1,\alpha),$ $\displaystyle\hskip
2.84544pt\pi_{k}=v_{k}\prod\limits_{j<k}(1-v_{k})\hskip 8.5359pt\text{for
}\hskip 2.84544pt{k=1,2,\dots}.$

This construction is known as stick-breaking construction of a Dirichlet
process. Let $\Xi=\\{\theta_{1},\dots,\theta_{N}\\}$ be the independent draws
from $G$.

###  II-B Inference

The posterior distribution of $G$ given the observations
$\Xi=\\{\theta_{1},\dots,\theta_{N}\\}$ is a Dirichlet process with
concentration parameter $\alpha_{\text{post}}$ and base probability measure
$H_{\text{post}}$ ,

$G|\Xi\sim\text{DP}(\alpha_{\text{post}},H_{\text{post}})$ (4)

where $\alpha_{\text{post}}=\alpha+N$ and

$H_{\text{post}}=\frac{N}{\alpha+N}\frac{\sum_{k=1}^{N}\delta_{\theta_{k}}}{N}+\frac{\alpha}{\alpha+N}H.$
(5)

where $\delta_{\theta}$ is a degenerate measure concentrated at $\theta$.

Definition of Dirichlet process implies that the posterior mean is
$H_{\text{post}}$ which is a convex combination of prior distribution and the
empirical distribution. Note that if $N\gg\alpha$ the second term in (5)
vanishes and the posterior base measure converges to the empirical
distribution and therefore dominates the posterior distribution. This fact
implies the consistency of the posterior distribution.

The predictive distribution $\mathbb{P}(\theta_{N+1}\in A|\Xi)$ for some
measurable event $A\subset\Theta$ is computed with marginalizing out $G$ as

$\displaystyle\begin{split}\mathbb{P}(\theta_{N+1}\in
A|\Xi)&=\mathbb{E}(\theta_{N+1}\mathbbm{1}_{\theta_{N+1}}(A)|\Xi)\\\
&=\frac{\sum_{k=1}^{N}\delta_{\theta_{k}}(A)+\alpha
H(A)}{\alpha+N}\end{split}$ (6)

where $\mathbbm{1}_{\theta_{N+1}}(A)$ is the indicator function, i.e.,
$\mathbbm{1}_{\theta_{N+1}}(A)=1$ if $\theta_{N+1}\in A$ and
$\mathbbm{1}_{\theta_{N+1}}(A)=0$ if $\theta_{N+1}\notin A$. One can observe
that drawing from the predictive distribution of $\theta_{N+1}$ given $\Xi$ is
equivalent to draw $\theta_{N+1}\sim H_{\text{post}}$. Discreteness of the
predictive distribution (6) guarantees the repeat of draws. Let
$n_{k}=\\#\\{\theta\in\Xi:\theta=\theta_{k}\\}$, we can re-write (6) as

$\displaystyle\theta_{N+1}|\Xi\sim\sum_{k=1}^{K}\frac{n_{k}}{\alpha+N}\delta_{\theta^{*}_{k}}+\frac{\alpha}{\alpha+N}H$
(7)

where $\Xi^{*}=\\{\theta^{*}_{1},\dots,\theta^{*}_{K}\\}\subset\Xi$ is the set
of unique parameters in $\Xi$. Induced clustering distribution from
distribution (7) is known as Chinese restaurant process (CRP). Intuitively
speaking, a cluster with parameter $\theta_{N+1}$ is either identical to one
of the elements of $\Xi^{*}$ with probability proportional to the popularity
of each parameter $n_{k}$ or forms a new cluster, i.e., $\theta_{N+1}\sim H$
with probability proportional to $\alpha$. Moreover, it is shown that given
$N$ points, the expected number of cluster is $\alpha\log(N)$ [21].

##  III Bayesian Nonparametric Prior for Joint Measurements

In Section III-A, we discuss the problem in hand. Our proposed solution
through Bayesian nonparametric modeling is studied in Section III-B. We
introduce a joint Dirichlet process prior to model the measurements parameters
as well the clutter. We, then in Section III-C, integrate our proposed method
into a Bayesian tracker to efficiently estimate the target trajectory.

###  III-A Object Tracking Modeling in Clutter

We consider the problem of tracking a target in an environment with high
clutter, given multiple measurements. We aim to track a single target in
clutter with unknown number of clutters. We assume that the unknown target
state parameter state vector $\mathbf{x}_{k}$ at time step $k$ is given by
$\mathbf{x}_{k}$ = $f(\mathbf{x}_{k-1})+\mathbf{n}_{k}$, where
$f(\mathbf{x}_{k})$ is the state transition function and $\mathbf{n}_{k}$ is
the transition state error process. Note that the normality of the target
motion is not required. The $n$th measurement vector $\mathbf{z}_{n,k}$, $n$ =
$1,\ldots,N_{k}$, is

$\displaystyle\mathbf{z}_{n,k}=\left\\{\begin{array}[]{ll}\mathbf{z}^{(\text{t})}_{n,k}=h(\mathbf{x}_{k})+\mathbf{v}_{n,k},&\text{Target
measurement}\\\\[14.45377pt] \mathbf{z}^{(\text{c})}_{n,k},&\text{Clutter
measurement}\end{array}\right.$ (10)

where $\mathbf{z}^{(\text{t})}_{n,k}$ and $\mathbf{z}^{(\text{c})}_{n,k}$
indicate the target and clutter measurements, respectively. The relationship
between the measurement $\mathbf{z}^{(\text{t})}_{n,k}$ and the target state
is given by $h(\mathbf{x}_{k})$ and $\mathbf{v}_{n,k}$ is the measurement
noise process.

Initialize target state: $\mathbf{x}_{0}$

for $k=1:K$ do

Predict $p\Big{(}\mathbf{x}_{k}\mid\mathbf{Z}^{(\text{t})}_{k-1}\Big{)}$ using
$p\Big{(}\mathbf{x}_{k}\mid\mathbf{Z}^{(\text{t})}_{k-1}\Big{)}$ =
$\displaystyle\int
p(\mathbf{x}_{k}\mid\mathbf{x}_{k-1})\;p\Big{(}\mathbf{x}_{k-1}\mid\mathbf{Z}^{(\text{t})}_{k-1}\Big{)}d\mathbf{x}_{k-1}$

Input measurements $\\{\mathbf{z}_{1,k},\ \ldots,\ \mathbf{z}_{N_{k},k}\\}$

Draw clutter prior $G^{(\text{c})}_{k}$ from $\text{DP}(\alpha_{c},H_{c})$

Use clutter prior to draw $\Theta_{k}\mid
G^{(\text{c})}_{k}\overset{\text{iid}}{\sim}{G^{(\text{c})}_{k}}$

Draw target prior $G^{(\text{t})}_{k}$ from
$G^{(\text{t})}_{k}\mid\Theta_{k}\sim\text{DP}\Big{(}\alpha_{t},H_{t}+\sum_{i=1}^{N_{k}}\delta_{\theta_{i,k}}\Big{)}$

Use target prior to draw $W_{k}\mid
G^{(\text{t})}_{k},\Theta_{k}\overset{\text{iid}}{\sim}{G^{(\text{t})}_{k}}$

For target measurements, draw $\mathbf{z}^{(\text{t})}_{n,k}$, using Equation
(14)

For clutter measurements, draw $\mathbf{z}^{(\text{c})}_{n,k}$, using Equation
(13)

Cluster into target measurements $\mathbf{Z}^{(\text{t})}_{k}$ with
cardinality $M^{(\text{t})}_{k}$

Cluster into clutter measurements $\mathbf{Z}^{(\text{c})}_{k}$ with
cardinality $M^{(\text{c})}_{k}$

Compute likelihood ratio
$L\Big{(}\mathbf{Z}^{(\text{t})}_{k};W_{k},\mathbf{x}_{k}\Big{)}$ using
Equation (15)

Compute and return the posterior density using
$p\Big{(}\mathbf{Z}^{(\text{t})}_{k}\mid\mathbf{x}_{k},W_{k}\Big{)}$

Update $p(\mathbf{x}_{k}\mid\mathbf{Z}^{(\text{t})}_{k})$ using
$p\Big{(}\mathbf{x}_{k}\mid\mathbf{Z}^{(\text{t})}_{k}\Big{)}\propto
p\Big{(}\mathbf{Z}^{(\text{t})}_{k}\mid
W_{k},\mathbf{x}_{k}\Big{)}\,p\Big{(}\mathbf{x}_{k}\mid\mathbf{Z}^{(\text{t})}_{k-1}\Big{)}$

end for

Algorithm 1 METRIC-Bayes: Tracking in high clutter using jointly DPs prior.

Given the measurements $\mathbf{z}_{n,k}$, $n$ = $1,\ldots,N_{k}$, we can
obtain the target state by estimating the posterior density
$p\big{(}\mathbf{x}_{k}\mid\mathbf{z}_{1,k},\ \ldots,\
\mathbf{z}_{N_{k},k}\big{)}$. However, if all available measurements are used,
the presence of clutter deteriorates the tracking performance. In order to
improve the performance, we propose the METRIC-Bayes approach, as described in
Algorithm 1. In what follows, we discuss this framework in detail.

###  III-B Bayesian Nonparametric Prior

In this section, we discuss the framework summarized in Algorithm 1 in
details. Our framework models the marginal distributions of the joint prior as
two conditionally independent DPs. Note that in this model, no prior
information on the measurements is required. Due to the joint measurement
distribution factorization
$p\big{(}\mathbf{z}^{(\text{t})},\mathbf{z}^{(\text{c})}\big{)}=p\big{(}\mathbf{z}^{(\text{t})}|\mathbf{z}^{(\text{c})}\big{)}p\big{(}\mathbf{z}^{(\text{c})}\big{)}$,
we set two conditionally independent Dirichlet processes on the parameters of
$\mathbf{z}^{(\text{c})}$ and
$\mathbf{z}^{(\text{t})}|\mathbf{z}^{(\text{c})}$. Therefore, we may model the
clutter prior independently as
$G^{(\text{c})}_{k}\sim\text{DP}(\alpha_{c},H_{c})$ where DP is a Dirichlet
process with scale parameter $\alpha_{c}$ and base distribution $H_{c}$. To
model the target prior, we first draw the sample set $\Theta_{k}$ =
$\\{\theta_{1,k}\ldots,\theta_{N_{k},k}\\}$ from the clutter prior as
$\Theta_{k}\mid
G^{(\text{c})}_{k}\overset{\text{iid}}{\sim}G^{(\text{c})}_{k}$ and then given
$\Theta_{k}$, the target prior is defined as

$\displaystyle
G^{(\text{t})}_{k}\mid\Theta_{k}\sim\text{DP}\Big{(}\alpha_{t},H_{t}+\sum_{n=1}^{N_{k}}\delta_{\theta_{n,k}}\Big{)},$
(11)

where $\delta(\theta_{n,k})$ is the Dirac delta concentrated at
$\theta_{n,k}$. The set $W_{k}$ = $\\{w_{1,k}\ldots,w_{N_{k},k}\\}$ is drawn
from according to the Equation 11 as $W_{k}\mid
G^{(\text{t})}_{k}\overset{\text{iid}}{\sim}{G^{(\text{t})}_{k}}$. Using sets
$\Theta_{k}$ and $W_{k}$, we can then draw the target and clutter
measurements, respectively, as

$\displaystyle\mathbf{z}^{(\text{t})}_{n,k}\mid W_{k},\Theta_{k}\ \sim\
F_{t}(\cdot\mid W_{k}),\;n=1,\ldots,N_{k}$ (12)
$\displaystyle\mathbf{z}^{(\text{c})}_{n,k}\mid W_{k},\Theta_{k}\ \sim\
F_{c}(\cdot\mid\Theta_{k}),\;n=1,\ldots,N_{k}.$ (13)

For some distributions $F_{t}$ and $F_{c}$. The generative model describing
this process is summarized as

$\displaystyle\begin{split}&\Theta_{k}\mid
G^{(\text{c})}_{k}\overset{\text{iid}}{\sim}G^{(\text{c})}_{k}\\\
&G^{(\text{t})}_{k}\mid\Theta_{k}\sim\text{DP}\Big{(}\alpha_{t},H_{t}+\sum_{n=1}^{N_{k}}\delta_{\theta_{n,k}}\Big{)}\\\
&\mathbf{z}^{(\text{t})}_{n,k}\mid W_{k},\Theta_{k}\ \sim\ F_{t}(\cdot\mid
W_{k}),\;n=1,\ldots,N_{k}\\\\[7.22743pt] &\mathbf{z}^{(\text{c})}_{n,k}\mid
W_{k},\Theta_{k}\ \sim\
F_{c}(\cdot\mid\Theta_{k}),\;n=1,\ldots,N_{k}.\end{split}$ (14)

###  III-C Bayesian Tracker

Thus far, we modeled the joint distribution of measurements through the joint
Dirichlet process. However, in order to get an accurate and efficient
estimate, one needs to incorporate true measurements into the Bayesian tracker
as follows. We cluster the target measurements into set
$\mathbf{Z}^{(\text{t})}_{k}$ with cardinality $M^{(\text{t})}_{k}$ and the
clutter measurements into set $\mathbf{Z}^{(\text{c})}_{k}$ with cardinality
$M^{(\text{c})}_{k}$, where $M^{(\text{t})}_{k}+M^{(\text{c})}_{k}=N_{k}$. The
likelihood ratio is thus computed as

$\displaystyle
L\Big{(}\mathbf{Z}^{(\text{t})}_{k};W_{k},\mathbf{x}_{k}\Big{)}=\frac{\displaystyle\prod_{m=1}^{N_{k}}p\Big{(}\mathbf{z}_{m,k}^{(t)}\mid\mathbf{x}_{k};\rm{target\
present}\Big{)}}{\displaystyle\prod_{m=1}^{N_{k}}p\Big{(}\mathbf{z}_{m,k}^{(c)}\mid\mathbf{x}_{k};\rm{target\
absent}\Big{)}}.$ (15)

Equation (15) is then incorporated into a Bayesian tracker to estimate the
target state as

$\displaystyle
p\Big{(}\mathbf{x}_{k}\mid\mathbf{Z}^{(\text{t})}_{k}\Big{)}\propto
p\Big{(}\mathbf{Z}^{(\text{t})}_{k}\mid
W_{k},\mathbf{x}_{k}\Big{)}\,p\Big{(}\mathbf{x}_{k}\mid\mathbf{Z}^{(\text{t})}_{k-1}\Big{)}.$
(16)

Joint Bayesian nonparametric prior allows us to effectively model the
measurements at every time step without prior information. Any knowledge about
the tracking scene or clutter may be included in the base measures
$([\alpha_{c},H_{c}],[\alpha_{t},H_{t}])$ of the joint Dirichlet process
prior. A Markov chain Monte Carlo method may be employed to sample from the
posterior. This model can be easily generalized to track multiple objects by
incorporating it into multiple object tracking techniques e.g., integrating
dependent priors on the states introduced in [15, 16] into the METRIC-Bayes
framework.

##  IV Expriments

In this section, we demonstrate the performance of the proposed method and
compare our results to that of the traditional Bayesian filtering as well
nearest-neighbor (NN) and probabilistic data association (PDA) filters. We
show through simulations that accounting for clutters drastically improves the
performance the tracker. Furthermore, we demonstrate that METRIC-Bayes
framework which incorporates a joint nonparametric prior in Bayesian tracker
outperforms existing methods such as nearest neighbor and PDA filters. In
particular, our simulation results indicate that estimating the trajectory of
a target is exceedingly benefited from incorporating METRIC-Bayes to learn the
distribution of measurements that are originated from the target.

We assume a linear Gaussian scenario so that we are able to compare our
framework to the existing methods. The following linear Gaussian single-target
model is used that was introduced in [22]. Assume that the state vector
$\mathbf{x}_{\ell,k}$ is given by $\mathbf{x}_{\ell,k}$ = $[x_{\ell,k}\
y_{\ell,k}\ \dot{x}_{\ell,k}\ \dot{y}_{\ell,k}]^{T}$, $\ell$ =
$1,\ldots,N_{k}$, where $[x_{\ell,k}\ y_{\ell,k}]^{T}$, $[\dot{x}_{\ell,k}\
\dot{y}_{\ell,k}]^{T}$ are the 2-D position and 2-D velocity, respectively. We
assume a linear Gaussian model with transition probability

$\displaystyle p(\mathbf{x}_{k}\mid\mathbf{x}_{k-1})=\mathcal{N}(\mu,\Sigma)$

where $\boldsymbol{\mu}$ = $A_{k}\mathbf{x}_{k-1}$ and covariance matrix
$\Sigma$ = $\text{diag}([\sigma^{2}B_{k}B^{T}_{k}])$ where

$\displaystyle A_{k}\\!\\!=\\!\\!\begin{bmatrix}1&0&\Delta&0\\\
0&1&0&\Delta\\\ 0&0&1&0\\\
0&0&0&1\end{bmatrix}\\!\\!,B_{k}=\Delta\begin{bmatrix}\frac{1}{2}\\\
\frac{1}{2}\\\ 1\\\ 1\end{bmatrix}\\!.$

with $\sigma$ = $7$ m/s2 and $\Delta=1$. We assume that the probability of a
target remaining at a scene during transitioning to be $\text{P}_{\ell,k\mid
k-1}$ = $0.95$, for all $\ell$.

We also assume a Gaussian likelihood for measurements originated from the
target according to the likelihood

$\displaystyle
p(\mathbf{z}_{l,k}|\mathbf{x}_{k})=\mathcal{N}(H_{k}\mathbf{x}_{k}),Q_{k})$

where

$\displaystyle H_{k}\\!\\!=\\!\\!\begin{bmatrix}1&0&0\\\
0&1&0\end{bmatrix}\\!\\!,Q_{k}={\sigma^{\prime}}^{2}\begin{bmatrix}1&0\\\
0&1\end{bmatrix}\\!$

where ${\sigma^{\prime}}=10m$. The observation region is the square
$\mathcal{Z}=[-1000,1000]\times[-1000,1000]$ (units are in m). We also
consider Poisson clutter with parameter $\lambda_{k}=5$. Given the problem
setup, we put our framework to test in the following scenarios. In both cases,
the results are obtained by 10,000 Monte Carlo (MC) runs on the same
trajectory.

###  IV-A METRIC-Bayes vs traditional Bayesian Filtering

In this scenario, we compare the performance of our proposed framework to that
of traditional Bayesian filtering. Traditional Bayesian filtering does not
take into account the importance of clutters and consider them as one of the
measurements. We compare the target location estimation mean-squared error
(MSE) obtained using METRIC-Bayes vs Bayesian filter that uses all the
measurements. A typical sample run of our proposed method is shown in Figure
1, which suggests that the traditional filters tend to lose the track due to
dealing with the clutters. Figure 1 also shows that we successfully estimate
the trajectory of the target. Figure 2 shows the mean square error (MSE) for
the location of the target which signifies the importance of utilizing the
actual measurements distribution to track the target. We computed the signal
to the noise ration to be SCR = 5.9379.

Figure 1: True and estimate $x$-coordinate (top) and $y$-coordinate (bottom)
using METRIC-Bayes.

Figure 2: Mean square error, METRIC-Bayes vs traditional Bayesian Filtering.

###  IV-B METRIC-Bayes vs Nearest-Neighbor and PDA Filters

In this section, we compare our proposed method to nearest-neighbor (NN) and
probability data association (PDA) filters on the same set of data, which
suggests that the NN and PDA filters tend to follow the pattern of extraneous
target-generated measurements (clutter) and thus lead to poor performance. The
mean square error (MSE) comparison in Figure 3 demonstrates that even in
linear Gaussian case, the METRIC-Bayes method outperforms the NN and PDA
filters.

Figure 3: Target location estimation mean-squared error (MSE) obtained using
METRIC-Bayes vs NN and PDA filters for tracking a single object.

It is worth mentioning we run 10’000 Monte Carlo runs for algorithms and
algorithms are evaluated on the same dataset.

##  V Conclusion

In this paper, we developed a nonparametric model for the measurements in the
presence of high clutter. Our generative model is integrated into a Bayesian
tracker to robustly track a target using time-varying measurements. We
exploited the joint Dirichlet process prior to model the clutter and the
measurements that are originated from the target. The output of our framework
was then fed to a Bayesian tracker to estimate the trajectory of the target.
METRIC-Bayes provided a robust and computationally efficient framework that
can model any clutter distribution as well as efficiently estimating the
target trajectory in high clutter with unknown number of clutters. We
demonstrated through simulations that the amalgamation of the Bayesian
nonparametric prior with a Bayesian tracker improves the tracking algorithm.

## References

  * [1] Y. Bar-Shalom, P. K. Willett, and X. Tian, _Tracking and Data Fusion_. YBS publishing Storrs, 2011.
  * [2] L. S. Davis, D. LS _et al._ , “Noise cleaning by iterated local averaging,” 1978.
  * [3] X. R. Li, “Tracking in clutter with strongest neighbor measurements. i. theoretical analysis,” _IEEE Transactions on Automatic Control_ , vol. 43, pp. 1560–1578, 1998.
  * [4] T. Kirubarajan and Y. Bar-Shalom, “Probabilistic data association techniques for target tracking in clutter,” _Proceedings of the IEEE_ , vol. 92, pp. 536–557, 2004.
  * [5] S. Jeong and J. K. Tugnai, “Multisensor tracking of a maneuvering target in clutter using IMMPDA filtering with simultaneous measurement update,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 41, pp. 1122–1131, 2005.
  * [6] A. Gad, F. Majdi, and M. Farooq, “A comparison of data association techniques for target tracking in clutter,” in _International Conference on Information Fusion_ , 2002, pp. 1126–1133.
  * [7] N. L. Hjort, C. Holmes, P. Müller, and S. G. Walker, _Bayesian Nonparametrics_. Cambridge Univ. Press, 2010.
  * [8] B. Moraffah, “Bayesian nonparametric modeling and inference for multiple object tracking,” Ph.D. dissertation, Arizona State University, 2019.
  * [9] F. Caron, M. Davy, A. Doucet, E. Duflos, and P. Vanheeghe, “Bayesian inference for linear dynamic models with dirichlet process mixtures,” _IEEE Transactions on Signal Processing_ , vol. 56, no. 1, pp. 71–84, 2007.
  * [10] E. Fox, E. B. Sudderth, M. I. Jordan, and A. S. Willsky, “Bayesian nonparametric inference of switching dynamic linear models,” _IEEE Transactions on Signal Processing_ , vol. 59, pp. 1569–1585, 2011.
  * [11] B. Moraffah, C. Brito, B. Venkatesh, and A. Papandreou-Suppappola, “Use of hierarchical Dirichlet processes to integrate dependent observations from multiple disparate sensors for tracking,” in _International Conference on Information Fusion_ , 2019.
  * [12] B. Moraffah and A. Papndreou-Suppopola, “Bayesian nonparametric modeling for predicting dynamic dependencies in multiple object tracking,” _arXiv preprint arXiv:2004.10798_ , 2020.
  * [13] B. Moraffah, A. Papandreou-Suppappola, and M. Rangaswamy, “Nonparametric bayesian methods and the dependent pitman-yor process for modeling evolution in multiple object tracking,” in _2019 22th International Conference on Information Fusion (FUSION)_. IEEE, 2019, pp. 1–6.
  * [14] L. Wang, S. Fortunati, M. S. Greco, and F. Gini, “Reinforcement learning-based waveform optimization for mimo multi-target detection,” in _Asilomar Conference on Signals, Systems and Computers_ , 2018, pp. 1329–1333.
  * [15] B. Moraffah and A. Papandreou-Suppappola, “Dependent Dirichlet process modeling and identity learning for multiple object tracking,” in _Asilomar Conference on Signals, Systems, and Computers_ , 2018, pp. 1762–1766.
  * [16] ——, “Random infinite tree and dependent Poisson diffusion process for nonparametric Bayesian modeling in multiple object tracking,” in _IEEE Int. Conf. on Acoustics, Speech and Signal Processing_ , 2019, pp. 5217–5221.
  * [17] B. Moraffah, “Inference for multiple object tracking: A bayesian nonparametric approach,” _arXiv preprint arXiv:1909.06984_ , 2019.
  * [18] T. S. Ferguson, “A bayesian analysis of some nonparametric problems,” _The annals of statistics_ , pp. 209–230, 1973.
  * [19] J. Sethuraman, “A constructive definition of dirichlet priors,” _Statistica sinica_ , pp. 639–650, 1994.
  * [20] J. K. Ghosh and R. Ramamoorthi, _Bayesian nonparametrics_. Springer Science & Business Media, 2003.
  * [21] J. Pitman, _Combinatorial Stochastic Processes: Ecole d’Eté de Probabilités de Saint-Flour XXXII-2002_. Springer, 2006.
  * [22] B.-T. Vo, B.-N. Vo, and A. Cantoni, “Bayesian filtering with random finite set observations,” _IEEE Transactions on signal processing_ , vol. 56, no. 4, pp. 1313–1326, 2008.


CC_Attribution_4.0_International

#  Autoignition of Methyl Valerate at Low to Intermediate Temperatures and
Elevated Pressures in a Rapid Compression Machine

Bryan W. Weber  Justin Bunnell  Department of Mechanical Engineering,
University of Connecticut, Storrs, CT, USA  Kamal Kumar  Department of
Mechanical Engineering, University of Idaho, Moscow, ID, USA  Chih-Jen Sung
Department of Mechanical Engineering, University of Connecticut, Storrs, CT,
USA<EMAIL_ADDRESS>
###### Abstract

Methyl valerate (C6H12O2, methyl pentanoate) is a methyl ester and a relevant
surrogate component for biodiesel. In this work, we present ignition delays of
methyl valerate measured using a rapid compression machine at a range of
engine-relevant temperature, pressure, and equivalence ratio conditions. The
conditions we have studied include equivalence ratios from $0.252.0$,
temperatures between $680\text{\,}\mathrm{K}1050\text{\,}\mathrm{K}$, and
pressures of $15\text{\,}\mathrm{bar}30\text{\,}\mathrm{bar}$. The ignition
delay data demonstrate a negative temperature coefficient region in the
temperature range of $720\text{\,}\mathrm{K}800\text{\,}\mathrm{K}$ for both
$\phi=2.0$, $15\text{\,}\mathrm{bar}$ and $\phi=1.0$,
$30\text{\,}\mathrm{bar}$, with two-stage ignition apparent over the narrower
temperature ranges of $720\text{\,}\mathrm{K}760\text{\,}\mathrm{K}$ for the
lower pressure and $740\text{\,}\mathrm{K}760\text{\,}\mathrm{K}$ at the
higher pressure. In addition, the experimental ignition delay data are
compared with simulations using an existing chemical kinetic model from the
literature. The simulations with the literature model under-predict the data
by factors between $210$ over the entire range of the experimental data. To
help determine the possible reasons for the discrepancy between simulations
and experiments, a new chemical kinetic model is developed using the Reaction
Mechanism Generator (RMG) software. The agreement between the experimental
data and the RMG model is improved but still not satisfactory. Directions for
future improvement of the methyl valerate model are discussed.

###### keywords:

chemical kineticsrapid compression machineautoignitionmethyl ester

##  1 Introduction

For transportation applications, biodiesel is an important constituent in
improving environmental friendliness of fuels, due to its renewability when
produced from sustainable agricultural crops and its ability to reduce
emissions relative to conventionally fueled engines [1]. A recent review paper
summarizes the work on methyl esters relevant to biodiesel combustion [2].
Autoignition of methyl butanoate (MB, C5H10O2) has been well-studied in both
shock tube and rapid compression machine experiments, and readers are referred
to the review of [2] for further details. The prevalence of MB data in the
literature is largely due to the early identification of MB as a potential
surrogate fuel for biodiesel [3]. However, the experiments have shown that MB
may not be an appropriate surrogate for biodiesel, due to its lack of negative
temperature coefficient (NTC) behavior, a requirement for a suitable biodiesel
surrogate [2].

Larger methyl esters such as methyl valerate (MV, C6H12O2, methyl pentanoate)
have also been studied as possible biodiesel surrogates. [4] used a rapid
compression machine (RCM) to study the autoignition of several methyl esters
including MV. Although MV exhibited two-stage ignition in this study, little
additional research has been done on its oxidation. [5] studied MV in premixed
laminar flames and extended a detailed high temperature chemical kinetic model
to include MV and methyl hexanoate. [6] added MV to n-heptane/toluene fuel
blends to determine the resulting intermediate species in premixed flames
using a flat burner at $1\text{\,}\mathrm{a}\mathrm{t}\mathrm{m}$ and an
equivalence ratio of 1.75. The addition of MV helped reduce soot forming
intermediates including benzene, cyclopentadienyl, acetylene, propargyl, and
vinylacetylene [6]. [7] computationally examined the peroxy radical
isomerization reactions for MV to better understand the low temperature
reaction pathways. Finally, [8] used diffusion flames in the counterflow
configuration to determine extinction limits for a number of methyl esters,
including MV, and validated a detailed kinetic model with the experimental
data.

This work provides additional data for the autoignition of MV. Data is
collected in a RCM under engine relevant conditions spanning from
$15\text{\,}\mathrm{bar}30\text{\,}\mathrm{bar}$, equivalence ratios from
$0.252.0$, and temperatures from
$682\text{\,}\mathrm{K}1048\text{\,}\mathrm{K}$. The NTC region of MV is
mapped out to provide additional information on the fidelity of using MV as a
biodiesel surrogate.

##  2 Experimental Methods

The RCM used in this study is a single piston arrangement and is pneumatically
driven and hydraulically stopped. The device has been described in detail
previously [9] and will be described here briefly for reference. The end of
compression (EOC) temperature and pressure ($T_{C}$ and $P_{C}$ respectively),
are independently changed by varying the overall compression ratio, initial
pressure $(P_{0})$, and initial temperature $(T_{0})$ of the experiments. The
primary diagnostic on the RCM is the in-cylinder pressure. The pressure data
is processed by a Python package called UConnRCMPy [10], which calculates
$P_{C}$, $T_{C}$, and the ignition delay(s). The definitions of the ignition
delays are shown in Fig. 2. The time of the EOC is defined as the maximum of
the pressure trace prior to the start of ignition and the ignition delays are
defined as the time from the EOC until local maxima in the first time
derivative of the pressure. Each experimental condition is repeated at least
five times to ensure repeatability of the data. As there is some random
scatter present in the data, the standard deviation $(\sigma)$ of the ignition
delays from the runs at a given condition is computed. In all cases, $\sigma$
is less than $10\text{\,}\mathrm{\char 37\relax}$ of the mean value of the
overall ignition delay.

In addition to the reactive experiments, non-reactive experiments are
conducted to determine the influence of machine-specific behavior on the
experimental conditions and permit the calculation of the EOC temperature via
the isentropic relations between pressure and temperature [11]. The EOC
temperature is calculated by the procedure described in Section 3.

$\displaystyle-10$$\displaystyle 0$$\displaystyle 10$$\displaystyle
20$$\displaystyle 30$Time, ms$\displaystyle 0$$\displaystyle 10$$\displaystyle
20$$\displaystyle 30$$\displaystyle 40$$\displaystyle 50$$\displaystyle
60$Pressure,
bar$\displaystyle\tau_{1}$$\displaystyle\tau$EOC$\displaystyle-0.25$$\displaystyle
0.00$$\displaystyle 0.25$$\displaystyle 0.50$$\displaystyle
0.75$$\displaystyle 1.00$$\displaystyle 1.25$$\displaystyle
1.50$$\displaystyle 1.75$$\displaystyle 2.00$Time Derivative of Pressure,
bar/msPressureNon-ReactivePressureTime Derivative

Figure 1: Definition of the ignition delays used in this work. The experiment
in this figure was conducted for a $\phi=2.0$ mixture with
$\ce{Ar}/(\ce{N2}+\ce{AR})=0.5$, $P_{0}=$0.7694\text{\,}\mathrm{bar}$$,
$T_{0}=$373\text{\,}\mathrm{K}$$, $P_{C}=$14.94\text{\,}\mathrm{bar}$$,
$T_{C}=$723\text{\,}\mathrm{K}$$, $\tau=$27.44\pm 0.99\text{\,}\mathrm{ms}$$,
$\tau_{1}=$16.57\pm 0.48\text{\,}\mathrm{ms}$$.

$\displaystyle 275$$\displaystyle 300$$\displaystyle 325$$\displaystyle
350$$\displaystyle 375$$\displaystyle 400$$\displaystyle T$, K$\displaystyle
10^{-1}$$\displaystyle 10^{0}$$\displaystyle 10^{1}$$\displaystyle
10^{2}$$\displaystyle P_{\text{sat}}$, kPaOrtega et al. [12]van Genderen et
al. [13]Verevkin and Emel’yanenko [14]This Work

Figure 2: Saturated vapor pressure of MV as a function of temperature, plotted
using the Antoine equation, Eq. 1, with $A=6.4030$, $B=1528.69$, and
$C=52.881$.

The RCM is equipped with heaters to control the initial temperature of the
mixture. After filling in the components to the mixing tanks, the heaters are
switched on and the system is allowed $1.5\text{\,}\mathrm{h}$ to come to
steady state. The mixing tanks are also equipped with magnetic stir bars so
the reactants are well mixed for the duration of the experiments.

The initial temperature is chosen such that the saturated vapor pressure
$(P_{\text{sat}})$ of the fuel at the initial temperature is at least twice
the partial pressure of the fuel in the mixing tank. The Antoine equation

$\mathop{\log_{10}}{P_{\text{sat}}}=A-\frac{B}{T-C}$ (1)

is used to model the saturated vapor pressure of MV as a function of
temperature, where $A$, $B$, and $C$ are substance-specific coefficients. The
coefficients in Eq. 1 are determined by least squares fitting of the data of
[12], [13], and [14] using the `curve_fit()` function of SciPy [15] version
0.18.1. Figure 2 shows that the coefficients fit with this procedure give good
agreement with the experimental data.

The mixtures considered in this study are shown in Table 1. Mixtures are
prepared in stainless steel mixing tanks. The proportions of reactants in the
mixture are determined by specifying the absolute mass of the fuel, the
equivalence ratio $(\phi)$, and the ratio of $\ce{Ar}:\ce{N2}$ in the
oxidizer. Since MV is a liquid with a relatively small vapor pressure at room
temperature, it is injected into the mixing tank through a septum. Proportions
of O2, Ar, and N2 are added manometrically at room temperature.

Table 1: Mixtures considered in this work $\phi$ | Mole Fraction (purity) |  Ar/(N2 + Ar)
---|---|---
|  MV ($100\text{\,}\mathrm{\char 37\relax}$) |  O2 ($99.994\text{\,}\mathrm{\char 37\relax}$) |  Ar ($99.999\text{\,}\mathrm{\char 37\relax}$) |  N2 ($99.999\text{\,}\mathrm{\char 37\relax}$) |
$0.25$ | 0.0065 | 0.2087 | 0.7848 | 0.0000 | 1.0
$0.5$ | 0.0130 | 0.2074 | 0.7798 | 0.0000 | 1.0
$1.0$ | 0.0256 | 0.2047 | 0.7697 | 0.0000 | 1.0
$1.0$ | 0.0256 | 0.2047 | 0.3849 | 0.3848 | 0.5
$2.0$ | 0.0499 | 0.1996 | 0.0000 | 0.7505 | 0.0
$2.0$ | 0.0499 | 0.1996 | 0.3752 | 0.3753 | 0.5

##  3 Computational Methods

###  3.1 RCM Modeling

The Python 3.5 interface of Cantera [16] version 2.2.1 is used for all
simulations in this work. Detailed descriptions of the use of Cantera for
these simulations can be found in the work of [17] and [18]; a brief overview
is given here. As mentioned in Section 2, non-reactive experiments are
conducted to characterize the machine-specific effects on the experimental
conditions in the RCM. This pressure trace is used to compute a volume trace
by assuming that the reactants undergo a reversible, adiabatic, constant
composition (i.e., isentropic) compression during the compression stroke and
an isentropic expansion after the EOC. The volume trace is applied to a
simulation conducted in an `IdealGasReactor` in Cantera [16] using the CVODES
solver from the SUNDIALS suite [19]. The ignition delay from the simulations
is defined in the same manner as in the experiments. The time derivative of
the pressure in the simulations is computed by second order Lagrange
polynomials, as discussed by [20].

To the best of our knowledge, there are three mechanisms for MV combustion
available in the literature. The first two, by [5] and [6], were developed to
simulate flames, and do not include the low-temperature chemistry necessary to
simulate the conditions in these experiments. The third model was developed by
[8] and includes low-temperature chemistry of MV, although it was only
validated by comparison with flame extinction limits. The detailed [8] model
includes 1103 species and 7557 reactions.

###  3.2 Reaction Mechanism Generator

In addition to using a mechanism from the literature, we investigate the use
of an automatic mechanism generator, the open-source Reaction Mechanism
Generator (RMG) [21] version 1.0.4. The Python version of RMG is used, which
requires Python 2.7, and version 1.10.0 of the RMG database is used. The final
RMG model contains 483 species and 19990 reactions. Note that the number of
species is much lower than the [8] model because the RMG model focuses on only
one fuel (MV), but the number of reactions is substantially higher.

##  4 Experimental Results

Figure 3 shows the ignition delay results measured in this study. Filled
markers denote the overall ignition delay and hollow markers indicate the
first-stage ignition delay. Vertical error bars are drawn on the symbols to
represent the uncertainty in the ignition delay; for many of the experiments,
the uncertainty is approximately the same size as the data point, so the error
bar is hidden. Horizontal error bars are shown on the first and last points of
each equivalence ratio indicating the estimated uncertainty in the EOC
temperature of $\pm$1\text{\,}\mathrm{\char 37\relax}$$ [22]. Fig. 3a shows
the results for a compressed pressure of $15\text{\,}\mathrm{bar}$, while Fig.
3b shows the results for a compressed pressure of $30\text{\,}\mathrm{bar}$.
Note that $\phi=2.0$ results were not collected for $30\text{\,}\mathrm{bar}$,
so there are no red data points in Fig. 3b.

It can be seen from Fig. 3 that the ignition delays for the $\phi=0.25\text{
and }0.5$ mixtures do not show an NTC region of the ignition delay for both of
the pressures studied in this work. However, the $\phi=1.0$ mixture shows an
NTC region at $P_{C}=$30\text{\ }\mathrm{bar}$$ between approximately
$720\text{\,}\mathrm{K}800\text{\,}\mathrm{K}$, with measured first-stage
ignition delays at $734\text{\,}\mathrm{K}757\text{\,}\mathrm{K}$. In
addition, the $\phi=2.0$ mixture shows an NTC region of ignition delay at
$15\text{\,}\mathrm{bar}$ from approximately
$720\text{\,}\mathrm{K}775\text{\,}\mathrm{K}$, with measured first-stage
ignition delays between $720\text{\,}\mathrm{K}750\text{\,}\mathrm{K}$.

$\displaystyle 1.0$$\displaystyle 1.1$$\displaystyle 1.2$$\displaystyle
1.3$$\displaystyle 1.4$$\displaystyle 1.5$10100a) $\displaystyle P_{C}=15\
\text{bar}$$\displaystyle\phi=0.25$$\displaystyle\phi=0.50$$\displaystyle\phi=1.00$$\displaystyle\phi=2.00$$\displaystyle
1.0$$\displaystyle 1.1$$\displaystyle 1.2$$\displaystyle 1.3$$\displaystyle
1.4$$\displaystyle 1.5$10b) $\displaystyle P_{C}=30\ \text{bar}$1000 K900 K800
K700 K1000 K900 K800 K700 KIgnition Delay, ms$\displaystyle 1000/T_{C}$, 1/K
Figure 3: Ignition delays of MV as a function of inverse temperature. Filled
points are the overall ignition delays and hollow points are the first stage
ignition delays. a) $15\text{\,}\mathrm{bar}$. b) $30\text{\,}\mathrm{bar}$

Figure 5 shows the pressure traces for selected experiments at $\phi=1.0$,
$P_{C}=$30\text{\ }\mathrm{bar}$$. The three reactive pressure traces shown
are at the low-temperature end of the NTC (blue, $700\text{\,}\mathrm{K}$),
one case with two-stage ignition (orange, $734\text{\,}\mathrm{K}$), and one
case near the high-temperature limit of the NTC region (green,
$775\text{\,}\mathrm{K}$). Also shown is the non-reactive pressure trace for
the $700\text{\,}\mathrm{K}$ case (red). By comparing the
$700\text{\,}\mathrm{K}$ pressure trace with the non-reactive pressure trace,
it can be seen that there is substantial heat release prior to main ignition
as measured by the deviation of the reactive pressure trace from the non-
reactive trace. However, there is only one peak in the time derivative of the
pressure, so no first-stage ignition delay is defined for this case. It can
also be seen in Fig. 5 that the $775\text{\,}\mathrm{K}$ case shows some heat
release prior to ignition, although again there is only one peak in the time
derivative of the pressure. Furthermore, the heat release at
$775\text{\,}\mathrm{K}$ appears to be more gradual than at the lowest
temperature.

$\displaystyle-20$$\displaystyle-15$$\displaystyle-10$$\displaystyle-5$$\displaystyle
0$$\displaystyle 5$$\displaystyle 10$$\displaystyle 15$Time, ms$\displaystyle
0$$\displaystyle 10$$\displaystyle 20$$\displaystyle 30$$\displaystyle
40$$\displaystyle 50$$\displaystyle 60$Pressure, bar$\displaystyle T_{C}=700$
K$\displaystyle T_{C}=734$ K$\displaystyle T_{C}=775$ KNon-Reactive

Figure 4: Selected pressure traces around the NTC region of ignition delay for
$\phi=1.0$

$\displaystyle 1.1$$\displaystyle 1.2$$\displaystyle 1.3$$\displaystyle
1.4$$\displaystyle 1000/T_{C}$, 1/K110100Ignition Delay, ms$\displaystyle
P_{C}=15\ \text{bar}$$\displaystyle P_{C}=30\ \text{bar}$Experimental
DataDiévart et al. [8] ModelRMG Model900 K800 K700 K

Figure 5: Comparison of experimental and simulated results for $\phi=1.0$

##  5 Computational Results

Figure 5 compares experimentally measured overall ignition delays with
ignition delays computed with the detailed model of [8] for the $\phi=1.0$
experiments. Results for the other equivalence ratios are similar to these
results, so are not shown here. It is important to note that the model of [8]
was not validated for MV ignition delays, only for extinction strain rates. At
$15\text{\,}\mathrm{bar}$, the model tends to under-predict the ignition delay
and predicts an NTC region that is not present in the experiments. At
$30\text{\,}\mathrm{bar}$, the model predicts the low-temperature ignition
delays well, but does not predict the NTC region found experimentally.

To understand the underlying reasons for the disagreement between the [8]
model and the data, we constructed an additional model using RMG (see Section
3.2). As can be seen in Fig. 5, the agreement between the RMG model and the
experimental data is similar to the [8] model for the
$30\text{\,}\mathrm{bar}$ data. At $15\text{\,}\mathrm{bar}$, the RMG model
predicts a somewhat longer ignition delay than the model of [8], but still
predicts an NTC region where none is present in the experimental data.

In general, there could be three likely sources of error in the models:
missing reaction pathways, incorrect values of the reaction rates, and
incorrect values for thermodynamic properties of the species. We have noted in
Section 3.2 that the RMG model has many more reactions than the [8] model and
the algorithm used in RMG considers a substantial number of the possible
pathways. This reduces the possibility of missing reaction pathways affecting
the model. Further detailed studies are required to ensure that the RMG model
includes all of the relevant reaction pathways.

The second source of error may be incorrect reaction rate parameters, either
because the rates are specified incorrectly in the model (e.g., typos) or
because the rates are not well estimated by the typical analogy based-rules.
It should be noted that errors of this type may affect the model generated by
RMG—if the rates are not estimated correctly, reactions that are important in
reality may not be included in the model. Determining the accuracy of the
reaction rates used in the RMG and [8] models requires further detailed
studies of the models. Another related source of error could be incorrect
estimation of the pressure dependence of the reaction rates, which may be
particularly important for the isomerization reactions prevalent in low-
temperature chemistry.

The third source of error may lie in the estimation of the thermodynamic
properties of the species, particularly their heats of formation. We have
begun to analyze the possibility of this source of error by conducting a
reaction pathway analysis to determine which radicals are formed from the
breakdown of the fuel. The following analysis is conducted for a constant
volume simulation at $700\text{\,}\mathrm{K}$, $30\text{\,}\mathrm{bar}$,
where the rates of production of the species have been integrated until the
time of $20\text{\,}\mathrm{\char 37\relax}$ fuel consumption. The results of
this analysis are shown in Fig. 6 and Table 2 for the two models. The
percentages shown in the Table 2 are the percent of the fuel destroyed to form
a particular fuel radical by all the reactions that can form that radical.

At the relatively low temperature and high pressure condition of this
analysis, all of the fuel is destroyed by H-atom abstractions to form the fuel
radicals shown. It can be seen that the two models have quite different
distributions of products from the first H-abstraction reactions. The model of
[8] predicts that H-abstraction from the second carbon is the most prevalent,
followed closely by abstraction from the methyl group. This is in line with
the bond energies of the C-H bonds for those carbon atoms; we expect that the
presence of the oxygen atoms will cause hydrogen abstraction at the nearby
carbons to be favored. However, the RMG model predicts that radicals in the
middle of the carbon chain will be primarily formed. The cause of this
discrepancy is under investigation, but it may be caused by the estimation of
thermodynamic properties of the radicals.

Figure 6: Structure of MV with carbon atoms labeled according to the
convention used in Table 2

Table 2: Percent of MV destroyed to form fuel radical species with a hydrogen
atom missing at the location indicated in the first column

Radical Site |  [8] [$\mathrm{\char 37\relax}$] | RMG Model [$\mathrm{\char 37\relax}$]
---|---|---
2 | $29.3$ | $7.4$
3 | $17.5$ | $36.0$
4 | $17.5$ | $41.1$
5 | $9.4$ | $3.7$
M | $26.3$ | $11.8$

##  6 Conclusions

In this study, we have measured ignition delays for methyl valerate over a
wide range of engine-relevant pressures, temperatures, and equivalence ratios.
An NTC region of the ignition delay and two-stage ignition were recorded for
pressures of $15\text{\,}\mathrm{bar}$ at $\phi=2.0$ and
$30\text{\,}\mathrm{bar}$ at $\phi=1.0$. A detailed chemical kinetic model
available in the literature was unable to reproduce the experimental results,
so a new model was constructed using the Reaction Mechanism Generator
software. Although the new model contains many more reactions than the
literature model, it is still unable to predict the experimental ignition
delays satisfactorily. Possible reasons for the discrepancy include missing
reaction pathways, incorrect rate estimates, and incorrect thermodynamic
property estimates. Future work will include investigation of the
discrepancies between models and experiments to further understand the
autoignition kinetics of methyl valerate.

## References

  * [1] S. Hoekman and Curtis Robbins  “Review of the Effects of Biodiesel on NOx Emissions”  In _Fuel Process. Technol._ 96, 2012, pp. 237–249  DOI: 10.1016/j.fuproc.2011.12.036
  * [2] L. Coniglio, H. Bennadji, P.A. Glaude, O. Herbinet and F. Billaud  “Combustion Chemical Kinetics of Biodiesel and Related Compounds (Methyl and Ethyl Esters): Experiments and Modeling – Advances and Future Refinements”  In _Prog. Energy Combust. Sci._ 39.4, 2013, pp. 340–382  DOI: 10.1016/j.pecs.2013.03.002
  * [3] E.M. Fisher, William J. Pitz, Henry J. Curran and Charles K. Westbrook  “Detailed Chemical Kinetic Mechanisms for Combustion of Oxygenated Fuels” 10  In _Proc. Combust. Inst._ 28.2, 2000, pp. 1579–1586  DOI: 10.1016/S0082-0784(00)80555-X
  * [4] K. Hadj-Ali, M. Crochet, G. Vanhove, Marc Ribaucour and Rodolphe Minetti  “A Study of the Low Temperature Autoignition of Methyl Esters” 23  In _Proc. Combust. Inst._ 32.1, 2009, pp. 239–246  DOI: 10.1016/j.proci.2008.09.002
  * [5] Oleg P. Korobeinichev, Ilya E. Gerasimov, Denis A. Knyazkov, Andrey G. Shmakov, Tatyana A. Bolshova, Nils Hansen, Charles K. Westbrook, Guillaume Dayma and Bin Yang  “An Experimental and Kinetic Modeling Study of Premixed Laminar Flames of Methyl Pentanoate and Methyl Hexanoate”  In _Z. Für Phys. Chem._ 229.5, 2015  DOI: 10.1515/zpch-2014-0596
  * [6] Artem M. Dmitriev, Denis A. Knyazkov, Tatyana A. Bolshova, Andrey G. Shmakov and Oleg P. Korobeinichev  “The Effect of Methyl Pentanoate Addition on the Structure of Premixed Fuel-Rich n-Heptane/Toluene Flame at Atmospheric Pressure”  In _Combust. Flame_ 162.5, 2015, pp. 1964–1975  DOI: 10.1016/j.combustflame.2014.12.015
  * [7] C. Hayes and Donald R. Burgess  “Exploring the Oxidative Decompositions of Methyl Esters: Methyl Butanoate and Methyl Pentanoate as Model Compounds for Biodiesel” 26  In _Proc. Combust. Inst._ 32.1, 2009, pp. 263–270  DOI: 10.1016/j.proci.2008.05.075
  * [8] Pascal Diévart, Sang Hee Won, Jing Gong, Stephen Dooley and Yiguang Ju  “A Comparative Study of the Chemical Kinetic Characteristics of Small Methyl Esters in Diffusion Flame Extinction”  In _Proc. Combust. Inst._ 34.1, 2013, pp. 821–829  DOI: 10.1016/j.proci.2012.06.180
  * [9] Gaurav Mittal and Chih-Jen Sung  “A Rapid Compression Machine for Chemical Kinetics Studies at Elevated Pressures and Temperatures”  In _Combust. Sci. Technol._ 179.3, 2007, pp. 497–530  DOI: 10.1080/00102200600671898
  * [10] Bryan William Weber and Chih-Jen Sung  “UConnRCMPy”, 2016  URL: https://github.com/bryanwweber/UConnRCMPy/releases/tag/v2.1.0
  * [11] Daeyup Lee and Simone Hochgreb  “Rapid Compression Machines: Heat Transfer and Suppression of Corner Vortex”  In _Combust. Flame_ 114, 1998, pp. 531–545  DOI: 10.1016/S0010-2180(97)00327-1
  * [12] Juan Ortega, Fernando Espiau, José Tojo, José Canosa and Ana Rodríguez  “Isobaric Vapor–Liquid Equilibria and Excess Properties for the Binary Systems of Methyl Esters + Heptane”  In _J. Chem. Eng. Data_ 48.5, 2003, pp. 1183–1190  DOI: 10.1021/je030117d
  * [13] Aad C.G. Genderen, J.Cees Miltenburg, Jacobus G. Blok, Mark J. Bommel, Paul J. Ekeren, Gerrit J.K. Berg and Harry A.J. Oonk  “Liquid–vapour Equilibria of the Methyl Esters of Alkanoic Acids: Vapour Pressures as a Function of Temperature and Standard Thermodynamic Function Changes”  In _Fluid Phase Equilibria_ 202.1, 2002, pp. 109–120  DOI: 10.1016/S0378-3812(02)00097-3
  * [14] Sergey P. Verevkin and Vladimir N. Emel’yanenko  “Transpiration Method: Vapor Pressures and Enthalpies of Vaporization of Some Low-Boiling Esters”  In _Fluid Phase Equilibria_ 266, 2008, pp. 64–75  DOI: 10.1016/j.fluid.2008.02.001
  * [15] Eric Jones, Travis Oliphant and Pearu Peterson  “SciPy: Open Source Scientific Tools for Python”, 2001  URL: http://www.scipy.org/
  * [16] David G. Goodwin, Harry K. Moffat and Raymond L. Speth  “Cantera: An Object-Oriented Software Toolkit for Chemical Kinetics, Thermodynamics, and Transport Processes”, 2016  URL: http://www.cantera.org
  * [17] Bryan William Weber and Chih-Jen Sung  “UConnRCMPy: Python-Based Data Analysis for Rapid Compression Machines”  In _Proceedings of the 15th Python in Science Conference_ , 2016, pp. 36–44
  * [18] Enoch E. Dames, Andrew S. Rosen, Bryan W. Weber, Connie W. Gao, Chih-Jen Sung and William H. Green  “A Detailed Combined Experimental and Theoretical Study on Dimethyl Ether/Propane Blended Oxidation”  In _Combust. Flame_ 168, 2016, pp. 310–330  DOI: 10.1016/j.combustflame.2016.02.021
  * [19] Alan C. Hindmarsh, Peter N. Brown, Keith E. Grant, Steven L. Lee, Radu Serban, Dan E. Shumaker and Carol S. Woodward  “SUNDIALS: Suite of Nonlinear and Differential/Algebraic Equation Solvers”  In _ACM Trans. Math. Softw._ 31.3, 2005, pp. 363–396  DOI: 10.1145/1089014.1089020
  * [20] Steven C. Chapra and Raymond P. Canale  “Numerical Methods for Engineers”  Boston: McGraw-Hill Higher Education, 2010
  * [21] Joshua W. Allen, C. Goldsmith and William H. Green  “Automatic estimation of pressure-dependent rate coefficients.” bibtex: Allen2012  In _Physical Chemistry Chemical Physics_ 14.3, 2012, pp. 1131–1155  DOI: 10.1039/c1cp22765c
  * [22] Bryan William Weber, Chih-Jen Sung and Michael W. Renfro  “On the Uncertainty of Temperature Estimation in a Rapid Compression Machine”  In _Combust. Flame_ 162.6, 2015, pp. 2518–2528  DOI: 10.1016/j.combustflame.2015.03.001