# Pruning Convolutional Filters via Reinforcement Learning with Entropy
Minimization

Muşat Bogdan
Transilvania University of Braşov, Romania,
Department of Electrical Engineering and Computer Science,
<EMAIL_ADDRESS>
&Andonie Răzvan
Central Washington University, USA,
Department of Computer Science
<EMAIL_ADDRESS>

###### Abstract

Structural pruning has become an integral part of neural network optimization,
used to achieve architectural configurations which can be deployed and run
more efficiently on embedded devices. Previous results showed that pruning is
possible with minimum performance loss by utilizing a reinforcement learning
agent which makes decisions about the sparsity level of each neural layer by
maximizing as a reward the accuracy of the network. We introduce a novel
information-theoretic reward function which minimizes the spatial entropy of
convolutional activations. This minimization ultimately acts as a proxy for
maintaining accuracy, although these two criteria are not related in any way.
Our method shows that there is another possibility to preserve accuracy
without the need to directly optimize it in the agent’s reward function. In
our experiments, we were able to reduce the total number of FLOPS of multiple
popular neural network architectures by 5-10$\times$, incurring minimal or no
performance drop and being on par with the solution found by maximizing the
accuracy.

_K_ eywords neural network pruning $\cdot$ reinforcement learning $\cdot$
AutoML

##  1 Introduction

Modern convolutional neural networks (CNNs) emerged with the publication of
AlexNet [10] in 2012, which paved the way for other architectures like VGG
[11], ResNet [12] and EfficientNet [13]. Although these networks posses a very
high capacity and perform at a super human level, they are often
overparametrized [14], which induces high latency and power consumption on
battery powered devices. Techniques like pruning and quantization [6, 7, 15,
46, 8] have recently become very popular, since they can generate power-
efficient sub-versions of these overparametrized networks.

While pruning deals with removing unimportant weights from a network by
applying a certain heuristic, quantization operates by using less bits for
weights and activations, thus speeding up overall computations. In our work we
only focus on structured pruning, which translates into removing entire
filters from a convolutional kernel. For this, we use an automated machine
learning (AutoML) framework to select the most suited percentage of structured
sparsity for each neural layer [1].

AutoML is a powerful strategy used for many tasks like neural architecture
search (NAS), hyperparameter search, data preparation, feature engineering [2,
3]. The principle behind it is to automate manual searching tasks and find
optimal solutions faster than we can manually do. Recently, AutoML was applied
for network compression via pruning [1]. By usage of a reinforcement learning
(RL) agent [9], the system can automatically choose the sparsity percentage
per layer, and then a magnitude-based pruning heuristic is applied, which
removes the top percentage filters with the smallest magnitude. The reward
criterion that the agent uses is the accuracy of the network obtained on a
randomly chosen subset from the training set or the validation set at the end
of the pruning phase.

We discovered that the accuracy of the network is not the only reward
criterion that can be used for AutoML network compression. Our central
contribution is an information-theoretical reward function (entropy
minimization) for the agent, which is completely different than the accuracy
used in [9]. We utilize this information-theoretical criterion for network
pruning.

Generally, a low entropic value of a system denotes more certainty, while a
high entropic value translates into more disorder. In neural networks, the
cross-entropy between two probability distributions is very often used as a
measure of error between the predicted output of a network and the true class
distribution of the target. In contrast, the entropy of neural activations the
hidden neural layers is seldom computed or used. This observation offered us
the motivation for the current work. As such, we propose to analyze the impact
of the entropy of all convolutional layers on network pruning. The question is
if such an impact exists and, if true, how to use it for neural pruning. Since
an ideal entropy value for the final output of the network is $0$ (i.e., to
predict the correct class with 100% confidence), our hypothesis is that the
entropy of neural activations should be minimized in order to preserve
essential information and also reduce uncertainty.

The intriguing result of our work is the discovery of an interesting
connection between entropy minimization and structural pruning. This could be
related to the structural entropy measure recently introduced in [47], where
"structural entropy refers to the level of heterogeneity of nodes in the
network, with the premise that nodes that share functionality or attributes
are more connected than others".

In practice, we utilize the AutoML framework from [1] to sparsify a neural
network and propose as an optimization reward criterion the minimization of
the spatial entropy (as defined in [4]) at each convolutional layer. Through
our experiments, we empirically show that this minimization acts as a proxy
for maintaining accuracy. The novelty of our work consists in discovering that
there are other, more principled approaches, to neural network pruning than
directly optimize the accuracy of the agent’s reward function.

The rest of the paper is structured as follows. Section 2 describes related
work. Section 3 introduces the basic notations of spatial entropy, used in
this paper. Section 4 details our information-theoretical pruning method. The
experimental results are described in Section 5 and Section 6 concludes with
final remarks.

##  2 Related Work

In this section we summarize relevant previous work related to neural network
pruning, AutoML and RL for pruning.

Pruning: The main objective of pruning is the reduction of the total number of
floating point operations per second (FLOPS) and parameters by removing
redundant weights from the network.

In the context of hardware accelerators, where computational availability
might be limited, we argue that the ratio of preserved FLOPS for the same
optimized architecture is a better measure of compression than simply counting
the number of parameters after pruning, because two networks with the same
number of parameters might posses totally different FLOPS counts, and as such
the one with less FLOPS for the same should be more desirable. Only when
memory space is an issue, a network with less parameters may be preferable.

The first approaches for neural pruning emerged in the ’90s with the classical
methods of optimal brain damage [31] and optimal brain surgeon [32]. Since
then, the importance of pruning was observed in improving training and
inference time, better generalization. With the emergence of large deep neural
networks, pruning became even more relevant and desirable, since modern
network architectures are overparametrized and there is a lot of space for
optimization. As such, a large suite of methods for pruning have been proposed
in the last years. Comprehensive surveys on neural network pruning can be
found in [6, 7].

Out of these methods, we highlight a few of them. Frankle et al. proposed the
lottery ticket hypothesis [33], representing a method to search for
subnetworks derived from a randomly initialized network which, if trained from
scratch, can attain the same performance as the original network and be sparse
at the same time. In [34] the authors argued that the traditional pipeline of
pruning: training, pruning, fine-tuning can result in networks with suboptimal
performance and what is actually important is the architecture configuration
itself resulted from pruning and not the weights that are maintained. As such,
they showed that training a pruned network configuration from scratch can
result in better accuracy than by fine-tuning it. The fine-tuning procedure is
also studied in [36] and they proposed two tricks to improve it. The first one
is called weight rewinding which turns back weights which were not pruned to
their earlier in time training values and continues retraining them using the
original training schedule. The second trick is called learning rate rewinding
which trains the final values of the unpruned weights using the same learning
rate schedule as weight rewinding. Both tricks were shown to outperform
classical fine-tuning. In [35] the authors proposed a method to estimate the
contribution of a filter to the final loss and iteratively remove those with
smaller scores. To estimate a filter’s importance they used two methods based
on the first and second order Taylor expansions. [37] introduced "The Rigged
Lottery" or RigL which trains sparse networks without the need of a "lucky"
initialization. Their method updates the topology of a sparse network during
training using parameter magnitudes and infrequent gradient calculations. They
obtained better solutions than most dense-to-sparse training algorithms.

AutoML: AutoML has been widely adopted by the community to solve tasks where a
huge amount of manual search would be involved. A comprehensive survey on
AutoML can be found in [2, 3]. We aim to present in the following some of the
most important problems associated with AutoML.

Data augmentation has become such an important part of an ML pipeline, that
manually searching for strategies is no longer feasible or desirable. As such,
methods like AutoAugment [17], RandAugment [25] and Population Based
Augmentation [18] have recently emerged in the literature and have been
integrated in many training pipelines [26, 27, 28]. Such methods are popular
because, depending on the input images, they can automatically select between
using multiple pre-existing augmentation techniques or choose the best
hyperparameters for them.

Another important task is to search for optimal architectures [19, 20, 21, 22,
23]. Relevant architectures like EfficientNet B0 [13] and AmoebaNet [29] were
found by such techniques. Given the huge search space of neural layers’
combinations, human inventiveness and intuition have fallen behind these
methods which can exhaustively look up for better solutions. Optimized
networks were obtained this way, under specific hardware constraints [30].

AutoML solutions for network pruning were also proposed [1, 24]. These methods
employ an RL agent to decide which individual weights or filters to drop, or
the optimal amount of sparsity for each layer. It is known that some layers
need more capacity [6]. Therefore, the uniform distribution approach for
sparsity doesn’t perform very well, and the sparsity has to be adapted by
taking into consideration each layer’s constraints.

Reinforcement learning for pruning: An important result, the one on which our
paper is based on, is by He et al. [1]. They used a DDPG agent [9] to choose
the percentage of sparsity for each learnable layer, be it either
convolutional or fully connected, by maximizing the accuracy of the network
after pruning. For pruning convolutional layers, the filters with the lowest
total magnitude were marked for sparsification, while for fully connected
layers the smallest weights were the ones being discarded. This form of
sparsification is considered to be structured, because weights are being
dropped respecting a certain structure (i.e., a whole filter), and not being
randomly dropped within a filter. This form of structured sparsification is
more efficient, as whole filters can be removed from computations, while for
the other form of sparsification, called random, some computations within
filters have to be performed and others not, complicating the final arithmetic
logic.

Another important work that conducts pruning by means of an RL agent is [16].
The important difference compared to the previous work is that pruning is
performed during runtime and adapted for the inputs. The assumption here is
that some samples need more computational resources to be dealt with and thus
the agent decides dynamically how much sparsity the network needs during
inference. The main disadvantage is that an additional recurrent neural
network (RNN) agent needs to run in order to sample actions during inference,
while the previous method finds a fixed sub-network from the original one,
which is also used during inference.

##  3 Background: Spatial Entropy

Our goal is to minimize the spatial entropy of neural convolutional
activations using the framework of AMC [1] for structural sparsity. We also
want to study whether this minimization can substitute direct accuracy
optimization. If this assumption holds true, it might mean that there is a
connection between pruning and entropy minimization. In the end, pruning
unimportant filters ideally removes useless information.

In [4] the notion of spatial entropy of saliency maps in CNNs was used to
study the layer-wise and time-wise evolution of them [4, 5]. To compute the
spatial entropy, the authors used the aura matrix entropy, initially defined
in [38]. In [4] it was noticed that the spatial entropy value decreases when
going along the depth of a neural network as the process called semiotic
superization takes place. Then, in [5] the authors studied the dynamics of the
spatial entropy in time, how it evolves during training, and found a link with
fitting and compression present in the information bottleneck framework [39].

To make this paper self-contained, we describe below the mathematical
formulations for computing the spatial entropy used in this work. Let us
define the joint probability of features at spatial locations $(i,j)$ and
$(i+k,\>j+l)$ to take the value $g$, respectively $g^{\prime}$ as:

$p_{gg^{\prime}}(k,l)=P(X_{i,\>j}=g,X_{i+k,\>j+l}=g^{\prime})$ (1)

where $g$ and $g^{\prime}$ are quantized convolutional activation values. In
the Experiments section we tested using multiple bin sizes for quantization to
verify the robustness of the procedure, and noticed that the proposed
minimization produces good solutions for a wide range of bin size values. For
each pair $(k,\>l)$ we define the entropy:

$H(k,\>l)=-\sum_{g}\sum_{g^{\prime}}p_{gg^{\prime}}(k,\>l)\log
p_{gg^{\prime}}(k,\>l)$ (2)

where the summations are over the number of bin sizes. A standardized relative
measure of bivariate entropy is [40]:

$H_{R}(k,\>l)=\frac{H(k,\>l)-H(0)}{H(0)}\in[0,\>1]$ (3)

The maximum entropy $H_{R}(k,\>l)=1$ corresponds to the case of two
independent variables. $H(0)$ is the univariate entropy, which assumes all
features as being independent, and we have $H(k,\>l)\geq H(0)$.

Based on the relative entropy for $(k,\>l)$, the Spatial Disorder Entropy
(SDE) for an $m\times n$ image X was defined in [40] as:

$H_{SDE}(\textbf{X})\approx\frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}\sum_{k=1}^{m}\sum_{l=1}^{n}H_{R}(i-k,\>j-l)$
(4)

Since the complexity of SDE computation is high, we decided to use a
simplified version - the Aura Matrix Entropy (AME, see [38]), which only
considers the second order neighbors from the SDE computation:

$\displaystyle\begin{split}H_{AME}(\textbf{X})\approx\frac{1}{4}\bigg{(}H_{R}(-1,\>0)+H_{R}(0,\>-1)+H_{R}(1,\>0)+H_{R}(0,\>1)\bigg{)}\end{split}$
(5)

Putting it all together, starting from a feature map obtained after the
application of a convolutional layer, we compute the probabilities
$p_{gg^{\prime}}$ in equation (1), and finally the AME in equation (5), which
results in the spatial entropy quantity for a channel. We compute the mean of
the spatial entropy values obtained from all the channels to get the final
estimate for a particular convolutional activation layer. When computing the
mean, we exclude channels which contain only $0$ values, which would produce a
spatial entropy of $0$.

##  4 Our Method

This section describes our method, which is a modification of the AMC
framework [1] for structural pruning. The main difference between our approach
and [1] is that our agent’s reward function minimizes the spatial entropy of
convolutional activations, instead of maximizing the accuracy of the model
(see Figure 1).

Figure 1: A DDPG agent is responsible with choosing the amount of sparsity
$a_{t}$ applied to each layer, based on the state of that layer $s_{t}$, by
considering multiple variables (see the original AMC paper for more details).
After the agent decides the amount of sparsity for each layer, the environment
computes a reward for the agent. In the original AMC formulation, this reward
was the accuracy of the pruned network computed on a validation dataset. In
our case, this reward consists in the mean of spatial entropies computed at
multiple convolutional layer outputs. The agent’s objective is to choose the
amount of sparsity for each layer such that this mean entropy is minimized.

The AMC framework is an AutoML tool for pruning which selects the percentage
of sparsity for each layer of a neural network (layer by layer), then an
algorithm based on $L_{2}$ magnitude marks the top percentage filters with
lowest magnitude for removal. Since the accuracy of a model is a non-
differentiable function for which gradients can not be computed and used
during backpropagation, a RL technique has to be used to optimize this
criterion, which will be treated as a reward function. Hence, the engine
driving the percentage selection for pruning is a DDPG agent [9], trained via
an actor critic technique [41], using the accuracy computed on a separate
dataset as a reward criterion. The reward function can be computed either on a
split from the training dataset or the validation dataset. By pursuing optimal
actions (with high reward), the agent will be rewarded correspondingly and
encouraged to perform similar actions in the future, while generally
discouraging actions with poor reward.

AMC stores inputs and outputs for each layer at the very beginning of
optimization by feed-forwarding a batch of input samples (called calibration
samples). After pruning using the magnitude-based heuristic, the channels from
the inputs of the calibration samples which have the same indexes as the
discarded filters will be dropped as well. A least squares regression is
applied to adjust the remaining weights to the new inputs and already stored
outputs. After the AMC framework finds the best subnetwork configuration which
maximizes the given reward function, and the weights have been adjusted as
well via the least squares regression, the new network configuration is fine-
tuned, as is standard in pruning literature.

Different from the original AMC formulation, we modify the accuracy based
reward by introducing a function which minimizes the average of the spatial
entropies of convolutional activations. Our goal is to observe if entropy
minimization can be used as a criterion in place of directly computing
accuracy, establishing thus a potential interesting link between the fields of
neural pruning and information theory. Because the framework of AMC tries to
constantly increase the amount of reward it receives, and since the mean
spatial entropy formulation we use is bounded between 0 and 1 [40], we
subtract it from 1 in order to minimize the term. Thus, the optimization
problem for the agent becomes finding the amount of sparsity for each layer
which would eventually lead to minimize the spatial entropy. In order to
compute the mean value (per layer) of the spatial entropy, we use the
convolutional outputs from 100 samples, which of course represents only an
estimate of the full dataset. Computing the mean spatial entropy using the
full dataset would be computationally too costly, and as such resorting to a
high smaller sample size is enough.

Our hypothesis is that by minimizing the spatial entropy, we can achieve on
par or better results than when the goal is to maximize the accuracy. If this
is the case, then we can establish an empirical connection between pruning and
information theory, showing that by removing redundant information from a
model we can achieve the same accuracy as when we directly try to maximize it.

##  5 Experiments

In this section we experimentally asses our hypothetical connection between
information theory and pruning, verifying whether pruning achieved via AutoML,
using minimization of spatial entropy for convolutional activations, can lead
to a more compact model with similar accuracy.

For training, we used the deep learning programming framework PyTorch [43]
(version 1.10.0) and the public implementation of AMC, modified to our needs.

We started by training a standard VGG-16 [11] on the CIFAR-10 dataset [42].
For that, we trained for 200 epochs using the SGD optimizer with a learning
rate of $0.01$ and cosine annealing scheduler [44].

In order to establish a baseline to compare our method with, we used the
original formulation of the AMC framework and optimize first the network using
the accuracy criterion. To achieve a certain level of pruning, AMC pushes up
the level of sparsity until only a predefined percentage of the total FLOPS
are maintained. The ratio between the number of FLOPS after compression and
the number of FLOPS before compression can measure indirectly the amount of
sparsity in a network. Table 1 depicts the results for various FLOPS
preservation percentages after fine-tuning on the CIFAR-10 dataset. For fine-
tuning we used the same training optimizer and hyperparameters as described
previously.

Table 1: Accuracy for a VGG-16 network using the original AMC framework for different FLOPS preservation percentage | Standard | VGG-16 with | VGG-16 with | VGG-16 with
---|---|---|---|---
| VGG-16 | 50% FLOPS | 20% FLOPS | 10% FLOPS
Accuracy | 93.58% | 93.85% | 93.26% | 92.18%
No. parameters | 14728266 | 4768242 | 912186 | 483402

Our intuition is that entropy minimization should be the key in removing
redundant information while preserving accuracy. Although, this should be the
case, we performed a sanity test to check whether the opposite is also true
(i.e., entropy maximization). We first performed some experiments for FLOPS
preservation of 50% and noticed that both entropy minimization and
maximization achieve comparable accuracy results (before fine-tuning) on the
mini validation set used for AMC agent training (Table 2).

Table 2: Accuracy on mini split during AMC agent training for entropy minimization/maximization and FLOPS preservation of 50% Minimization | Maximization
---|---
100% | 99.88%

Our next experiment used random rewards $\in[0,1]$ to check whether 50% FLOPS
preservation is not a very simple task for the agent. On the mini split
dataset used for agent training, we obtained an accuracy of $86\%$, concluding
that even a reward which makes no sense can result in a network with decent
performance. Therefore, we decided to use a scenario with 10% preservation
ratio in order to better stress test the framework and obtain more relevant
results. Table 3 presents the accuracies obtained on CIFAR-10 after fine-
tuning two networks obtained via AMC with entropy minimization/maximization
and FLOPS preservation of 10%. We notice that with entropy minimization we
achieved the same performance as when accuracy is used as a reward. The
solution found by this method has $10\times$ less FLOPS and $\approx 38\times$
less parameters than the original VGG-16 network. For entropy maximization the
framework produces a solution which has indeed fewer parameters, but uses the
same number of FLOPS as the method with entropy minimization. We can see
though that the resulting network architecture has a much poorer accuracy
performance.

Table 3: Experiments with entropy minimization and maximization and FLOPS preservation of 10%. The accuracies are computed on the CIFAR-10 test set after fine-tuning.  | Minimization | Maximization
---|---|---
Accuracy | 92.36% | 83.23%
No. parameters | 386442 | 91290

In the above experiments, we used a bin size of 256 for quantizing the
convolutional activations before computing the spatial entropy. Table 4 shows
results for entropy minimization when using various bin sizes. It can be
observed that our method is robust against a wide range of values, but the
fewer bins results in faster computation - as expected.

Table 4: Accuracy on CIFAR-10 test set after fine-tuning. FLOPS preservation of 10%, different bin size configurations | 32 bins | 64 bins | 128 bins | 256 bins | 512 bins
---|---|---|---|---|---
Accuracy | failed | 92.43% | 92.31% | 92.36% | 91.97%
No. parameters | failed | 511938 | 619962 | 386442 | 578922

The authors of [34] discovered that training from scratch a subnetwork
configuration found via pruning can produce better results than fine-tuning
the preserved weights. We verified this hypothesis with the architectures
found by entropy minimization/maximization and noticed that in neither case
training from scratch can produce better results (Table 5). In our case,
performance is better preserved when the weights are fine-tuned, because the
remaining weights are properly adjusted using calibration samples, as
described in Section 4.

Table 5: Accuracy on CIFAR-10 test set when training a subnetwork from scratch Acc. maximization | Acc. maximization | Acc. maximization | Entropy minimization
---|---|---|---
50% FLOPS | 20% FLOPS | 10% FLOPS | 10% FLOPS
92.92% | 92.18% | 90.99% | 91.06%

In order to test the generality of our method for various other architectures,
we repeated the same experiments for other popular networks: MobileNetV2 [45]
and ResNet50 [12]. The results are depicted in Table 6. Our method is on par
with the original AMC framework for various architectures and FLOPS
preservation percentages. The only noticeable drop in performance is for
ResNet50, which was previously observed to contain less redundancy [46] and
was the most difficult to compress, even when using accuracy as a criterion.

Table 6: Accuracy on CIFAR-10 test set with other architectures and various FLOPS preservation percentages Architecture | Original | Accuracy | Accuracy | Entropy | Entropy
---|---|---|---|---|---
| performance | 50% FLOPS | 20% FLOPS | 50% FLOPS | 20% FLOPS
MobileNetV2 |  94.58% | 94.62% |  93.59% |  94.31% | 93.75%
ResNet50 | 95.21% | 95.34% | 95.09% | 95.27% | 94.27%

In our last experiment, we compared our method with other popular pruning
methods. We used the ShrinkBench framework proposed in [6] to prune the same
VGG16 network that we used in our experiments. From this framework, we
selected the two most popular methods used for pruning: Global Magnitude
Weight, which places together weights from all the layers and selects the ones
with the largest magnitude and Local Magnitude Weight, which selects the
weights with the largest magnitudes on a per layer basis. Table 7 shows the
results obtained using ShrinkBench. It can be noticed that the accuracies
resulted when using this framework are similar to what we obtain using AMC
combined with our entropic criterion. One disadvantage of this method is that
the type of pruning is unstructured, having the risk that when the network
would be deployed to a specialized hardware, it might not reduce computational
costs as much as when structured pruning is employed.

Table 7: Accuracy for VGG16 on CIFAR-10 test set using the ShrinkBench framework Original | Global | Global | Local | Local
---|---|---|---|---
Accuracy | 50% FLOPS | 10% FLOPS | 50% FLOPS | 10% FLOPS
93.58% | 93.78% | 92.83% | 93.27% | 92.34%

Using an information-theoretical optimization criterion, which aims to
minimize entropy, we achieved the same performance as when we optimize
directly the accuracy of the model. We were able to reduce the total number of
FLOPS of a VGG-16 architecture by $10\times$ and the number of parameters by
$\approx 38\times$, while incurring minimal accuracy drop, with similar
results for other popular architectures.

##  6 Conclusions

A standard neural network’s output should ideally have a close to zero
entropic value in order to confidently predict a class - not necessarily the
correct one. Usually, this behavior is achieved by minimization of the cross-
entropy between the network’s output and the one hot encoding of the true
class. This task can be sometimes burdensome, because internal layers of the
network are not forced in any way to minimize the final entropy of the output
layer.

In our experiments, we explicitly forced the spatial entropy of internal
convolutional activations to decrease with the goal to achieve neural pruning.
According to our results, using the spatial entropy as an optimization
criterion in an AutoML pruning framework, we can achieve good performance for
an object recognition task, without directly optimizing the final evaluation
metric (accuracy in this case). Because of the overparametrization of a neural
network, removing unessential information via entropy minimization helps
reduce the model to its relevant (essential) components.

We established an interesting connection between information theory and neural
pruning. Our result creates the premises for future applications in neural
network pruning. In the future, we aim to explore novel ways in which the
entropy can be used for the optimization of neural architectures. One such
idea would be to include the entropy measure as a heuristic for selecting the
channels to be preserve, instead of using the well known L2 magnitude. Another
direction we aim to pursue is whether there is a link between pruning by
entropy and semiotic aggregation, as described in [4].

## References

  * [1] He, Y., Lin, J., Liu, Z., Wang, H., Li, L. & Han, S. AMC: AutoML for Model Compression and Acceleration on Mobile Devices. European Conference On Computer Vision (ECCV). (2018)
  * [2] He, X., Zhao, K. & Chu, X. AutoML: A survey of the state-of-the-art. Knowledge-Based Systems. 212 pp. 106622 (2021), https://www.sciencedirect.com/science/article/pii/S0950705120307516
  * [3] Yao, Q., Wang, M., Escalante, H., Guyon, I., Hu, Y., Li, Y., Tu, W., Yang, Q. & Yu, Y. Taking Human out of Learning Applications: A Survey on Automated Machine Learning. CoRR. abs/1810.13306 (2018), http://arxiv.org/abs/1810.13306
  * [4] Muşat, B. & Andonie, R. Semiotic Aggregation in Deep Learning. Entropy. 22 (2020), https://www.mdpi.com/1099-4300/22/12/1365
  * [5] Musat, B. & Andonie, R. Information Bottleneck in Deep Learning - A Semiotic Approach. International Journal of Computers Communications & Control, 17 (2022), http://univagora.ro/jour/index.php/ijccc/article/view/4650
  * [6] Blalock, D., Ortiz, J., Frankle, J. & Guttag, J. What is the State of Neural Network Pruning?. (2020)
  * [7] Gale, T., Elsen, E. & Hooker, S. The State of Sparsity in Deep Neural Networks. (2019)
  * [8] Deng, L., Li, G., Han, S., Shi, L. & Xie, Y. Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey. Proceedings Of The IEEE. 108, 485-532 (2020)
  * [9] Lillicrap, T., Hunt, J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D. & Wierstra, D. Continuous control with deep reinforcement learning. ICLR. (2016), http://dblp.uni-trier.de/db/conf/iclr/iclr2016.html#LillicrapHPHETS15
  * [10] Krizhevsky, A., Sutskever, I. & Hinton, G. ImageNet Classification with Deep Convolutional Neural Networks. Proceedings Of The 25th International Conference On Neural Information Processing Systems - Volume 1. pp. 1097-1105 (2012)
  * [11] Simonyan, K. & Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. 3rd International Conference On Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. (2015), http://arxiv.org/abs/1409.1556
  * [12] He, K., Zhang, X., Ren, S. & Sun, J. Deep Residual Learning for Image Recognition. 2016 IEEE Conference On Computer Vision And Pattern Recognition (CVPR). pp. 770-778 (2016)
  * [13] Tan, M. & Le, Q. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings Of The 36th International Conference On Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA.
  * [14] Oymak, S. & Soltanolkotabi, M. Toward Moderate Overparameterization: Global Convergence Guarantees for Training Shallow Neural Networks. IEEE Journal On Selected Areas In Information Theory. 1 pp. 84-105 (2020)
  * [15] Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. & Keutzer, K. A Survey of Quantization Methods for Efficient Neural Network Inference. ArXiv. abs/2103.13630 (2022)
  * [16] Lin, J., Rao, Y., Lu, J. & Zhou, J. Runtime Neural Pruning. Advances In Neural Information Processing Systems.
  * [17] Cubuk, E., Zoph, B., Mane, D., Vasudevan, V. & Le, Q. AutoAugment: Learning Augmentation Strategies From Data. Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition (CVPR). (2019,6)
  * [18] Ho, D., Liang, E., Chen, X., Stoica, I. & Abbeel, P. Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules. Proceedings Of The 36th International Conference On Machine Learning. 97 pp. 2731-2741 (2019,6,9), https://proceedings.mlr.press/v97/ho19b.html
  * [19] Zoph, B. & Le, Q. Neural Architecture Search with Reinforcement Learning. CoRR. abs/1611.01578 (2016), http://arxiv.org/abs/1611.01578
  * [20] Pham, H., Guan, M., Zoph, B., Le, Q. & Dean, J. Efficient Neural Architecture Search via Parameters Sharing. Proceedings Of The 35th International Conference On Machine Learning. 80 pp. 4095-4104 (2018,7,10), https://proceedings.mlr.press/v80/pham18a.html
  * [21] Liu, H., Simonyan, K. & Yang, Y. DARTS: Differentiable Architecture Search. CoRR. abs/1806.09055 (2018), http://arxiv.org/abs/1806.09055
  * [22] A.-C. Florea and R. Andonie, “Weighted Random Search for Hyperparameter Optimization”, International Journal of Computers Communications & Control, vol 14, pp. 154-169, 04 2019.
  * [23] R. Andonie and A.-C. Florea, “Weighted Random Search for CNN Hyperparameter Optimization”, International Journal of Computers Communications & Control, vol 15, no 2, 2020.
  * [24] Huang, Q., Zhou, K., You, S. & Neumann, U. Learning to Prune Filters in Convolutional Neural Networks. 2018 IEEE Winter Conference On Applications Of Computer Vision (WACV). pp. 709-718 (2018)
  * [25] Cubuk, E., Zoph, B., Shlens, J. & Le, Q. RandAugment: Practical Automated Data Augmentation with a Reduced Search Space. Advances In Neural Information Processing Systems.
  * [26] Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C. & Krishnan, D. Supervised Contrastive Learning. Advances In Neural Information Processing Systems. 33 pp. 18661-18673 (2020)
  * [27] Grill, J., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., Kavukcuoglu, K., Munos, R. & Valko, M. Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning. Advances In Neural Information Processing Systems. 33 pp. 21271-21284 (2020), https://proceedings.neurips.cc/paper/2020/file/f3ada80d5c4ee70142b17b8192b2958e-Paper.pdf
  * [28] Xie, Q., Luong, M., Hovy, E. & Le, Q. Self-Training With Noisy Student Improves ImageNet Classification. Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition (CVPR). (2020,6)
  * [29] Real, E., Aggarwal, A., Huang, Y. & Le, Q. Regularized Evolution for Image Classifier Architecture Search. (2018), https://arxiv.org/pdf/1802.01548.pdf
  * [30] Cai, H., Zhu, L. & Han, S. ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware. 7th International Conference On Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. (2019), https://openreview.net/forum?id=HylVB3AqYm
  * [31] LeCun, Y., Denker, J. & Solla, S. Optimal Brain Damage. Advances In Neural Information Processing Systems. 2 (1989)
  * [32] Hassibi, B., Stork, D. & Wolff, G. Optimal Brain Surgeon and general network pruning. IEEE International Conference On Neural Networks. pp. 293-299 vol.1 (1993)
  * [33] Frankle, J. & Carbin, M. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. International Conference On Learning Representations. (2019)
  * [34] Liu, Z., Sun, M., Zhou, T., Huang, G. & Darrell, T. Rethinking the Value of Network Pruning. International Conference On Learning Representations. (2019)
  * [35] Molchanov, P., Mallya, A., Tyree, S., Frosio, I. & Kautz, J. Importance Estimation for Neural Network Pruning. Proceedings Of The IEEE/CVF Conference On Computer Vision And Pattern Recognition (CVPR). (2019,6)
  * [36] Renda, A., Frankle, J. & Carbin, M. Comparing Rewinding and Fine-tuning in Neural Network Pruning. International Conference On Learning Representations. (2020)
  * [37] Evci, U., Gale, T., Menick, J., Castro, P. & Elsen, E. Rigging the Lottery: Making All Tickets Winners. Proceedings Of The 37th International Conference On Machine Learning. 119 pp. 2943-2952 (2020,7,13), https://proceedings.mlr.press/v119/evci20a.html
  * [38] E. Volden, G. Giraudon, and M. Berthod, “Modelling image redundancy,” in _1995 International Geoscience and Remote Sensing Symposium, IGARSS ’95. Quantitative Remote Sensing for Science and Applications_ , vol. 3, 1995, pp. 2148–2150.
  * [39] Shwartz-Ziv, R. & Tishby, N. Opening the Black Box of Deep Neural Networks via Information. CoRR. abs/1703.00810 (2017), http://arxiv.org/abs/1703.00810
  * [40] A. G. Journel and C. V. Deutsch, “Entropy and spatial disorder,” _Mathematical Geology_ , vol. 25, no. 3, pp. 329–355, 1993.
  * [41] Konda, V. & Tsitsiklis, J. Actor-Critic Algorithms. Advances In Neural Information Processing Systems. 12 (1999)
  * [42] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  University of Toronto, 05 2012.
  * [43] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems 32_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds. Curran Associates, Inc., 2019, pp. 8026–8037. [Online]. Available: http://papers.nips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * [44] Loshchilov, I. & Hutter, F. SGDR: Stochastic Gradient Descent with Warm Restarts. 5th International Conference On Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. (2017), https://openreview.net/forum?id=Skq89Scxx
  * [45] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. & Chen, L. MobileNetV2: Inverted Residuals and Linear Bottlenecks. (2018,6)
  * [46] Wang, Z., Li, C. & Wang, X. Convolutional Neural Network Pruning with Structural Redundancy Reduction. 2021 IEEE/CVF Conference On Computer Vision And Pattern Recognition (CVPR). pp. 14908-14917 (2021)
  * [47] Almog, A. & Shmueli, E. Structural entropy: monitoring correlation-based networks over time with application to financial markets. Scientific Reports. 9, 1-13 (2019)


Simultaneous PIV and LIF measurements in stratified flows using pulsed lasers

Paolo Luzzatto-Fegiz111<EMAIL_ADDRESS>

Department of Mechanical Engineering, University of California, Santa Barbara,
93106, USA

###### Abstract

We examine the problem of performing simultaneous and coplanar Particle Image
Velocimetry (PIV) and Laser-Induced Fluorescence (LIF) measurements in a
stratified fluid initially at rest. Our focus is on enabling detailed velocity
and density measurements in long internal waves and gravity currents, through
relatively small modifications of typical existing PIV systems comprising
pulsed lasers, using dye concentration as a proxy for fluid density. Several
issues have limited such measurements. These include: (1) variations in the
laser intensity and beam structure between laser pulses; (2) PIV particles
concentrating preferentially at their neutral buoyancy depth, thereby yielding
nonuniform dye illumination; and (3) the need to maintain a constant index of
refraction across large stratified fluid volumes. These issues can cause large
errors in the measured dye concentration, and therefore in the inferred
density. Here we focus on an experimental setup comprising a stratified layer
overlaying a deep homogeneous region. We produce long internal waves using a
lock-release setup, in order to investigate the structure of waves comprising
recirculating fluid regions (known as “trapped cores”), which are of current
interest in oceanographic applications. We maintain a short optical path and
use velocity information from PIV data to minimize index-of-refraction issues.
We exploit the fact that the system is initially at rest to devise a mapping
that links apparent and actual dye concentration, thus sidestepping nonuniform
illumination issues due to particle clustering. Finally, we devise a procedure
to correct for laser power variations along each ray in the sheet. We test our
technique for a small-amplitude wave, where the density can also be separately
calculated without resorting to LIF, finding good agreement. Furthermore, we
show results for large-amplitude waves, where our approach yields the first
measurements of the density field in long internal waves with trapped cores,
thereby providing detailed data for the development of theoretical models.

Particle Image Velocimetry (PIV) and Laser Induced Fluorescence (LIF) can
provide complementary information about a flow, as they may enable tracking
simultaneously the evolution of the velocity field, as well as the behaviour
of a scalar. Early experiments combing PIV and LIF appear to have focused
primarily on uniform-density flows (e.g. [1]), with the scalar behaving as a
passive tracer. When considering density-stratified flows, the scalar can in
principle be used as a proxy for the fluid density, by defining a suitable
calibration. However, the introduction of density gradients brings several new
issues, which must be resolved in order to perform accurate measurements of
the scalar field.

Figure 1: Sketch of the PIV particle concentration in a stratified fluid,
leading to enhanced illumination near the depth at which the particles are
neutrally buoyant.

A first issue involves optical distortions associated with density gradients
(for example, if a straightforward salt stratification is used), since the
resulting scalar measurements are affected in two ways. Firstly, errors are
introduced throughout the imaged field, as the optical path through which each
fluid particle is imaged depends on time; this error can be estimated through
a ray-tracing exercise, and can often be limited by maintaining a short
optical path. A second problem can arise when sharp spatial gradients in
refractive index can reshape the light pattern used for illumination. When
vortex shedding or other relatively small-scale features are involved, this
results in characteristic ‘streaks’ appearing in the scalar field [2]. Several
experiments have made use of refractive-index-matching techniques to minimize
these optical distortions (for example, by using glycerol and potassium
phosphate [3] or salt and isopropyl alcohol [4]); however, index matching
becomes more expensive and technically challenging as the scale of the
facility increases. [2] sidestepped the need for index matching by maintaining
relatively short optical paths, and applying a Fourier filter to their scalar
fields, thereby effectively removing the streaks. However, this came at the
cost of also removing some of the smaller features in the scalar field.

A second issue, when performing simultaneous PIV and LIF in stratified fluids,
is associated with the tendency of PIV particles to concentrate preferentially
near the depth where they would be neutrally buoyant, as sketched in figure 1.
Since the particles scatter light, this leads to stronger effective
illumination in regions of higher particle density, resulting in a locally
higher dye fluorescence, and therefore in greater apparent dye concentration,
thereby introducing an error into the measurement. Note that even a camera
with a filter that would pass only the fluorescence wavelength would be
affected by this error. This is a consequence of the fact that the additional
scattered light is absorbed by the dye, resulting in greater fluorescence.
Therefore the camera receives an increased amount of light that follows the
expected fluorescence spectrum of the dye. By contrast, any issues associated
with light that reaches the camera directly after being scattered by the
particles could be removed by a suitable filter.

[5] addressed this issue by introducing an additional calibration, which
essentially links particle concentration to enhancement in illumination.
Implementing this technique requires measuring the particle concentration
throughout the experiment and establishing this additional calibration.
Furthermore, stratified flow experiments often use particles that are not
monodisperse, to help ensure that the full range of depths are seeded (since
different-sized particles can retain different amounts of gas on their
surface, leading to a broader effective range of specific gravities; see e.g.
[6]). This would introduce, as an additional complication, the need to account
also for particle size in Law & Wang’s calibration.

Figure 2: Sketch of the correction, introduced in [7], to account for time-
dependent nonuniformities in the laser sheet. The approach was developed for a
dyed jet, flowing into a clear ambient. Only rays traversing regions of known
concentration (here, near the jet source) can be corrected.

Choosing how to illuminate the particles and dye has also been a source of
complications when performing combined PIV and LIF. Pulsed lasers are often
used to perform PIV measurements; however, they can give significant
variations in power output between shots, resulting in inaccurate dye
concentration measurements [8]. On the other hand, continuous lasers offer
more stable output, but their lower intensity might limit the resolution in
PIV measurements. A comprehensive and effective solution involves combining
two lasers into a single optical path, and switching between the two (e.g.
[9]). While very effective, this solution brings substantial additional
complexity; furthermore, it does not address the two issues described earlier,
involving index of refraction and increased dye illumination. Another
(possibly practically simpler, as well as cheaper) choice of illumination
involves using only one laser. A continuous-beam laser offers best performance
for LIF measurements (while placing some limits on the resolution of the
velocity measurements). On the other hand, since pulsed lasers are often used
in PIV setups, it is interesting to look for a relatively simple extension to
also enable accurate LIF with these systems. One can meter the variation in
output between pulses, and correct the LIF data accordingly [8]. However,
shot-to-shot variations are not necessarily limited to intensity, but can also
affect beam structure, resulting in time-dependent nonuniformities in the
resulting light sheet. In principle, one could account for these spatial
variations by separately applying a correction to each ray composing the
sheet. However, one must be able to calculate the correction for each ray.
This can be achieved for rays that are traversing regions where the actual dye
concentration is known at all times. [7] developed this approach for a dyed
jet flowing into a clear ambient fluid, as sketched in figure 2. In this
example, the jet source is at a known concentration. Note that a persistent
limit of this approach is that rays that do not traverse any regions of known
concentration cannot be corrected (as shown in figure 2).

In summary, the issues discussed above, which affect simultaneous PIV and LIF
measurements in stratified fluids, using pulsed lasers, are:

  * •

Time- and space-dependent variations in the light sheet;

  * •

Preferential clustering of PIV particles near their neutral buoyancy depth,
leading to locally higher dye fluorescence; and

  * •

Index-of-refraction variations, affecting both illumination and data
acquisition.

Figure 3: Sketch of the overall experimental setup (not to scale).

In this paper, we develop solutions to address each of these issues, for an
experiment investigating the structure of long internal waves. In section 1 we
describe our experimental setup. Section 2.1 explains how we address power
fluctuations, whereas section 2.2 describes how we minimize errors due to
scatter from PIV particles, as well as from index-of-refraction variations.
Section 3 shows examples involving internal solitary wave measurements.
Finally, in section 4 we draw conclusions from the use of the overall
technique.

##  1 Experimental Setup

The overall tank length, width and depth are 8 m $\times$ $0.4\,$m $\times$
0.5 m, respectively (a sketch of the setup is shown in figure 3). In all
experiments, a deep and uniform layer of saltwater was covered with a shallow
stratified layer of height $h$, yielding an overall depth $H$. An
approximately linear stratification was produced using the two-bucket method
[10], using mixtures of freshwater and filtered seawater. The background
profile was measured using a conductivity-temperature probe from Precision
Measurement Engineering [11]; density was calculated using the Seawater
library, version 3.3.

Immersion pumps were used to help remove dissolved gases from the water
reservoirs, to minimize opportunities for bubble formation on the walls or on
the conductivity probe. Before each experiment, we waited for the water to be
in thermal equilibrium with the room (the immersion pumps proved useful in
initially raising the temperature of the water closer to that of the room).

In all experiments reported here, $h/H\simeq 0.15$, whereas the total depth
$H\simeq 0.35$ m. The internal buoyancy frequency $N$, defined by

$N^{2}=-\frac{g}{\rho}\frac{\partial\rho}{\partial z}$ (1)

had a typical value, in the stratified layer, of $N_{0}\simeq 1.5$ s-1 (here
$g$ is the acceleration due to gravity, $\rho$ is the density, and $z$ is the
vertical (upward pointing) direction. While experimental Reynolds numbers in
the literature are often based on overall depth $H$, for small $h/H$ the most
important length scale is arguably $h$. For a wave with phase velocity $c$, we
therefore define a Reynolds number $Re_{h}=ch/\nu$, where $\nu$ is the
kinematic viscosity. In our experiments, $Re_{h}\simeq 4,000$. Waves are
produced using a classic lock-release arrangement [12, 13] at one end of the
tank. Once the background stratification has been set up, a gate is introduced
(approximately 0.3 m from one end of the tank), and additional liquid (closely
matching the surface density) is added. Removing the gate generates a surface-
propagating gravity current, whose front radiates long internal waves.

We image two downstream regions, located approximately 3.2 m and 6.2 m from
the end of the tank at which the lock-release mechanism is located. We acquire
velocity data in a single plane by means of a particle image velocimetry (PIV)
setup, comprising an Nd:Yag pulsed laser (operating at 532 nm) and CCD cameras
with resolution $2048\times 2048$ (LaVision). The field of view is
approximately 0.35 m wide, and is located in a plane that is about $0.07$ m
away from the side of the tank, to minimize side-wall effects, while
maintaining a relatively short optical path to reduce optical distortion. The
flow is seeded with Vestosint particles with a nominal size of 15 $\mu$m; a
relatively high seeding density is used, thus enabling calculation of
independent velocity vectors over a length scale of about $3\times 10^{-3}$m.
The typical data acquisition rate is 7 Hz; velocity fields are calculated
using DaVis 7.2.

To also measure density fields that are co-planar with the velocity data, we
performed simultaneous planar laser-induced fluorescence (LIF), using
Rhodamine WT as the fluorescent dye. The dye is added to either the saltwater
or freshwater reservoir before the experiment (this choice is guided by the
analysis developed in section 2.2). In order to neutralize any chlorine in the
freshwater (which would bleach the dye), we introduce 12 PPM (by weight) of
ascorbic acid crystals in the freshwater reservoir. The PIV camera is equipped
with a notch filter, together with a polarized filter to reduce light from
reflections. The LIF camera uses a 570 nm high-pass filter to avoid capturing
any light directly scattered by the PIV particles. Following [14], we warm up
the lasers for approximately 20 minutes before each experiment. The power
setting for the laser was determined through preliminary experiments to ensure
that, in the field of view, the fluorescent response was linear with respect
to dye concentration (at least up to a maximum concentration of 10 PPB).

Recent work has shown that, in the presence of a free surface, solitary
internal waves can be drastically affected by Marangoni effects developing at
the air-water interface [15]. For this reason, we perform experiments with a
no-slip boundary at the top, which is introduced shortly after the tank has
been filled, using styrofoam panels whose width matches precisely that of the
tank. We were careful to ensure that the panels had no seams or joins at least
1 m upstream and downstream of each imaging window, and did not have
appreciable gaps elsewhere (aside from the ends of the tank). A small opening
was cut for lowering the conductivity-temperature probe. The panels were
lowered carefully until fully wet, and were held in place rigidly. In order to
enhance wetting and minimize bubbles, we treated the panels with a weak
solution of dechlorinated water and wetting agent (0.2% Photo-Flo by volume),
which was allowed to dry before the panels were lowered.

Once a set of images was acquired, and the laboratory-frame velocities $(u,w)$
had been calculated, the phase velocity $c$ was estimated by tracking the
$x$-location of the wave trough. To achieve this, we followed the horizontal
motion, at fixed depth, of the point at which $w(x,z=z_{0})=0$, where $z_{0}$
is the depth corresponding to the lower edge of the stratified layer. The
uncertainty in $c$ is estimated (by bootstrapping) to be less than 1%. The
amplitude $\eta_{\mathrm{pycno}}$ is defined as the displacement of the
isopycnal at the lower end of the stratification (relative to initial
condition), and is measured either by first computing streamlines in the
moving frame (for small-amplitude waves, without instabilities or trapped
cores) or by tracing the appropriate isopycnal in the density field. To obtain
an overall view of the waves, we construct composite views, by shifting
successive images horizontally by a distance $c\Delta t$ (where $\Delta t$ is
the time between images), interpolating to a common grid, and averaging data
from overlapping image regions.

##  2 Performing simultaneous PIV and LIF measurements in stratified flows
using pulsed lasers

###  2.1 Accounting for nonuniform power fluctuations

In this section, we consider the effect of power fluctuations, in the absence
of PIV particles. Figure 4($a$) shows an image taken with a sample of uniform
density and dye concentration. In figure 4($b$), we plot the fluctuation in
the measured fluorescence, for a small region of $6\times 6$ pixels (in blue),
as well as for the whole field of view (in red). While local and global
fluctuations exhibit some correlation, they can also be markedly different.
This shows the need for calculating and applying a local correction (that is,
one that can be applied along each ray).

Figure 4: Fluctuation in laser power over time. ($a$) Measured fluorescence
for a sample of uniform 10 PPB dye concentration. ($b$) time traces for
average measurements over a $6\times 6$ pixels window (blue) and over the
whole field (red).

In order to avoid having to correct for dye absorption, we employ relatively
low maximum dye concentrations (approximately 10 PPB). If the laser power is
steady, the measured intensity $I$ at each CCD pixel (with horizontal and
vertical indices $i,j$) is related to the concentration field $C$ for each
image $n$ by [16]

$I_{n}(i,j)=\alpha(i,j)\,C_{n}(i,j)+I_{0}(i,j),$ (2)

where $\alpha(i,j)$ is an empirically constructed calibration matrix, and
$I_{0}(i,j)$ is the background level for each pixel (which can be acquired by
taking a series of images with the lens cap on). If the illumination is time-
dependent, such that the laser intensity changes with each image $n$, we write

$I_{n}(i,j)=\alpha(i,j)\,a_{n}(\theta)\,C_{n}(i,j)+I_{0}(i,j),$ (3)

where $a_{n}(\theta)$ is a (time-dependent) factor capturing the power
variation with azimuthal angle $\theta$ across the sheet. Without loss of
generality, we define $a_{n}(\theta)$ such that its long-term average is 1,
that is

$\bar{a}(\theta)=\lim_{N\rightarrow\infty}\frac{1}{N}\sum_{n=1}^{N}a_{n}(\theta)\equiv
1,$ (4)

Therefore we evaluate $\alpha(i,j)$ by taking $N$ images at a known reference
concentration $C(i,j)=C_{ref}$, and averaging (3) to get

$\alpha(i,j)=\frac{\bar{I}(i,j)-I_{0}(i,j)}{C_{ref}}$ (5)

where the overbar denotes the average over $N$ images (in practice, we used
$N=100$).

As explained in the introduction, in order to correct for power fluctuations,
we need to ensure that each ray traverses a region with known dye
concentration. In our problem, the deeper portions of the tank remain at
constant concentration throughout the experiments. For this reason, we choose
to set up the experiments such that the maximum dye concentration is reached
in the bottom layer. In practice, we achieve this by dyeing the saltwater
reservoir, and leaving the freshwater reservoir clear. Therefore, in the
bottom layer, the concentration is known, that is $C(i,j)=C_{bot}$, as
sketched in figure 5($a$) (for completeness, a sketch of the corresponding
implementation for the case of a jet shown earlier in figure 2 is sketched in
figure 5$b$). Therefore, for each image $n$, we can compute, along each ray
(characterized by azimuthal angle $\theta$):

$a_{n}(\theta)=\frac{I_{n}(i,j)-I_{0}(i,j)}{\alpha(i,j)\;C_{bot}},\quad\quad(i,j)\text{\;in\;bottom\;layer.}$
(6)

Once $a_{n}(\theta)$ has been computed, we can therefore estimate $C_{n}(i,j)$
as

$C_{n}(i,j)=\frac{I_{n}(i,j)-I_{0}(i,j)}{\alpha(i,j)\;a_{n}(\theta)}.$ (7)

Figure 5: ($a$) Sketch of the power correction approach used in our
experiment: we ensure that the bottom layer is at a known concentration,
thereby enabling us to correct the complete field of view. ($b$) A possible
extension to problems involving jet flows (as sketched previously in figure
2); here the ambient fluid is dyed, whereas the jet is clear (or at a
different known concentration). Low maximum concentrations must be used to
minimize attenuation (the light path could also be shortened by making it
vertical in this figure, so as to further minimize attenuation).

###  2.2 Correcting the effect of PIV particles on measured fluorescence,
while minimizing streaks

In the absence of stratification, one might sometimes assume that particles
are uniformly distributed, and simply perform the calibration (5) in the
presence of particles. However, in stratified flows, particles will migrate
toward their neutral buoyancy depth, making this technique not applicable. One
solution, mentioned in the introduction, involves deriving calibration curves
as a function of particle and dye concentrations [5]. However, in order to use
this technique one needs to estimate the particle concentration, which varies
across the field. We suggest a simpler alternative, which is well-suited to
the stratified flows considered here. The key assumption behind the technique
is that both the PIV particles, as well as the dye, behave approximately as
Lagrangian tracers (that is, any diffusion taking place between the
calibration and the experiment is negligible). Our procedure is constructed as
follows.

Figure 6: ($a$) An example showing $\beta$, the illumination enhancement due
to a local excess of PIV particles, as a function of dye concentration $C$,
illustrating the fact that $\beta$ can be well-approximated by a function of
$C$ in these problems. ($b$) Relationship between $C\;\beta(C)$ and $C$,
showing that the mapping is invertible, but becomes singular as one approaches
$C=C_{bot}$ (since $d[C\;\beta(C)]/dC\rightarrow 0$ here). Note that the flat
slope at large $C$ is due to the fact that $C$ is uniform in the lower layer,
in our experiment. Figure 7: Schematic illustration for the procedure used to
continue the upstream density profile (which is known) into the wave region,
for steady waves. Since for steady flow isopycnals coincide with streamlines,
the streamfunction calculated from PIV data is used for this purpose. This
procedure cannot be applied to obtain the density in the core interior, where
streamlines are disconnected from the upstream region.

Firstly, we calculate $\alpha(i,j)$, using a solution of uniform
concentration, without any PIV particles, using (5). We assume that the
relationship between measured fluorescence and dye concentration can be
written as

$I_{n}(i,j)=\alpha(i,j)\,a_{n}(\theta)\,\beta_{n}(i,j)\,C_{n}(i,j)+I_{0}(i,j),$
(8)

where $\beta_{n}$ depends on the local particle concentration. We now describe
a procedure to calculate $\beta_{n}$. Once the tank has been filled, before
running an experiment, we measure the density profile $\rho_{init}(z)$ (where
$z$ is the vertical direction) using a conductivity-temperature probe. Since,
by construction, $\rho$ and $C$ are linearly related, we have $C_{init}(z)$
from

$C_{init}(z)=\frac{\rho_{init}(z)-\rho_{bot}}{\rho_{top}-\rho_{bot}}(C_{top}-C_{bot})+C_{bot}.$
(9)

In this initial state, we may reasonably assume that the PIV particle
concentration is only a function of $z$ (since particle migration is due to
buoyancy, and occurs vertically), such that
$\beta_{init}(i,j)=\beta_{init}(z)$. Furthermore, since in the stratified
layer $C_{init}$ changes monotonically with $z$, we may invert the
$C_{init}(z)$ relation and write $z=z_{init}(C)$, which in turn enables us to
write, in this region:

$\beta(i,j)=\beta_{init}{(z)}=\beta_{init}{(C)},$ (10)

such that, at this initial time, $\beta$ can be calculated as (after averaging
equation 8 over $N$ images, with the fluid at rest)

$\beta_{init}(C_{init})=\frac{\bar{I}(i,j)-I_{0}(i,j)}{\alpha(i,j)\;C_{init}(z)}.$
(11)

Note that this requires that $C>0$ everywhere in the field of view, which is
easily achieved by appropriately dyeing the reservoirs (in our experiments,
$C$ was small but finite at the surface).

At this stage we make use of the key assumption introduced earlier in this
section, namely that PIV particles and dye will both behave as Lagrangian
tracers. Since $\beta$ is affected only by the local particle concentration,
this implies that (10) will remain applicable throughout the evolution of the
flow, such that $\beta{(i,j)}=\beta_{init}(C_{init})\equiv\beta(C)$ for all
times.

At each instant in time, we now calculate the power correction using a
slightly modified version of (6), which now includes $\beta$

$a_{n}(\theta)=\frac{I_{n}(i,j)-I_{0}(i,j)}{\alpha(i,j)\;\beta(C_{bot})\;C_{bot}},\quad(i,j)\text{\;in\;bottom\;layer.}$
(12)

We can use this to calculate the product of $C(i,j)$ and $\beta(C(i,j))$:

$\beta(C_{n}(i,j))\;C_{n}(i,j)=\frac{I_{n}(i,j)-I_{0}(i,j)}{\alpha(i,j)\;a_{n}(\theta)},$
(13)

where all quantities on the right-hand-side are known.

A somewhat technical (but practically important) requirement to extract
$C(i,j)$ is that the mapping relating $\beta(C)\,C$ and $C$ must be invertible
(that is, we need $d[\beta(C)\,C]/dC\neq 0$). Note that $\beta$ corresponds,
essentially, to an amplification factor associated with the presence of
particles; if these are neutrally buoyant at a level inside the stratified
layer, $\beta$ will have a maximum inside the layer. Therefore, to ensure
invertibility, $C$ must change sufficiently rapidly across the layer. Since
$\beta(C)$ is known from (11), we store the relation $\beta(C)\,C$ versus $C$
in a lookup table, and we readily extract $C$ from the values of $\beta(C)C$
found using (13). Since, in order to construct this mapping, the dye and
particles must follow Lagrangian trajectories, we propose referring to this
technique as a “Lagrangian correction”. Finally, the density field
$\rho_{strat}$ in the stratified layer is found from $C$ as

$\rho_{strat}(C)=\frac{C-C_{bot}}{C_{top}-C_{bot}}(\rho_{top}-\rho_{bot})+\rho_{bot}.$
(14)

An example from our data, showing $\beta$, as well as $\beta(C)\,C$ versus
$C$, is shown in figure 6. Note that, at the edge between the stratified
region and the deep, homogeneous layer (corresponding to the maximum $C$),
$\beta(C)\,C$ changes weakly with $C$, meaning that this relationship cannot
be inverted reliably. This implies that our Lagrangian correction is not
applicable in this transitional layer. To address this issue, we exploit the
fact that the background density profile is known, and that the flow will
revert to the background (initial) state sufficiently far ahead of the wave.
Furthermore, at the edge of the stratification, and further below (away from
the bottom boundary layer) the flow is essentially two-dimensional (regardless
of any instabilities in the stratified layer), and streamlines can be computed
reliably from PIV in our experiments (as assessed by comparing streamlines
obtained using different paths of integration).

Figure 8: Example showing results before and after our correction has been
applied. ($a$): the dashed line shows the initial (background) density
profile, measured using the conductivity-temperature probe. The continuous
line corresponds to the density profile at the wave trough, calculated from
the LIF data, without applying a correction for the presence of PIV particles.
An unphysical overturn is present near the surface. ($b$) illustrates the
results after applying our correction (continuous line). Since this wave is
stable and does not have a trapped core, in this example we can use
streamlines (from PIV) together with the initial profile to separately
calculate the density at the wave trough (dot-dashed line), thereby validating
our Lagrangian correction. Figure 9: Density fields calculated without (left)
and with (right) application of the Lagrangian correction. This field
corresponds to the trough measurements shown earlier in figure 8. Each field
is an average of three images.

If the flow in this bottom region is near-steady, then streamlines (in the
frame of reference moving with the wave) will coincide with contours of
constant density (note that, if this were not the case, there would be a
velocity component normal to the density contours, which would result in a
time-varying shape, in the frame of reference translating with the wave).
Therefore the density field in the deep region, starting below the stratified
layer can be found as

$\rho_{deep}(x,z)=\rho_{deep}(\psi(x,z))=\rho_{init}(\psi_{init}(z)),$ (15)

where $\psi$ is the streamfunction in the frame translating with a wave with
phase velocity $c$, such that it is related to the fixed-frame streamfunction
by $\psi=\psi_{fixed}-cz$, and $\psi_{init}(z)=-cz$. In practice, we use
$\psi_{init}$ and $\rho_{init}$ to construct a table that maps $\psi$ to
$\rho$, then for each value of $\psi(x,z)$ in the flow field we use this table
to interpolate the corresponding $\rho(x,z)$. A sketch of this procedure is
shown in figure 7.

Finally, we combine $\rho_{deep}$ and $\rho_{strat}$ using a weight-function
$W$, such that

$\displaystyle\rho(x,z)$ $\displaystyle=$
$\displaystyle[1-W(\rho_{strat}(x,z))]\;\rho_{strat}(x,z)$ (16)
$\displaystyle+$ $\displaystyle W(\rho_{strat}(x,z))\;\rho_{deep}(x,z),$

where we set

$W(\rho_{strat})=\frac{1}{2}\left\\{1+\text{erf}\left(\frac{\rho_{strat}-\rho_{switch}}{\rho_{scale}}\right)\right\\},$
(17)

where we choose $\rho_{scale}=0.1(\rho_{bot}-\rho_{top})$ and
$\rho_{switch}=0.993\,\rho_{bot}$. An additional benefit of inferring density
from PIV data, in this deep region, is that any streaks in the LIF data (which
would originate with instabilities in the stratified layer) do not affect the
deep layer density field. Therefore this approach also helps to mitigate some
index-of-refraction effects.

Figure 10: Vorticity (upper panel of pair) and density (lower panel of pair)
fields for waves without and with trapped cores (from [15]). Core regions are
drawn in ($b$). Right-hand panels show vertical slices at the wave trough;
background conditions are marked by dashed lines. Note that these waves
propagate from left to right.

An example illustrating the various steps described above is shown in figure
8. Here, for the sake of testing the procedure, we choose a small-amplitude
wave, whose velocity field is essentially two-dimensional everywhere. Panel
($a$) shows the initial density profile (dashed line), as well as the
estimated density profile at the wave trough, calculated using a LIF procedure
that neglects the effect of PIV particles (which corresponds to setting
$\beta=1$ everywhere). Note that, near the surface, this would erroneously
indicate an overturn, with a density value below the lowest density introduced
in the system! In panel ($b$), the density profile estimated by our technique,
featuring the Lagrangian correction, is shown by the black continuous line.
Note that no overturns are present. Figure 9 shows the corresponding density
fields, calculated without (left) and with (right) the Lagrangian correction.
The two results are clearly markedly different.

Since this wave is essentially two-dimensional, in this specific example
$\rho$ can be found reliably from the PIV at all depths. The corresponding
density profile is shown by the dot-dashed line in figure 8($b$). This
compares very favorably with the profile found using our Lagrangian correction
on the LIF data, thereby providing a check on the accuracy of our procedure.
However, our corrected LIF technique is also applicable to flows where the
density may not be inferred exclusively from the PIV data, as shown in the
next section.

##  3 Examples involving regular and trapped-core waves

As a further illustration of our technique, here we briefly show two examples,
involving both regular and trapped-core waves (the flow physics associated
with these results are discussed in detail in [15]). Figure 10($a$) shows a
smaller-amplitude solitary wave. The top-left panel corresponds to the
vorticity $\omega=\partial u/\partial z-\partial w/\partial x$, whereas the
panel below shows the density field (where the rescaled density $\rho^{*}$
ranges between zero and one). The right-hand panels show profiles of vorticity
and density at the wave trough; dashed lines indicate the initial profiles.

Figure 10($b$) displays similar data to figure 10($a$), but for a trapped-core
wave. The closed contours, in the left-hand panel, show the trapped core
region (this is identified as a “Lagrangian coherent structure”, by examining
the rate at which neighboring particles separate [15]). As highlighted by the
right-hand panels, we discover that the trapped-core region has an
approximately uniform density (and vorticity), thereby providing essential
information for the development of oceanographic models of these waves. In
further work, we have used the data obtained through our combined PIV/LIF
technique to also evaluate other wave properties, including kinetic and
potential energy, as well as to test stability theories for stratified flows
[15].

##  4 Conclusions

In this paper, we address three issues affecting simultaneous PIV and LIF in
stratified flows. These issues comprise (1) time- and space-dependent
variations in the laser sheet; (2) preferential clustering of PIV particles
near their neutral buoyancy depth, leading to locally higher dye illumination
and fluorescence; and (3) index-of-refraction variations, affecting both
illumination and data acquisition. Our experiments consist of large-amplitude
solitary internal waves, propagating in a stratification consisting of a thin
approximately linearly stratified region overlaying a deep uniform-density
layer.

Issue (1) is addressed by exploiting the fact that, in our experiments, the
dye concentration does not change in the deeper layer, thereby allowing us to
calculate an instantaneous power correction, for each ray, using a reference
fluorescence (based on a long-term average) from the lower layer. In order to
deal with issue (2), we devise an additional correction technique, based on
the assumption that, in the stratified layer, both PIV particles and
fluorescent dye behave in a purely Lagrangian manner. In the deeper layer, we
use streamlines from the PIV (together with the initial density profile) to
calculate the density field. This last step, together with a setup that
minimizes the optical path in the tank, also helps to contain the effects of
index-of-refraction variations.

We test the overall technique by considering a small-amplitude, nonbreaking
wave example, where the density field can also be obtained solely by using
streamlines from PIV, thereby providing a direct comparison for the density
extracted from the LIF. The two data sets are in good agreement. Finally, we
show vorticity and density fields for larger amplitude waves, including a wave
with a trapped core, where density could not be obtained exclusively using
streamlines from PIV. Our technique has provided, to the best of our
knowledge, the first experimental measurements of density fields for long
internal waves with trapped cores. A full discussion of these results is
presented in a separate paper [15].

We should clarify that some of the correction steps presented here are not
always applicable; for example, if diffusion of dye is significant during the
experiment (as may be the case, for example, in a flow dominated by
turbulence), the Lagrangian assumption will not hold, and the correction for
the presence of PIV particles will not be feasible. In those cases, one may
pursue an alternative approach, such as simultaneous PIV and synthetic
schlieren (see [17]; note that the schlieren measurements implicitly report a
density average across the lateral extent of the tank, rather than being
coplanar with the PIV).

However, it is worth noting that the corrections introduced here require no
additional investment or complications in the experimental setup. In fact, our
approach eliminates the need for a power meter (which would, in any case, not
enable an accurate power correction across the laser sheet), and the
additional effort associated with our corrections is applied almost
exclusively at the postprocessing stage. We expect that the approach described
here may prove valuable in other problems involving stratified flows.

We gratefully acknowledge support by the Postdoctoral Scholar Program at the
Woods Hole Oceanographic Institution, the Devonshire Foundation, and Churchill
College, Cambridge. We are grateful to Anders Jensen for his work on the
construction of the wave tank, and for his assistance with running the
experiments.

## References

  * 1. S. Simoens and M. Ayrault.  Concentration flux measurements of a scalar quantity in turbulent flows.  Exp. Fluids, 16:273–281, 1994.
  * 2. A. Petracci, R. Delfos, and J. Westerweel.  Combined piv/lif measurements in a rayleigh-benard convection cell.  In 13th international symposium on applications of laser techniques to fluid mechanics, Lisbon, Portugal, 2006.
  * 3. A. Alahyari and E. K. Longmire.  Particle Image Velocimetry in a Variable-Density Flow - Application to a Dynamically Evolving Microburst.  Exp. Fluids, 17(6):434–440, October 1994.
  * 4. C. D. Troy and J. R. Koseff.  The generation and quantitative visualization of breaking internal waves.  Exp. Fluids, 38(5):549–562, 2005.
  * 5. A. W. K. Law and H. W. Wang.  Measurement of mixing processes with combined digital particle image velocimetry and planar laser induced fluorescence.  Exp. Thermal Fluid Sci., 22:213–229, 2000.
  * 6. M Carr, P A Davies, and R P Hoebers.  Experiments on the structure and stability of mode-2 internal solitary-like waves propagating on an offset pycnocline.  Phys. Fluids, 27(4):046602, 2015.
  * 7. F. Guillard, R. Fritzon, J. Revstedt, C. Tragardh, M. Alden, and L. Fuchs.  Mixing in a confined turbulent impinging jet using planar laser-induced fluorescence.  Exp. Fluids, 25(2):143–150, 1998.
  * 8. J. E. Martin and M. H. Garcia.  Combined PIV/PLIF measurements of a steady density current front.  Exp. Fluids, 46(2):265–276, 2008.
  * 9. C. Fukushima, L. Aanen, and J. Westerweel.  Investigation of the mixing process in an axisymmetric turbulent jet using PIV and LIF.  In Laser techniques for fluid mechanics, pages 1–11, Lisbon, Portugal, 2002.
  * 10. M. Economidou and G. R. Hunt.  Density stratified environments: the double-tank method.  Exp. Fluids, 46(3):453–466, 2008.
  * 11. M. J. Head.  The use of miniature four-electrode conductivity probes for high resolution measurement of turbulent density or temperature variations in salt-stratified water flows.  PhD thesis, UC San Diego, 1983.
  * 12. J. Grue, A. Jensen, P. O. Rusas, and J. K. Sveen.  Breaking and broadening of internal solitary waves.  J. Fluid Mech., 413:181–217, 2000.
  * 13. B. R. Sutherland, K. J. Barrett, and G. N. Ivey.  Shoaling internal solitary waves.  J. Geophys. Res.-Oceans, 118(9):4111–4124, 2013.
  * 14. L. A. Melton and C. W. Lipp.  Criteria for quantitative PLIF experiments using high-power lasers.  Exp. Fluids, 35(4):310–316, 2003.
  * 15. P. Luzzatto-Fegiz and K. R. Helfrich.  Laboratory experiments and simulations for solitary internal waves with trapped cores.  J. Fluid Mech., 757:354–380, 2014.
  * 16. J. P. Crimaldi.  Planar laser induced fluorescence in aqueous flows.  Exp. Fluids, 44(6):851–863, 2008.
  * 17. S. B. Dalziel, M. Carr, J. K. Sveen, and P. A. Davies.  Simultaneous synthetic schlieren and PIV measurements for internal solitary waves.  Meas. Sci. Tech., 18(3):533–547, January 2007.