# Brevity is the soul of wit: Pruning long files for code generation

Aaditya K. Singh<EMAIL_ADDRESS>
Gatsby Unit, UCL Yu Yang<EMAIL_ADDRESS>
UCLA Kushal Tirumala<EMAIL_ADDRESS>
Meta FAIR Mostafa Elhoushi<EMAIL_ADDRESS>
Meta FAIR Ari S. Morcos<EMAIL_ADDRESS>
DatologyAI

###### Abstract

Data curation is commonly considered a “secret-sauce” for LLM training, with
higher quality data usually leading to better LLM performance. Given the scale
of internet-scraped corpora, data pruning has become a larger and larger
focus. Specifically, many have shown that de-duplicating data, or sub-
selecting higher quality data, can lead to efficiency or performance
improvements. Generally, three types of methods are used to filter internet-
scale corpora: embedding-based, heuristic-based, and classifier-based. In this
work, we contrast the former two in the domain of finetuning LLMs for code
generation. We find that embedding-based methods are often confounded by
length, and that a simple heuristic—pruning long files—outperforms other
methods in compute-limited regimes. Our method can yield up to a 2x efficiency
benefit in training (while matching performance) or a 3.5% absolute
performance improvement on HumanEval (while matching compute). However, we
find that perplexity on held-out long files can increase, begging the question
of whether optimizing data mixtures for common coding benchmarks (HumanEval,
MBPP) actually best serves downstream use cases. Overall, we hope our work
builds useful intuitions about code data (specifically, the low quality of
extremely long code files) provides a compelling heuristic-based method for
data pruning, and brings to light questions in how we evaluate code generation
models.

Keywords: large language models, data pruning, code generation, data quality

##  1 Introduction

Large language models (LLMs) trained on internet-scale corpora showcase
impressive capabilities (Brown et al., 2020). With their increasing
prevalence, one of the main paradigms has become training models on more and
more data (Touvron et al., 2023a). In fact, some go on to argue that the
biggest factor in modern LLM training pipelines is the data, rather than
architectural differences or other hyperparameters (jbetker, 2023). One of the
key findings of modern LLMs has been that mixing in large amounts of diverse,
but lower quality, data with higher quality data sources (e.g., Wikipedia)
leads to improved performance (Touvron et al., 2023a). Currently, the scale of
internet-scale corpora (Together Computer, 2023) still exceeds the training
token budgets of most models, begging the question of how to sub-select data
to use for training given a fixed compute budget.

In answer to this question, many recent works have focused on data pruning.
Data pruning aims to subselect data in order to 1. increase training
efficiency (same number of epochs, but each epoch is smaller because it’s on a
pruned dataset) or 2. increase performance (compute-controlled, meaning more
epochs). The former may have more impact in low-resource or academic settings,
whereas the latter may have more impact on frontier LLMs. While historically
models have been trained in the sub-1-epoch regime, recent work has shown that
multiple epochs can be performed without sacrificing performance (Muennighoff
et al., 2023), and that perhaps sub-selecting data and epoch’ing on it
outperforms training less than one epoch on the full dataset (Tirumala et al.,
2023). These findings emphasize the need for good data pruning methodologies
for training LLMs.

While the field of data pruning in machine learning has a long history with
many types of methods (Wang et al., 2023), we focus here on data pruning
approaches for large-scale language-like datasets,111The scale of these
datasets can make many classical pruning techniques infeasible. which loosely
fit into three categories. Embedding-based pruning instead uses a self-
supervised pre-trained embedding network to embed all data points, and then a
heuristic to prune examples based on their embeddings (e.g., points that are
within $\epsilon$ of each other (Abbas et al., 2023)). Heuristic-based pruning
involving using human intuition about datasets to prune data points, such as
removing files containing “lorem ipsum” (Raffel et al., 2020) or removing
nearly duplicated training data by comparing n-gram statistics (Lee et al.,
2022). Classifier-based pruning trains a classifier to identify “good” data
from “bad” data, and then prune the bad data. Of course, classifier-based
pruning requires access to some form of label, which can make it difficult to
apply; in practice, some heuristic is used to identify “good” data (e.g.,
webpages referenced on Wikipedia (Touvron et al., 2023a)) and a random sample
of the rest is used as “bad” data.222One could consider improving this scheme
by some version of self-distillation (Pham et al., 2022). Each of these
methods comes with its own pros and cons: embedding-based methods rely on good
embedding networks, heuristic-based methods rely on good human intuition, and
classifier-based methods rely on labeled “good” data.

In this work, we illustrate a simple, yet strong, heuristic-based method in
the domain of finetuning LLMs for code generation, which outperforms existing
embedding-based methods. We choose code generation as our domain of study due
to its increasing importance as an end use case for LLMs (Rozière et al.,
2024) and in line with recent works on embedding-based data pruning (Yang et
al., 2023). In this domain, we demonstrate that embeddings used in prior work
are confounded by document length, which we use to motivate our length
heuristic for data pruning. Surprisingly, we find that long files, which are
often believed to be “higher quality” (Longpre et al., 2023), are often
actually very low quality (in the domain of code), yet make up
disproportionately large amounts of training datasets due to their length. By
pruning long files, we show improvements in compute-limited regimes on the
popular coding benchmarks of HumanEval (Chen et al., 2021) and MBPP (Austin et
al., 2021). However, we find these benefits to be minimal in compute-rich
regimes involving more than 2 epochs, with aggressive pruning methods
overfitting to shorter files, thus emphasizing the recent emerging notion that
data pruning methods may not be compute-agnostic (Goyal et al., 2024). We hope
our findings motivate practitioners to consider simple heuristics more,
especially document length as it directly impacts prevalence in training data,
and that our work adds to the increasing discourse on data pruning
methodologies.

##  2 Related work

Embedding-based data pruning was first shown to work at scale by Sorscher et
al. (2023), who introduced SSL-prototypes and demonstrated it’s efficacy on
Imagenet-scale pretraining for vision. Subsequent work, SemDeDup (Abbas et
al., 2023), applies the embedding-based data pruning paradigm to language
modeling, showing gains when training OPT models (Zhang et al., 2022) up to
1.3B parameters on the C4 dataset (Raffel et al., 2020). D4 (Tirumala et al.,
2023) combined Semdedup with SSL-prototypes at even larger scales (up to 6.7B
parameter OPT models), showing persistent benefits for language modeling. More
recently, SCIP (Yang et al., 2023) used embedding-based data pruning in the
fine-tuning stage for code generation language models, showing performance
benefits on common benchmarks (Chen et al., 2021; Austin et al., 2021).

Heuristic-based data pruning is perhaps the oldest approach in data curation
for language models. Heuristics can range from simple string-based removal,
such as removing documents containing boilerplate “lorem ipsum” or bad
words333https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-
Otherwise-Bad-Words (Raffel et al., 2020), to more sophisticated recent
techniques for removing near-duplicates using MinHash (Lee et al., 2022).
Recent work (Sharma et al., 2024) has shown that such heuristics can be
combined to provide a useful notion of text quality to be used for pruning.
One issue with heuristic-based methods is that they are often customized to
the underlying data domain—for example, Raffel et al. (2020) remove all
documents containing curly braces to purposefully remove code data. Obviously,
such a method would not extend to domains such as code generation. Notably,
it’s common for prior work (Raffel et al., 2020) to remove short documents
(fewer than 3 sentences), but to our knowledge removing long documents has not
been studied. Here, we show that long documents in code data are often “low
quality” and that removing them can lead to efficiency and performance gains.

LLMs for code generation became popular with the advent of generally capable
language models (Brown et al., 2020) which could be finetuned for domain-
specific use cases (Chen et al., 2021). Recent work (Rozière et al., 2024)
showed that finetuning a pretrained Llama model can outperform, at a fixed
compute budget, training from scratch on code data (Li et al., 2022; Fried et
al., 2023; Li et al., 2023a), which motivates our approach. In this setting,
LLMs may be finetuned on domain-specific code datasets, such as The Stack
(Kocetkov et al., 2022). Models are evaluated across many benchmarks, with
HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) being two of the
most common. We refer readers to Xu and Zhu (2022) and Rozière et al. (2024)
for more extensive related work on LLMs for code generation.

##  3 Motivating a length-based heuristic for code data

Figure 1: Cumulative distribution of document lengths in the Python subset of
the Stack dataset (Kocetkov et al., 2022). Documents are ordered on the x-axis
in increasing length order; y-axis shows fraction of whole dataset accounted
for. Green lines highlight that the 2% of longest files account for the last
20% of tokens. Right shows an example long file (the second longest file).
Long files tend to have very little useful signal, being full of large data
arrays or “spaghetti code”.

A common aphorism in software engineering is “Don’t Repeat Yourself” (DRY),
which is linked to the often frowned upon notion of “spaghetti code”—long
files with lots of repeated code. From a language modeling perspective, these
tokens likely provide little-to-no signal for learning (similar to the
decreased information when epoching (Muennighoff et al., 2023)), and thus
waste valuable compute.

To get a sense of how such spaghetti code may be influencing our large-scale
code datasets, we consider the distribution of document lengths in the Python
subset of the popular Stack dataset (Kocetkov et al., 2022). Replicating
recent findings on other language datasets (De Vries, 2023), we find the
distribution to be extremely right-skewed, with e.g., the longest 2% of files
making up a shocking 20% of tokens.

We qualitatively examine many of these files, with an example shown in Figure
1. We find that most long Python files consist of large data arrays or very
bad “spaghetti code”: The longest file is a single list of names and phone
numbers. Second longest is shown in Figure 1b. Third longest is a single
docstring of chinese characters and anglicized pronunciations. Fourth longest
is a game of hangman, but coded without for-loops it would seem, resulting in
essentially the same if-statement repeated thousands of times (for each index
and word). Fifth longest is a single dictionary of glyphs mapping to unicode
characters. Each of these files has about 500,000 tokens.

Prior work has often pruned short files (e.g., shorter than 3 sentences as
done by Raffel et al. (2020)), but we’re unaware of existing work that
specifically prunes long files, which are often believed to be higher quality
and useful for long-context training (Longpre et al., 2023). For the case of
Python code, we see that this is false (at least for the longest files)
motivating pruning long files as a heuristic.

Figure 2: Visualizations of StarEncoder embedding space. a) Cosine similarity
between every pair of cluster centroids. Oddly, the first 37 cluster centroids
are all very close to each other. b) We visualize each document as an $(x,y)$
pair where the $x$-axis is the distance to nearest cluster centroid (computed
as one minus the cosine distance), and the $y$-axis is the length of the file
in tokens (using the Llama tokenizer). We immediately see the reason for the
37 super close clusters – StarEncoder appears to tightly pack most documents
below a certain length (bottom-left cloud of points), while longer documents
tend to be more spread out in embedding space (top-right cloud of points). We
also highlight the ten longest files in the dataset in red.

Beyond this qualitative inspection of the data, we also find that existing
state-of-the-art (SOTA) methods for data pruning in the domain of code
generation (Yang et al., 2023) are actually confounded by file length.
Specifically, the embedding space used for clustering by prior embedding-based
methods is based off StarEncoder (Li et al., 2023a). Following the setup of
Yang et al. (2023), we embedded all files in the Python subset of The Stack
dataset using StarEncoder, and then performed K-means clustering with $K=100$
using the FAISS library (Johnson et al., 2019; Douze et al., 2024). In Figure
2a, we find that many of the clusters are actually very close together, in
terms of cosine similarity. In Figure 2b, we see a strong correlation between
these clusters and file length.444We discuss further implications of this for
embedding-based methods in Appendix A. Specifically, shorter files appear to
be mapped to a very small and dense region of embedding space.555We also
qualitatively inspected these files. Besides finding hundreds of thousands
($\approx 3$% of The Stack Python) of short, auto-generated Django migrations,
we could not discern any correlates besides length. As a result, well
performing embedding-based methods, such as pruning points far from centroids
(Yang et al., 2023), often correlate to pruning long files, further motivating
the use of this heuristic.

##  4 Methods

We finetuned Llama2 7B models on pruned versions of the Python subset of The
Stack dataset (which has 20.4B tokens). We start by separating a small
validation set, controlling for document length, to use in Section 5.1. To get
a notion of variance in the results, we employed a boostrapping framework.
Specifically, we conducted experiments on 3 random 50% subsets of The Stack
Python dataset. We applied each pruning technique to each subset
independently, then trained models on each subset. We then compute standard
error bars over the 3 subsets to get a sense of performance differences
between “different” datasets. We hope this form of dataset bootstrapping
becomes more common in the field, as we found it useful to get a sense of
inherent noise.

As baselines, we consider no pruning, as well as the best performing method
from Yang et al. (2023), which prunes 16% of files from small clusters, and
then 4% of files that lie farthest from centroids (which we refer to as
“SCIP”).666See Appendix Table 1 for a reproduced table from Yang et al. (2023)
comparing various embedding-based pruning methods. For our length pruning
experiments, we consider pruning 10%, 20% or 50% of tokens, coming from the
longest files.777As a sanity check, we also conducted some runs pruning from
the shortest files, in smaller scale settings, and found that this was not
effective. Note the distinction between tokens and files; pruning 20% of
tokens from the longest files on the full dataset would correspond to pruning
just 2% of files (Figure 1).

We train all models for 16,000 steps using a batch size of 1.3M
tokens.888Hyperparameters, compute requirements, etc. are provided in Appendix
B. This would correspond to just over 1 epoch on the full Stack dataset, which
means it corresponds to just over 2 epochs for our baseline (recall: our
bootstrapping uses random 50% subsets of the full dataset), and just over 4
epochs for our strictest pruning condition (50% of tokens). We evaluate
intermediate checkpoints every 2,000 steps on the common code generation
benchmarks HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021).

##  5 Results

In Figure 3, we provide three views on our results:999Full training curves are
provided in Appendix C, for maximum transparency. for considering training
efficiency, we look at performance after a single epoch on all conditions
(left column). We see that the 50% length pruning run performs as well as no
pruning, despite only seeing half the data, indicating a 2x training
efficiency improvement. In the middle column (Figure 3), we provide a
comparison targeted at looking for performance improvements. Namely, we
consider the checkpoints after 8k steps in each condition (roughly 2 epochs
for the 50% length pruning condition, and roughly 1 epoch for the no pruning
baseline). In this plot, we can see strong improvements (absolute: 3.5%) from
length pruning on HumanEval, and modest improvements on MBPP (absolute:
1.5%).101010Note, 8k steps corresponds to two epochs over the aggressively
pruned dataset, which is a bit contrary to classical “sub-1-epoch” training
recommendations. Our work is thus closer to more recent findings (Muennighoff
et al., 2023) which show multiple epochs can be beneficial for language
models. When training for longer (Figure 3, third column), however, we see
that the delta between length pruning and baselines does not persist,
replicating the findings of prior work showing that data pruning cannot be
compute agnostic (Goyal et al., 2024).

Figure 3: Results across baselines and length pruning. All plots show the same
5 conditions: no pruning (“none”) and the best performing (embedding-based)
method from Yang et al. (2023) (“SCIP”) as baselines, and then pruning $P$% of
tokens from the longest documents (our method) for $P=10,20,50$. Leftmost
column shows performance after roughly 1 epoch for each condition, which can
be used to look at training efficiency. We note that aggressive length pruning
($P=50$%) matches the performance of no pruning, indicating a 2x efficiency
improvement. The latter two columns show compute-controlled experiments, where
all conditions are trained on the same number of tokens (but possibly more
epochs, e.g., the 50% length pruning run will have seen twice as many epochs
as no pruning). At 8000 steps (middle column), aggressive length pruning still
seems to outperform no pruning and SCIP baselines, but this advantage
diminishes at the larger compute scale (16000 steps, rightmost column). Error
bars indicate standard error.

Notably, length pruning outperforms SOTA embedding-based pruning methods
(SCIP) in this domain. SCIP does not perform well at smaller compute scales
(single-epoch regime, or even 8k steps). At larger compute scales (rightmost
column) it surpasses no pruning on HumanEval, and matches on MBPP. In the SCIP
paper (Yang et al., 2023), models were trained for 60B tokens—it may be that
SCIP performs better in these larger compute regimes. That said, we note that
even in the larger compute regime, the length pruning method performs as well
as SCIP.

###  5.1 The importance of diverse evaluation

So far, we’ve considered standard downstream benchmarks, where we found
pruning long files leads to training efficiency and performance improvements.
However, we note that most code generation benchmarks tend to be short-form
generation. When used as the sole signals of performance, these benchmarks
could give us an overestimated benefit of pruning long files. To account for
this, we look at perplexity on held out code, binned by length.

In Appendix Figure 5, we find that models can overfit to shorter code
(indicated by increasing validation perplexity), especially for the most
aggressive pruning strategies (where 50% of tokens are removed, coming from
the longest files) in the largest compute regimes (and thus the most
epochs111111Mirroring the findings of Sorscher et al. (2023), pruning long
files could possibly perform better if the source corpus is larger, as more
steps could be taken on the pruned dataset without epoch’ing.). These results
offer a nice counterpoint to the work of Rozière et al. (2024), who found that
finetuning for long-context code generation often deteriorates performance on
short-form code generation (as measured by HumanEval and MBPP). Both of these
results taken together may point to an inherent tradeoff between short- and
long-form code generation capabilities.121212Related, we note that the
performance improvements seen in Li et al. (2023b) may also be partially
explained by the introduction of short-form synthetic code data.

Depending on the downstream use case, such “overfitting” to short-form code
generation may be okay—for example, if end users can only access a model up to
its context length (4096 tokens, in our case), presumably practitioners will
only care about generalization on code sequences up to the context length.

##  6 Discussion

In this work, we showed that a simple heuristic-based data pruning method
outperforms SOTA embedding-based methods when finetuning LLMs for code
generation. Specifically, we found that file lengths are right skewed, leading
to a small percentage of (typically “low quality”) files making up a large
percentage of code data. Removing these files from finetuning data yielded
efficiency and performance improvements on downstream benchmarks in the
compute-limited regime. We emphasize that these results were intended to
introduce a new heuristic that we believe holds merit, and to contrast with
embedding-based pruning methods; we are not asserting that length-based
pruning is the “best” pruning method for general datasets and models. Overall,
we hope our findings encourage practitioners to more carefully inspect subsets
of data that have disproportionately large presence (i.e., long files), and
also think about more principled ways of iterating on data pruning
methodologies (e.g., bootstrapping for error bars, using a diverse suite of
evaluations).

###  6.1 Limitations

While our experiments show compelling performance improvements in the
finetuning setting, due to compute constraints, we were not able to perform
data ablations for pretraining models. It could be that longer files (e.g.,
with a lot of repeated code) could be useful at earlier points of training in
incentivizing in-context learning abilities. Beyond this pre-/fine-tuning
distinction, we also did not consider the use of classifier-based pruning.
Quality classifiers are typically only used on CommonCrawl text data (Touvron
et al., 2023b). However, extending their usage to code data may end up
filtering out many of the “useless” long files, such as those containing
serial numbers (Figure 1), thus reducing the need and/or impact of an explicit
length-based heuristic filter. Finally, we note that our results are limited
to the domain of code generation models. Further work would be needed to see
if long files are generally lower quality data—in fact, we suspect this is
generally not the case (e.g., for natural language corpora), but may be a
factor in web scrapes for other subdomains.

###  6.2 Finding “high quality” long-context code data

As LLMs are deployed more broadly, many model developers have focused on
increasing the context length (Gemini Team, 2024). When increasing context
length, to make sure the model effectively learns to “use it”, we need
corresponding datasets with long documents. However, our findings on the low
quality of longer Python files may beg the question of where such data could
be found for the domain of code generation. To this end, we suggest the use of
techniques such as In-context pretrain (Shi et al., 2023), whereby similar
documents are concatenated within a single context, to boost long-context
capabilities. Given that much of The Stack is sourced from Github, model
practitioners could also consider concatenating files from a single repository
into a training sequence—for codebases utilizing object-oriented programming,
this could be an especially compelling technique as some of the information
needed to predict the next token in a given file may come from a different
file.

Broader Impact Statement

Our work focuses on a simple new heuristic-based method for data pruning. We
do not foresee negative societal consequences from this work, as our work does
not unlock new capabilities. We believe the work has positive consequences for
the research community, such as possibly enabling academic researchers to more
efficiently train models and urging data pruning researchers to consider
simple methods more commonly in their research.

Acknowledgments and Disclosure of Funding

The authors would like to thank William Held, Newsha Ardalani, Sho Yaida,
Niladri Chatterji, Melanie Kambadur, Mike Rabbat, DJ Strouse, and Andrew Saxe
for insightful discussions throughout the course of the work and feedback on
the draft. All work was completed during A.K.S. and Y.Y.’s internships at Meta
FAIR. The authors also thank Jeffery Yu for crucial latex help on earlier
versions of the draft.

## References

  * Abbas et al. (2023) A. Abbas, K. Tirumala, D. Simig, S. Ganguli, and A. S. Morcos.  Semdedup: Data-efficient learning at web-scale through semantic deduplication, 2023.
  * Austin et al. (2021) J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton.  Program synthesis with large language models, 2021.
  * Brown et al. (2020) T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei.  Language models are few-shot learners.  _CoRR_ , abs/2005.14165, 2020.  URL https://arxiv.org/abs/2005.14165.
  * Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba.  Evaluating large language models trained on code, 2021.
  * De Vries (2023) H. De Vries.  Go smol or go home, 2023.  URL https://www.harmdevries.com/post/model-size-vs-compute-overhead/.
  * Douze et al. (2024) M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazaré, M. Lomeli, L. Hosseini, and H. Jégou.  The faiss library.  2024\.
  * Fried et al. (2023) D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W. tau Yih, L. Zettlemoyer, and M. Lewis.  Incoder: A generative model for code infilling and synthesis, 2023.
  * Gemini Team (2024) Gemini Team.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
  * Goyal et al. (2024) S. Goyal, P. Maini, Z. C. Lipton, A. Raghunathan, and J. Z. Kolter.  Scaling laws for data filtering – data curation cannot be compute agnostic, 2024.
  * jbetker (2023) jbetker.  The “it” in ai models is the dataset., 2023.  URL https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/.
  * Johnson et al. (2019) J. Johnson, M. Douze, and H. Jégou.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547, 2019.
  * Kocetkov et al. (2022) D. Kocetkov, R. Li, L. B. Allal, J. Li, C. Mou, C. M. Ferrandis, Y. Jernite, M. Mitchell, S. Hughes, T. Wolf, D. Bahdanau, L. von Werra, and H. de Vries.  The stack: 3 tb of permissively licensed source code, 2022.
  * Lee et al. (2022) K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini.  Deduplicating training data makes language models better, 2022.
  * Li et al. (2023a) R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, Q. Liu, E. Zheltonozhskii, T. Y. Zhuo, T. Wang, O. Dehaene, M. Davaadorj, J. Lamy-Poirier, J. Monteiro, O. Shliazhko, N. Gontier, N. Meade, A. Zebaze, M.-H. Yee, L. K. Umapathi, J. Zhu, B. Lipkin, M. Oblokulov, Z. Wang, R. Murthy, J. Stillerman, S. S. Patel, D. Abulkhanov, M. Zocca, M. Dey, Z. Zhang, N. Fahmy, U. Bhattacharyya, W. Yu, S. Singh, S. Luccioni, P. Villegas, M. Kunakov, F. Zhdanov, M. Romero, T. Lee, N. Timor, J. Ding, C. Schlesinger, H. Schoelkopf, J. Ebert, T. Dao, M. Mishra, A. Gu, J. Robinson, C. J. Anderson, B. Dolan-Gavitt, D. Contractor, S. Reddy, D. Fried, D. Bahdanau, Y. Jernite, C. M. Ferrandis, S. Hughes, T. Wolf, A. Guha, L. von Werra, and H. de Vries.  Starcoder: may the source be with you!, 2023a.
  * Li et al. (2022) Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, T. Hubert, P. Choy, C. de Masson d’Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. Sutherland Robson, P. Kohli, N. de Freitas, K. Kavukcuoglu, and O. Vinyals.  Competition-level code generation with alphacode.  _Science_ , 378(6624):1092–1097, Dec. 2022.  ISSN 1095-9203.  doi: 10.1126/science.abq1158.  URL http://dx.doi.org/10.1126/science.abq1158.
  * Li et al. (2023b) Y. Li, S. Bubeck, R. Eldan, A. D. Giorno, S. Gunasekar, and Y. T. Lee.  Textbooks are all you need ii: phi-1.5 technical report, 2023b.
  * Longpre et al. (2023) S. Longpre, G. Yauney, E. Reif, K. Lee, A. Roberts, B. Zoph, D. Zhou, J. Wei, K. Robinson, D. Mimno, and D. Ippolito.  A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity, 2023.
  * Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter.  Fixing weight decay regularization in adam.  _CoRR_ , abs/1711.05101, 2017.  URL http://arxiv.org/abs/1711.05101.
  * Muennighoff et al. (2023) N. Muennighoff, A. M. Rush, B. Barak, T. L. Scao, A. Piktus, N. Tazi, S. Pyysalo, T. Wolf, and C. Raffel.  Scaling data-constrained language models, 2023.
  * Paszke et al. (2019) A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  _CoRR_ , abs/1912.01703, 2019.  URL http://arxiv.org/abs/1912.01703.
  * Pham et al. (2022) M. Pham, M. Cho, A. Joshi, and C. Hegde.  Revisiting self-distillation, 2022.
  * Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67, 2020.  URL http://jmlr.org/papers/v21/20-074.html.
  * Rozière et al. (2024) B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. Défossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaeve.  Code llama: Open foundation models for code, 2024.
  * Sharma et al. (2024) V. Sharma, K. Padthe, N. Ardalani, K. Tirumala, R. Howes, H. Xu, P.-Y. Huang, S.-W. Li, A. Aghajanyan, G. Ghosh, and L. Zettlemoyer.  Text quality-based pruning for efficient training of language models, 2024.
  * Shi et al. (2023) W. Shi, S. Min, M. Lomeli, C. Zhou, M. Li, R. James, X. V. Lin, N. A. Smith, L. Zettlemoyer, S. Yih, and M. Lewis.  In-context pretraining: Language modeling beyond document boundaries, 2023.
  * Sorscher et al. (2023) B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. S. Morcos.  Beyond neural scaling laws: beating power law scaling via data pruning, 2023.
  * Tirumala et al. (2023) K. Tirumala, D. Simig, A. Aghajanyan, and A. S. Morcos.  D4: Improving llm pretraining via document de-duplication and diversification, 2023.
  * Together Computer (2023) Together Computer.  Redpajama: an open dataset for training large language models, 2023.  URL https://github.com/togethercomputer/RedPajama-Data.
  * Touvron et al. (2023a) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample.  Llama: Open and efficient foundation language models, 2023a.
  * Touvron et al. (2023b) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023b.
  * Wang et al. (2023) Z. Wang, W. Zhong, Y. Wang, Q. Zhu, F. Mi, B. Wang, L. Shang, X. Jiang, and Q. Liu.  Data management for large language models: A survey, 2023.
  * Xu and Zhu (2022) Y. Xu and Y. Zhu.  A survey on pretrained language models for neural code intelligence, 2022.
  * Yang et al. (2023) Y. Yang, A. K. Singh, M. Elhoushi, A. Mahmoud, K. Tirumala, F. Gloeckle, B. Rozière, C.-J. Wu, A. S. Morcos, and N. Ardalani.  Decoding data quality via synthetic corruptions: Embedding-guided pruning of code data, 2023.
  * Zhang et al. (2022) S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer.  Opt: Open pre-trained transformer language models, 2022.
  * Zhao et al. (2023) Y. Zhao, A. Gu, R. Varma, L. Luo, C.-C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y. Hao, A. Mathews, and S. Li.  Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023.

##  Appendix A Embedding-based pruning methods on documents may prune vastly
different numbers of tokens

Table 1: (Reproduced with minor edits and permission from Yang et al. (2023))
Pass@1 performance on HumanEval and MBPP for different pruning methods with
20% documents pruned. Each number here is the result of finetuning a 1.5B
Llama1 (Touvron et al., 2023a) model on 67B tokens (multiple epochs). We refer
readers to the original work for more detail. We add a third row quantifying
the number of tokens pruned by each method. Different methods often prune
vastly different amounts of tokens.

| No | Random | SSL- | SemDeDup | D4 | Small | Far from | Combined
---|---|---|---|---|---|---|---|---
| pruning | Pruning | Prototypes | Clusters | Centroids | Small+Far
HumanEval | 25.0% | 24.0% | 23.8% | 20.7% | 23.2% | 23.2% | 26.8% | 28.0%
MBPP | 33.4% | 31.9% | 32.2% | 32.4% | 31.2% | 35.0% | 30.8% | 33.0%
Tokens pruned | 0.0% | 20.0% | 4.1% | 5.7% | 5.8% | 25.1% | 20.2% | 24.6%

In Table 1, we show the comparison of different embedding-based data pruning
methods from Yang et al. (2023), where each method pruned 20% of documents. In
the third row, we computed the percentage of tokens pruned in each case. These
results emphasize the difference between pruning in terms of documents vs. in
terms of tokens for natural language datasets. In our work, we prune $P$% of
tokens, coming from the longest files.

##  Appendix B Hyperparameters

All models were finetuned using the AdamW (Loshchilov and Hutter, 2017)
optimizer ($\beta_{1}=0.9,\beta_{2}=0.95$) with weight decay of 0.1 and a
learning rate of $1e-4$ that was cosine annealed down to $1e-5$. There was a
linear warmup from 0 to $1e-4$ for the first 1000 steps, as is common
practice. Model code was implemented using PyTorch (Paszke et al., 2019) and
fully-sharded data parallel (Zhao et al., 2023) was used to accelerate
training. Each run took about 26 hours on 64 A100 GPUs.

##  Appendix C Additional results

Full curves for eval performance (HumanEval and MBPP) throughout the
finetuning process are provided in Figure 4.

Validation perlexities are calculated in the standard way by packing documents
together (as done during training), breaking up into context length chunks (in
our case, $2^{12}=4096$), and then calculating average per-token perplexity.
We present two views on the results in Figure 5.

Figure 4: Raw curves of evaluation benchmark performance over training. Shaded
area indicates standard error. Note the results in Figure 3 are taken from
this plot, to more clearly show snapshots at various points during training.

a)
b)

Figure 5: Two views on validation perplexity evolution on various length bins.
a) We show the curves throughout training with each panel showing a different
length bin. It’s clear that the most aggressively pruned setting (red curve,
Prune longest 50%) overfits with multiple epochs (around 10k steps) on all
lengths, and generally has worse perplexity on longer length
documents—intuitively this makes sense since the longest file in the 50%
pruned subset was only $\approx 3600$ tokens. Mirroring our results from
Figure 3, this most aggressive pruning setting performs best in the compute-
limited regime (up to 8k steps) on shorter documents (more akin to the
downstream benchmarks HumanEval and MBPP). b) We instead focus on the
perplexities across length bins, with each color showing a different step.
Considering any time step of the “no pruning” baseline, we see that
perplexities decrease with longer files, up to the context length (black
dotted line), as we’d expect since there’s more “useful” context enabling the
model to lower perplexity. Beyond the context length, perplexities increase as
expected (since all documents are being cut somewhere in the middle).
Surprisingly, perplexities again decrease on the longest documents, seeming to
indicate that being split isn’t affecting these documents. This is in line
with our qualitative observations of repetitiveness in Section 3, and further
motivates pruning these files.


# Dependence of Solar supergranular lifetime on surface magnetic activity and
rotation

Sowmya G M1,Rajani G2, U Paniveni3, R Srikanth3

###### Abstract

The lifetimes and length-scales for supergranular cells in active and
quiescent regions of the Solar chromosphere, and the relation between the two,
were studied using a time series of Ca II K filtergrams. The lifetimes, in
contrast to supergranular length scale and fractal dimension, show no
significant dependence on Solar latitude, suggesting that cell lifetimes are
independent of the differential rotation and a possible supergranular super-
rotation. The functional form of the relation was obtained guided by a
comparison of the distributions of the two supergranular parameters. We infer
a linear dependence of cell lifetime on area, which can be understood by the
assumption of the network’s evolution via a diffusion of the magnetic field.
Our analysis suggests that the diffusion rate in quiet regions is about 10%
greater than in active regions.

1 GSSS Institute of Engineering and Technology for Women, KRS Road, Metagalli
Mysuru-570016, Karnataka, India
2 PES College of Engineering, Mandya - 571401, Karnataka, India.
3 Poornaprajna Institute of Scientific Research,Devanahalli, Bangalore-562110,
Karnataka, India

##  1 Introduction

The supergranular network is the superficial manifestation of Solar convection
and is important for solar flux transport. The existence of a strong
correlation between the chromospheric networks and the supergranulation
structure was pointed out first by Leighton on the basis of Dopplergrams
(Leighton et al.,, 1962; Leighton,, 1963). Subsequent studies made use of Ca
II K spectroheliograms and then filtergrams, an important tool to probe Solar
convection and also magnetism (Chatzistergos et al.,, 2022). Supergranular
network cells (called “supergranules”) are characterized by distributions
centered around a lifetime $T\approx 25$ hours and length scale $L\approx 35$
Mm. Since the early work of Simon and Leighton, (1964), these two parameters
along with the velocity and magnetic fields associated with supergranules have
been studied and reported in a wide range of values by various workers, cf.
McIntosh et al., (2011); Mandal et al., (2017); Chatterjee et al., (2017);
Rajani et al., (2022) and reference therein. More recently, space borne
instruments such as Helioseismic Magnetic Imager (HMI) on board Solar Dynamics
Observatory (SDO) and Solar and Michelson Doppler Imager (MDI) on board
Heliospheric Observatory (SOHO) Williams et al., (2014) have been used to
study supergranulation.

The derived lifetimes of supergranulation shows good dependence on the choice
of method and region. Janssens, (1970), using H$\alpha$ filtergrams, estimated
that $T\approx 21$ hours. Livingston and Orrall, (1974) obtained a similar
value of 22 hour and this was also reported by Singh et al., (1994) based on
the observation of the appearance and disappearance of the cell. On the other
hand, the same authors observed that certain exceptional supergranules found
in the vicinity of active regions can survive for several days. By visual
examination of individual supergranules Wang and Zirin, (1988) detected $T>50$
hour, and pointed out that this can be larger than that obtained by the cross-
correlation (CC) method (Rogers,, 1970), owing to the latter’s sensitivity to
shape changes. Simon and Leighton, (1964) estimated a CC lifetime of 20 hours
using a time series of Ca K spectroheliograms. Worden and Simon, (1976)
employed the CC technique to estimate a lifetime of 36 hours using magnetogram
data. Observing the Fe I 8988Å and Ca II K networks, Duvall, (1980) and Raju
et al., 1998b  derived a CC lifetime of 42 hr and 25 hr, respectively.

Visual inspection techniques possess the advantage of being able to directly
follow intricate morphological changes such as merging, splitting, migration,
disappearance, and appearance of magnetic fluxes that make up the evolution of
the network (Harvey and Martin,, 1973; Wang et al.,, 1995). Therefore,
lifetime estimated using this would better reflect the processes underlying
supergranular dynamics. By contrast, correlation techniques fail to
distinguish between true aspects of cell evolution such as the disappearance
or appearance of certain features and shape changes arising from the
relocation of magnetic elements.

On the question of whether a supergranule survives beyond its correlation
lifetime, there has been conflicting evidence: (Wang and Zirin,, 1988) reports
in the affirmative, but comparable results found by Rogers, (1970) and
Janssens, (1970), using the correlation and morphological techniques for
H$\alpha$ data, respectively, report in the negative. Similarly, lifetimes
estimated by Raju et al., 1998a  and Singh et al., (1994); Paniveni et al.,
(2010) on Ca II K using correlation and visual inspection techniques show
comparable results for both active and quiescient network regions. In the case
of extended supergranular networks, correlation lifetimes can frequently
assume values as large as 45-60 hours. However, in contexts involving
intricate morphological changes, as for example the long-lived features such
as magnetic pukas or plages (Livingston and Orrall,, 1974), it is more
advantageous to study lifetimes visually.

In this work, we investigate and contrast the relations between lifetimes and
length-scales for supergranular cells in active and quiescent regions of the
Solar chromosphere was studied using a time series of Ca II K filtergrams,
extending work done by Singh et al., (1994) and Srikanth et al., (1999). Here
it may be noted that the Ca II K network traces out magnetic flux
concentrations at the supergranular boundary thanks to the enhanced network
brightness they produce (Spruit et al.,, 1990; Hagenaar et al.,, 1997; Raju
and Singh,, 2002). Based on a data analytic method proposed by the latter, we
deduce the functional form with guidance by the distributions of the two
supergranular parameters. Our results are found to support the expected
picture of the network’s evolution through a diffusion of the magnetic fields,
and the influence of the fields on cell properties.

The paper is structured as follows. In Section 2, we introduce the data used
and the method of its analysis. The basic results for cell lifetime and length
scale are presented in Section 3.

##  2 Data and Analysis

Supergranular size $L$ was estimated as square root of the area enclosed
within the cell boundaries traced out on the Ca II K filtergrams. For lifetime
estimation, observations were made for the Kodaikanal Solar Observatory (KSO)
data for the year 1998, 2002, 2004 and 2007 for the descending, minimum and
active phases. In this analysis used data consisting of approximately 1200 Ca
II K filtergrams of the 23rd Solar cycle. Time averaging over 10 min is used
in order to eliminate noise due to 5-min oscillations. This method yields
approximately six data-frames per hour. To estimate lifetime at a given epoch,
approximately 72 hours of data are considered, which span about 432 frames at
10-minute inter-frame intervals. A specific supergranular cell is tracked
across frames sequentially, with lifetime being estimated as the time interval
between the frame of its initial appearance and that of its last disappearance
(Paniveni et al.,, 2004). Supergranular lifetime has been estimated for
quiescent, active and semi-active regions. By quiet region cells, we mean
those found far from the magnetically active regions (see Figure 1). Active
region cells are found in close proximity of active regions (see Figure 2),
whilst the semi-active region supergranules are found in regions of
intermediate magnetic activity (see Figure 3).

In previous studies supergranular lifetime was obtained via cross-correlation
applied to time series data (Srikanth et al.,, 1999). The behaviour was
analysed assuming the diffusion elements of the magnetic network with observed
lifetime identified an a diffusion time-scale. By contrast, lifetime estimates
based on visual inspection, as done here, are implicitly related to the
crossing time of the plasma from the cell center to its edge (Krishan,, 1999).
Therefore, visual inspection is expected to yield the eddy turnover time.
While lifetime estimate by visual inspection is rather tedious, still it is
fairly reliable (Paniveni et al.,, 2010). Our sample size is small, but brings
out characteristic features contrastng the different activity epochs and
regions.

Figure 1: Quiescent region cells: Selection of supergranules in quiet regions
of the Solar chromosphere; from the KSO archive of the 23rd cycle.

Our work here has been focused on studying the behaviour of supergranules in
quiet, intermediate and active regions. The contrast in the cell lifetime
across regions of different activity levels may be theoretically modelled as
arising out of differences in diffusion rates of the magnetic flux transport
(Schrijver et al.,, 1989).

Figure 2: Active region cells: Selection of supergranules in active regions of
the Solar chromosphere; from the KSO archive of the 23rd cycle. Figure 3:
Semi-active region cells: Selection of supergranules in semi-active regions of
the Solar chromosphere; from the KSO archive of the 23rd cycle.

##  3 Results

###  3.1 Cell Lifetime dependence on Solar latitude

Our estimates of cell lifetime in quiet region cells are comparable to those
previously reported based on the KSO data (Chatterjee et al.,, 2017; Mandal et
al.,, 2017; Sowmya et al.,, 2022; Rajani et al.,, 2022). For example, the
estimate for quiet and semi-active regions in this data match that obtained by
Singh et al., (1994), who find $T\in[15,40]$ hour, with the most likely
lifetime being 22 hours. The active region lifetime is estimated by those
authors to be almost double the quiescent value, in agreement with our result
as indicated in Table 1.

A plot of cell lifetimes versus latitude for the data points is shown in
Figure 4. This stands in contrast to length scale and fractal dimension. The
latter shows a latitude dependence, which may be potentially linked to the
Sun’s differential rotation Sowmya et al., (2022). Supergranular scale shows
the vertical-horizontal asymmetry at higher latitudes (Raju,, 2020). Thus, our
observation here suggests that cell lifetime is unaffected by Solar rotation,
whereas spatial properties of cells indeed manifest an influence.

We may consequently also rule out any dependence of supergranulation lifetime
on superrotation, the possible faster rotation of supergranules with respect
to magnetic structures and plasma. However, it may be noted that superrotation
may well be an artefact of projection effects in Dopplergrams (Meunier and
Roudier,, 2007).

Figure 4: Plot of supergranular lifetime (in hours) with respect to Solar latitude (in degrees) using data of the 23rd Solar cycle. Region | Lifetime(hour)
---|---
Quiet | 23.58 $\pm$ 1.3
Semi Active | 34 $\pm$ 1.7
Active | 54.4 $\pm$ 1.6
Table 1: Solar supergranular lifetime over the quiet, semi and active regions.

###  3.2  Deducing the functional dependence of cell lifetime on the size

The parameters such as cell lifetimes $T$ and Length scales $L$, are evidently
interdependent. One can directly estimate the functional dependence of $L$ on
$T$ by curve fit algorithm. We also expect this dependence to be reflected in
the distribution of these two parameters, given in Figures 5 and 6. This
information can also be used to help with the estimation of the functional
relation between $L$ and $T$ for the quiet or active region.

Figure 5: Histogram of lifetime of the Ca II K network cells in the active
region. The curve shows a left-hand side tail. The skewness and kurtosis
derived for this distribution are given in Table 2. Figure 6: Histogram of
length scale of supergranules in the active region. The skewness and kurtosis
derived for this distribution are given in Table 2.

Here we employ an indirect method to estimate the functional relationship
between $L$ and $T$ by estimating the transformation that would map the
distribution of the latter with respect to that of the former. Somewhat
simplistically, we shall assume that their respective distribution is
sufficiently represented by two parameters: (1) the skewness $\varsigma$,
which is a measure of asymmetry of the distribution about the mean $\mu$, and
(2) the kurtosis $\kappa$, which is a measure of the “tailedness” in the
distribution.

In statistics, given a random variable described by probability distribution
$f(x)$, skewness is a quantification of asymmetry of $f(x)$. It is given by
the third standardized moment, defined as follows:

$\displaystyle\centering\varsigma=\frac{1}{\alpha^{3}}\int_{-\infty}^{\infty}(x-\mu)^{3}f(x)dx,$
(1)

where $\alpha$ is the standard deviation. A distribution may be right-skewed
(resp., left-skewed), when it has a more prominent tail on the positive
(resp., negative) side about the mean. A zero-skew distribution is perfectly
symmetric on both wings about the mean. As basic examples, a normal
distribution has zero skewness; an exponential distribution has skewness
$\varsigma=2$, and for a lognormal distribution describing a random variable
$X$ whose logarithm $\ln(X)$ is described the normal distribution with
variance $\beta$, we have $\varsigma=(e^{\beta}+2)\sqrt{e^{\beta}-1}$.

Given a random variable described by probability distribution $f(x)$, kurtosis
is a quantification of how tailed $f(x)$ is, i.e., how well the distribution
features outliers in the extreme, rather than concentration of data closer to
the mean. It is given by the fourth standardized moment, defined as follows

$\displaystyle\centering\kappa=\frac{1}{\alpha^{4}}\int_{-\infty}^{\infty}(x-\mu)^{4}f(x)dx.$
(2)

As basic examples, a normal distribution has kurtosis $\kappa=3$; a Laplace
distribution has $\kappa=6$, and the uniform distribution has $\kappa=1.8$. A
distribution may be platykurtic (resp., leptokurtic), when it has lesser
(resp., greater) kurtosis than the normal distribution. The skewness and
kurtosis for our data is summarized in Table 2.

Field | For Length scale distribution | For Lifetime distribution
---|---|---
Skewness | (0.463, 0.779) | (0.865, 1.278)
Kurtosis | (2.75, 2.638) | (2.045, 2.028)
Table 2: Statistics of lifetime and scale distribution for the Ca II K
networks cells for (active, quiet) regions. We note that all distributions are
platykurtic, i.e., having a lower kurtosis than a normal distribution for
which kurtosis $\kappa=3$.

When a random variable $X$ is subjected to a transformation $\eta$, then the
properties of the distribution of the transformed variable $\eta(X)$, in
particular $\varsigma$ and $\kappa$, will in general be different from those
of the distribution of $X$. For example, above we saw that whereas the normal
distribution of some random variable $X$ has zero skewness, the lognormal
distribution (which characterizes $e^{X}$) is skewed positively. This means
that if we let cell lifetimes and scale to be related by $T\equiv\eta(L)$,
then the right $\eta$ will ensure that the skewness and kurtosis of $\eta(L)$
is close to the corresponding values of the distribution of $T$. Obviously
infinitely many functions $\eta$ may satisfy this requirement. We must thus
restrict $\eta$ to a reasonable family of two parameters for this approach to
work. This is done as follows.

Under an invertible transformation $\eta$ of the random variable $x$ given by
$y\equiv\eta(x)$, let the distribution function $f(x)$ become $g(y)$, which is
determined as follows. By definition:

$\displaystyle\centering\int_{x_{1}}^{x_{2}}f(x)dx=\int_{\phi(x_{1})}^{\phi(x_{2})}g(y)dy,$
(3)

owing to conservation of probability. The positive definiteness of $f(x)$
implies that: $g(y)|dy|=f(x)|dx|,$ whereby

$\displaystyle\centering g(y)=f[\eta^{-1}(y)]|(\eta^{-1})^{\prime}(y)|,$ (4)

where the prime symbol denotes the first derivative.

Let $T=\eta(L)$. Further, let $f(L)$ and $g(T)$ denote the respective
distribution function. In order for our method to work, we must restrict to a
two-parameter family of transformations. It is reasonable to confine $\eta$ to
a polynomial relation of the form

$\displaystyle\centering T=aL^{n}+b$ (5)

According to equation (4)

$\displaystyle\centering
g(T)=\frac{f(L)}{\left[(L-b)^{n-1}a\right]^{\frac{1}{n}}n}$ (6)

We now apply this exercise to our lifetime vs length scale data.

Table 2 summarizes the skewness and kurtosis data for the active region given
in histograms Figures 5 and 6, and additionally for quiet regions (not
included, for brevity). Based on the skewness, we find that in either region,
supergranular scales are less asymmetric than lifetimes. This feature seems
generic for supergranules, irrespective of the activity level.

To determine $n$ for a given region (active or quiet), the values of skewness
and kurtosis for the transformed length scale are plotted as a function of $n$
in the range $1.0\leq n\leq 2.5$. The plots for the two, namely $\varsigma(n)$
and $\kappa(n)$, are given in Figures 7 and 8, respectively. As expected, both
plots exhibit a monotonic increase for $x\geq 1.0$.

Figure 7: Plot of skewness of distribution for various powers indices of
length scale. The skewness of the lifetime distribution is 0.865.

Figure 8: Plot of kurtosis distribution for various powers ’n’ of length scale

.

In the case of active regions, for the observed lifetime distribution skewness
$\varsigma$ = 0.463 (Table 2), the skewness for length scale distribution
corresponds to the Figure 7, range 2.125$\leq$ n $\leq$ 2.25. Similarly, for
the observed lifetime distribution, kurtosis $\kappa=2.75$, the kurtosis of
the transformed length scale distribution corresponds to the range $1.875\leq
n\leq 2$ as shown in Figure LABEL:fig:7 . We choose $n=2$, and the Monte-Carlo
least-squares curve fitting algorithm yields the function:

$T=7.45+3.5A,$ (7)

where $T$ is given in hours and supergranular area $A\equiv L^{2}$ is in units
of Mm2. More specfically, indicating error bars, we may give in place of Eq.
(7), the fit function $T=\alpha+\beta A$, where $\alpha$ and $\beta$ are the
fit constants having the units of $T$ and $TL^{-2}$, respectively, with
$\alpha=7.45\pm 0.025$ hours and $\beta=3.5\pm 0.01$ Mm2. Figure 9 depicts the
observed data on the dependence of lifetime on scale in active regions, as
well as the fit based on Eq. (7). This shows a reasonably good agreement
between the two.

Figure 9: Variation of lifetime as a function of length scale for Ca II K
network cells for active region. The fit function is given by Eq. (7)

For quiet regions, the analogous calculation yields, in case of the skewness
data of of length scale and lifetime distributions, the range $2\leq n\leq
2.125$., and in the case of kurtosis of the length scale and lifetime
distributions, the range $1.75\leq n\leq 2$. Analogous to Eq. (7), in the
quiet region, the Monte-Carlo curve fitting algorithm yields the function:

$T=6.75+3.25A,$ (8)

where $T$ is given in hours and $L^{2}$ in units of Mm2. As before, indicating
error bars, we may give in place of Eq. (8), the fit function $T=\mu+\nu A$,
where $\mu$ and $\nu$ are the fit constants having the units of $T$ and
$TL^{-2}$, respectively, with $\mu=6.75\pm 0.023$ hours and $\nu=3.25\pm 0.02$
Mm2. Figure 10 depicts the observed data on the dependence of lifetime on
scale in active regions, as well as the fit based on Eq. (8). This shows a
reasonably good agreement between the two.

Figure 10: Variation of lifetime (hr) as a function of length scale for Ca II
K network cells for quiet region. The plot is a fit using the functional form
of Eq. (5).

Eqs. (7) and (8) are consistent with the dependence reported by Singh et al.,
(1994); Srikanth et al., (1999), where the authors find a linear relation
between lifetime and scale of supergranules, which can broadly be understood
through a model where cell lifetime is related to the diffusion of magnetic
elements. However, the above authors restrict their study only to quiet
regions, whereas we extend the study to a comparative study of quiet and
active regions.

##  4 Discussion and Conclusion

We studied the lifetimes and length-scales of supergranular cells in active
and quiescent regions of the Solar chromosphere, and the relation between the
two, using a time series of Ca II K filtergrams. We find that the lifetimes
show no significant dependence on Solar latitude, suggesting that cell
lifetimes are independent of the differential rotation. This independence
stands in contrast to supergranular length scale and fractal dimension. For
example, Raju et al., 1998b  have noted that supergranular size as observed in
the Ca II K shows upto a 7% latitudinal variation. Sowmya et al., (2022)
report an anticorrelation between fractal dimension and latitude in the belt
between 20∘ N and 20∘ S.

Our results on the lifetime-scale relation can be interpreted to shed light on
the relative dynamics of the active and quiet regions of the Sun. From Eqs.
(7) and (8), we find that the slope is 3.5 hr Mm-2, which is slightly larger
in the case of active regions than in the quiet regions, namely 3.25 hr Mm-2.
This difference may be understood as a consequence of the fact that lifetime
and scale of an active or quiet region cell can depend on its interaction with
the ambient magnetic fields. In specific, the above noted difference in slopes
may be attributed to two related factors: (a) lowering of cell size in the
presence of magnetic activity (Singh and Bappu,, 1982), and (b) the
enhancement of cell lifetime in active regions, as noted in Table 1. The
effect of magnetic flux can be understood as due to plasma confinement by the
magnetic field (Sowmya et al.,, 2022).

Accordingly, the slope $dT/dA$ in Eqs. (7) or (8) may be interpreted as the
inverse of the diffusion coefficient $D$ associated with the cell, i.e., as
$1/D$. For active regions, we then have $D=10^{6}/(3.5~{}[\pm 0.01]\times
3600)\approx 79.3\pm 0.2$ km2/s. Similarly, we obtain about $D\approx 85.5\pm
0.5$ km2/s for quiet regions. Intuitively, the longer lifetime of cells in the
active region is due to the lower diffusion rate, and our results imply that
in the active region, the diffusion happens about $79.3/85.5\approx 0.93$
slower than in the quiet region. It may be noted that this is in agreement
with recent works (Abramenko,, 2017, 2018) demonstrating superdiffusivity in
quiet regions and nearly-normal diffusion in active regions. Specifically, the
pattern of greater diffusivity in quiet regions than in active ones is found
to be pronounced in the case of cells with scale greater than 5 Mm
(Abramenko,, 2018), which is compatible with the scale range appropriate to
the present data.

As such the relative diffusion of the active and quiet regions is expected to
be dependent on the phase of the cycle. For example, during solar minimum a
large cell in the active region may retain its identity for upto 10 months,
where during solar maximum, the exceptional large lifetimes rarely exceed 4
months. These issues may be studied in the future in contnuation of work
reported here.

## References

  * Abramenko, (2017) Abramenko, V. I. (2017).  Dispersion of the solar magnetic flux in the undisturbed photosphere as derived from sdo/hmi data.  Monthly Notices of the Royal Astronomical Society, 471(4):3871–3877.
  * Abramenko, (2018) Abramenko, V. I. (2018).  Dispersion of small magnetic elements inside active regions on the sun.  Monthly Notices of the Royal Astronomical Society, 480(2):1607–1611.
  * Chatterjee et al., (2017) Chatterjee, S., Mandal, S., and Banerjee, D. (2017).  Variation of supergranule parameters with solar cycles: results from century-long kodaikanal digitized ca ii k data.  The Astrophysical Journal, 841(2):70.
  * Chatzistergos et al., (2022) Chatzistergos, T., Krivova, N. A., and Ermolli, I. (2022).  Full-disc ca ii k observations—a window to past solar magnetism.  Frontiers in Astronomy and Space Sciences, page 336.
  * Duvall, (1980) Duvall, T. L. (1980).  The equatorial rotation rate of the supergranulation cells.  Solar Physics, 66(2):213–221.
  * Hagenaar et al., (1997) Hagenaar, H. J., Schrijver, C. J., et al. (1997).  The distribution of cell sizes of the solar chromospheric network.  The Astrophysical Journal, 481(2):988.
  * Harvey and Martin, (1973) Harvey, K. L. and Martin, S. F. (1973).  Ephemeral active regions.  Solar Physics, 32(2):389–402.
  * Janssens, (1970) Janssens, T. (1970).  Long term observations of the h$\alpha$ chromospheric network.  Solar Physics, 11(2):222–242.
  * Krishan, (1999) Krishan, V. (1999).  Two-fluid description of plasmas.  In Astrophysical Plasmas and Fluids, pages 197–234. Springer.
  * Leighton, (1963) Leighton, R. B. (1963).  The solar granulation.  Annual review of astronomy and astrophysics, 1:19.
  * Leighton et al., (1962) Leighton, R. B., Noyes, R. W., and Simon, G. W. (1962).  Velocity fields in the solar atmosphere. i. preliminary report.  The Astrophysical Journal, 135:474.
  * Livingston and Orrall, (1974) Livingston, W. and Orrall, F. (1974).  Magnetic pukas and the lifetime of the supergranulation.  Solar Physics, 39:301–304.
  * Mandal et al., (2017) Mandal, S., Chatterjee, S., and Banerjee, D. (2017).  Association of supergranule mean scales with solar cycle strengths and total solar irradiance.  The Astrophysical Journal, 844(1):24.
  * McIntosh et al., (2011) McIntosh, S. W., Leamon, R. J., Hock, R. A., Rast, M. P., and Ulrich, R. K. (2011).  Observing evolution in the supergranular network length scale during periods of low solar activity.  The Astrophysical Journal Letters, 730(1):L3.
  * Meunier and Roudier, (2007) Meunier, N. and Roudier, T. (2007).  The superrotation of solar supergranules.  Astronomy & Astrophysics, 466(2):691–696.
  * Paniveni et al., (2004) Paniveni, U., Krishan, V., Singh, J., and Srikanth, R. (2004).  Relationship between horizontal flow velocity and cell lifetime for supergranulation.  arXiv preprint physics/0407079.
  * Paniveni et al., (2010) Paniveni, U., Krishan, V., Singh, J., and Srikanth, R. (2010).  Activity dependence of solar supergranular fractal dimension.  Monthly Notices of the Royal Astronomical Society, 402(1):424–428.
  * Rajani et al., (2022) Rajani, G., Sowmya, G. M., Paniveni, U., and Srikanth, R. (2022).  Solar supergranular fractal dimension dependence on the solar cycle phase.  Research in Astronomy and Astrophysics, 22(4):045006.
  * Raju, (2020) Raju, K. (2020).  Asymmetry in the length scales of the solar supergranulation network.  The Astrophysical Journal Letters, 899(2):L35.
  * Raju and Singh, (2002) Raju, K. and Singh, J. (2002).  Dependence of supergranular length-scales on network magnetic fields.  Solar Physics, 207:11–16.
  * (21) Raju, K., Srikanth, R., and Singh, J. (1998a).  The correlation lifetimes of chromospheric ca ii k network cells.  Solar Physics, 178(2):251–257.
  * (22) Raju, K., Srikanth, R., and Singh, J. (1998b).  The dependence of chromospheric ca ii k network cell sizes on solar latitude.  Solar Physics, 180(1):47–51.
  * Rogers, (1970) Rogers, E. H. (1970).  Lifetime of the h$\alpha$ chromospheric network.  Solar Physics, 13(1):57–77.
  * Schrijver et al., (1989) Schrijver, C., Cote, J., Zwaan, C., and Saar, S. (1989).  Relations between the photospheric magnetic field and the emission from the outer atmospheres of cool stars. i-the solar ca ii k line core emission.  The Astrophysical Journal, 337:964–976.
  * Simon and Leighton, (1964) Simon, G. and Leighton, R. (1964).  Velocity fields in the solar atmosphere. iii. large-scale motions, the chromospheric network, and magnetic fields.  The Astrophysical Journal, 140:1120.
  * Singh and Bappu, (1982) Singh, J. and Bappu, M. K. V. (1982).  Solar Phys., 71:161.
  * Singh et al., (1994) Singh, J., Nagabhushana, B., Babu, G., and Uddin, W. (1994).  Study of calcium-k network evolution from antarctica.  Solar Physics, 153(1):157–167.
  * Sowmya et al., (2022) Sowmya, G. M., Rajani, G., Paniveni, U., and Srikanth, R. (2022).  Supergranular fractal dimension and solar rotation.  Research in Astronomy and Astrophysics, 22(9):095018.
  * Spruit et al., (1990) Spruit, H. C., Nordlund, A., and Title, A. (1990).  Solar convection.  Annual review of astronomy and astrophysics, 28(1):263–303.
  * Srikanth et al., (1999) Srikanth, R., Raju, K., and Singh, J. (1999).  The chromospheric network: Dependence of cell lifetime on length-scale.  Solar Physics, 184(2):267–280.
  * Wang and Zirin, (1988) Wang, H. and Zirin, H. (1988).  The velocity pattern of weak solar magnetic fields.  Solar physics, 115(2):205–219.
  * Wang et al., (1995) Wang, J., Wang, H., Tang, F., Lee, J. W., and Zirin, H. (1995).  Flux distribution of solar intranetwork magnetic fields.  Solar Physics, 160(2):277–288.
  * Williams et al., (2014) Williams, P. E., Pesnell, W. D., Beck, J. G., and Lee, S. (2014).  Analysis of supergranule sizes and velocities using solar dynamics observatory (sdo)/helioseismic magnetic imager (hmi) and solar and heliospheric observatory (soho)/michelson doppler imager (mdi) dopplergrams.  Solar Physics, 289:11–25.
  * Worden and Simon, (1976) Worden, S. P. and Simon, G. W. (1976).  A study of supergranulation using a diode array magnetograph.  Solar Physics, 46(1):73–91.