: Robust Training for Code Generation Models

Yuhao Zhangequal,uwm
Shiqi Wangamazon
Haifeng Qianamazon
Zijian Wangamazon
Mingyue Shangamazon
Linbo Liuamazon
Sanjay Krishna Goudaamazon
Baishakhi Rayamazon
Murali Krishna Ramanathanamazon
Xiaofei Maamazon
Anoop Deorasamazon

uwmDepartment of Computer Science, University of Wisconsin-Madison, Madison, USA
amazonAWS AI Labs

Yuhao<EMAIL_ADDRESS>Shiqi<EMAIL_ADDRESS>
Machine Learning, ICML

*Work done while the author was at Amazon.

Code generation models are not robust to small perturbations, which often lead to inconsistent and incorrect generations and significantly degrade the performance of these models.
Improving the robustness of code generation models is crucial to better user experience when these models are deployed in real-world applications.
However, existing efforts have not addressed this issue for code generation models.
To fill this gap, we propose , a framework to improve the robustness of code generation models, generalizing a large variety of code perturbations to enrich the training data and enabling various robust training strategies, mixing data augmentation, batch augmentation, adversarial logits pairing, and contrastive learning, all carefully designed to support high-throughput training.
Extensive evaluations show that we improve the average robust pass rates of baseline CodeGen models from 14.79 to 21.74.

Notably, the improvement in robustness against code-syntax perturbations is evidenced by a significant decrease in pass rate drop from 95.04% to 53.35%.

Despite the impressive performance of code generation models, they are not robust as small perturbations on a prompt could lead
to unstable and wrong generations.
Improving the robustness of code generation models is the key to better user experience when the model is deployed in real-world applications.
Contrastive learning (CL) is widely used for improving the robustness of code representations for masked language modeling.
However, code generation models are mainly trained by casual language modeling (CLM) and the applicability of CL on CLM remains unexplored.
To fill this gap, we propose a framework, comprising a dataset generation module and a robust training module, to investigate the robustness improvements of CL and other approaches on CLM.
The dataset generation module groups various concrete code perturbations into two general perturbations, context-free and context-sensitive, enabling easily extension to new perturbations.
The robust training module contains different robust training approaches, including data augmentation, batch augmentation, adversarial logits pairing, and contrastive learning, that have small overhead in order to be applicable to CLM, which usually requires thousands of GPU-hours to train.
One surprising finding is that traditional CL objective decreases the robust pass rate by 0.03 compared to the baseline.
Extensive evaluations show that our approach can improve the average robust pass rates of baseline CodeGen-350M models from 8.64 to 12.40, while data augmentation only improves to 11.71.
update results, include starcoder and other codegen models.

§ CONTEXT-SENSITIVE ALIGNMENT

The context-sensitive perturbation requires the code generation model $\model{}$ to generate different token sequences, while satisfying certain constraint for an original prompt and its perturbed version.
In this paper, we focus on a special type of context-sensitive perturbation called rename perturbation that renames a variable or a function name in a program.
Let $\perturbation_\mathrm{rename}(\bfx) = \{\substitution_1(\bfx), \substitution_2(\bfx),\ldots,\}$ be the set of all possible perturbed prompts from $\bfx$ by the rename perturbation $\perturbation_\mathrm{rename}$, where $\substitution_i$ is a function substituting old names with new names.
Each perturbed prompt $\perturbed{\bfx} \in \perturbation_\mathrm{rename}(\bfx)$ corresponds to a name substitution $\substitution$, i.e., $\substitution(\bfx)=\perturbed{\bfx}$.

Consider the following code snippet $\bfc$, “def identity(x): return x”, if we apply the rename perturbation $\perturbation_\mathrm{rename}$ to this snippet, one possible perturbed snippet $\perturbed{\bfc}$ would be: “def identity(a): return a”, where the variable name “x” is substituted by “a”.
The perturbed snippet $\perturbed{\bfc}$ corresponds to a substitution $\substitution_{x\mapsto a}$, i.e., $\substitution_{x\mapsto a}(\bfc)=\perturbed{\bfc}$.

Given the original prompt $\bfx$, the perturbed prompt $\perturbed{\bfx}$, and the corresponding substitution $\substitution$, the constraint of rename perturbation $\cons_{\perturbation_\mathrm{rename}}$ requires the generation of the perturbed prompt $\perturbed{\bfy}$ to be the same as the original generation $\bfy$ after applying the substitution $\substitution$.
Formally, $\cons_{\perturbation_\mathrm{rename}}(\bfy, \perturbed{\bfy})$ is defined as $\substitution(\bfy)=\perturbed{\bfy}$.

Let $\bfx$ = “def identity(x): ”, $\perturbed{\bfx}$ = “def identity(a): ”, and the expected outputs for model for two prompts be $\model{}(\bfx)=\bfy$ = “return x”, $\model{}(\perturbed{\bfx})=\perturbed{\bfy}$ = “return a”.
These two expected outputs satisfy the constraint $\cons_{\perturbation_\mathrm{rename}}$ because $\substitution_{x \mapsto a}(\bfy)=\perturbed{\bfy}$.

§.§ Context-Sensitive Alignment

In context-free alignment, we design our loss functions to encourage the model compute the same hidden states for the original prompt and the perturbed prompt.
However, the context-free alignment does not work for context-sensitive perturbation because from the same hidden states, the model will generate same output sequences which contradict the requirement of the context-sensitive perturbation.

Why not use [MASK] tokens to “neutralize” the influence of the substituted variable/function name? It's a good idea, I plan to try out it later.

Given the following original prompt $\bfx = x_1, x_2, \ldots, x_{k-1}, x_k$ and its perturbed version $\perturbed{\bfx} = \perturbed{x_1}, x_2, \ldots, x_{k-1}, \perturbed{x_k}$[To simplify illustration, we only consider the case where two tokens $x_1$ and $x_k$ have been substituted.], we denote the model hidden states after seeing $\subseq{\bfx}{1}{i}$ and $\subseq{\perturbed{\bfx}}{1}{i}$ as $h_i$ and $\perturbed{h}_i$.
The hidden state $h$ contains two information 1) $h_{name}$, the information related to the identifier that will be substituted, e.g., what is the name of this identifier, and 2) $h_{null}$, other information that is orthogonal to the identifier, e.g., the overall program control flow.

We assume that the sum of this two information vector $h_{name}$ and $h_{null}$ gives back the hidden state $h$.
Note that we do not assume that $h_{name}$ and $h_{null}$ are orthogonal because $h_{name}$ might contribute to $h_{null}$, e.g., a variable name “i” has a large influence to the program control flow.
We further decompose $h_{name}$ as $h_{name\_mere}$ and $h_{name\_null}$.
$h_{name\_name}$ contains mere information about the name of the variable and it is orthogonal to both $h_{null}$ and $h_{name\_null}$.

\begin{align}
h = h_{name\_mere} + h_{name\_null} + h_{null},\quad h_{name\_mere} \perp h_{name\_null} \wedge h_{name\_mere} \perp h_{null}
\end{align}

In context-sensitive alignment, we want to keep the difference between $h_{name\_mere}$ and $\perturbed{h_{name\_mere}}$ and to align $h_{null} + h_{name\_null}$ and $\perturbed{h_{null}} + \perturbed{h_{name\_null}}$.
A stricter version is to only align $h_{null}$ and $\perturbed{h_{null}}$, but it is hard to filter out the name contribution $h_{name\_null}$. Note that the original name contribution $h_{name\_null}$ might be better than $\perturbed{h_{name\_null}}$, e.g., in VarRenamerRN, a meaningful variable name will be replaced by a random valid variable name.

§.§.§ Disentangle $h_{name\_mere}$ from $h$

The key challenge of context-sensitive alignment is to disentangle $h_{name\_mere}$ from $h$.

Between two occurrence of the substituted name, say, $x_1$ and $x_k$ ($\perturbed{x_1}$ and $\perturbed{x_k}$, respectively), each $h_{i,name\_mere}$ in $h_i$ should be the same for $1\le i <k$.

For all $1 < i < k$, let $g_i = h_i - h_{i-1}$, then $g_i \perp h_{i,name\_mere} \wedge g_i \perp h_{i-1,name\_mere}$.

Corollary <ref> holds when $\forall 1< i < k.\ h_{i,name\_mere}=h_{i-1,name\_mere}$, which is equivalent to Assumption <ref>.

Let $g_i = \frac{1}{k-1} \sum_{j=1}^{k-1} h_j - h_i$, then $\forall 1\le i < k.\ g_i \perp h_{i,name\_mere}$.

Corollary <ref> holds when $\forall 1\le i<k.\ h_{i,name\_mere}=\frac{1}{k-1} \sum_{j=1}^{k-1} h_{j,name\_mere}$, which is also equivalent to Assumption <ref>.

Using Corollary <ref> or Corollary <ref>, we can identify a vector $g_i$ that is orthogonal to $h_{i,name\_mere}$. We can use $g_i$ to disentangle $h_{i,name\_mere}$ from $h$ by projecting $h_i$ onto $g_i$.

\begin{align}
h_{i,name\_mere} = h_i - \frac{\langle g_i, h_i \rangle}{\|g_i\|\|h_i\|} g_i, \quad h_{i,null} + h_{i,name\_null} = \frac{\langle g_i, h_i \rangle}{\|g_i\|\|h_i\|} g_i \label{eq: disentanglement}
\end{align}

Using Theorem <ref>, we can get $h_{i,null} + h_{i,name\_null}$ for the original prompt and get $\perturbed{h}_{i,null} + \perturbed{h}_{i,name\_null}$ for the perturbed prompt.
We then use contrastive loss to align $h_{i,null} + h_{i,name\_null}$ and $\perturbed{h}_{i,null} + \perturbed{h}_{i,name\_null}$.
Do we need to make sure the assumption 1.2 holds? We can compute $h_{i,name\_mere}$ from Theorem <ref> as well, we can design loss to ensure $h_{i,name\_mere}$ are all the same for all $1 \le i <k$.

§ INTRODUCTION

Code generation models [Li et al., 2023, Nijkamp et al., 2023, Nijkamp et al., 2023, Fried et al., 2023, Luo et al., 2023, Rozière et al., 2023] have demonstrated impressive performance in generating code from natural language descriptions, completing sections of code, and even tackling complex coding contest challenges.
These models have the potential to offer assistance to software engineers and increase their productivity.

However, code generation models are not robust to minor perturbations in the input prompts (e.g., inserting whitespaces/typos in docstrings or substituting variable names in code), i.e., they often generate inconsistent and incorrect outputs, thus significantly degrading their impressive performance on nominal prompts and hurting user experience when deployed in real-world applications [Wang et al., 2023].
Figure <ref> shows that the performance of the state-of-the-art (SOTA) public code models [Nijkamp et al., 2023, Li et al., 2023, Luo et al., 2023] significantly declines under semantic-preserving program transformations, particularly in the case of code-syntax perturbations.
Thus, it is necessary to improve the robustness of models before they can be universally adopted and deployed.

The performance drop of the state-of-the-art public code models on four classes of code perturbations.

Despite extensive research efforts to improve the robustness of code-related tasks, beyond code generation,
such as vulnerability prediction, clone detection, and code summarization, existing work has not tackled two unique challenges of improving the robustness of code generation models, primarily trained using casual language modeling (CLM).

Distinct Robustness Definition

Unlike traditional classification tasks like vulnerability detection, where models produce a single classification, code generation models generate sequences, leading to a shift in the definition of robustness for certain perturbations.
In code generation, model robustness is defined by generating a coherent output given an input perturbation. In contrast, in classification tasks, models are expected to maintain the same classification before and after perturbation.
For instance, if a variable i is renamed to b in the input prompt (as illustrated in Figure <ref>), a robust code generation model should generate completions with the variable b rather than maintain the original name i. This distinction necessitates defining a new category of perturbations for code generation models and designing corresponding robust training approaches to tackle this new category.

Designing Robust Training Approaches

As code perturbations can insert dead code or typos into training data, directly training using data augmentation could adversely affect the model performance, leading to issues like generating dead code or typos.
Furthermore, applying more deliberate robust training approaches such as adversarial logits pairing (ALP) [Kannan et al., 2018] and contrastive learning (CL) to CLM presents unique challenges.
ALP, designed for single-class classification, requires careful alignment between original and perturbed sequences, complicated by potential differences in sequence lengths.
Although CL has demonstrated efficacy in improving the robustness of code representations in masked language modeling (MLM) [Devlin et al., 2019], its applicability to improving the robustness of code generation models remains unexplored.
Directly applying the CL objective (ContraSeq) from ContraBERT [Liu et al., 2023] and ContraCode [Jain et al., 2021] on sequence representations may not cater to CLM, which involves discriminating representations at a finer level and goes beyond just sequence representations.
Notably, this adoption shows negligible robustness improvement on CLM code models (Section <ref>). Thus, designing CL objectives tailored to finer granularities is imperative.

To tackle the above two challenges and improve the robustness of code generation models, we introduce a structured definition of code perturbations (Section <ref>) and design a novel framework named (Section <ref>).
generalizes various code perturbations to enrich the training data and enables robust training with different approaches.

To address Challenge <ref>, we classify existing code perturbations into two categories: context-free and context-sensitive, based on the formal definition of code perturbations provided in Section <ref>.
Context-free perturbations, such as the docstring perturbation in Figure <ref>, follow the traditional notions of robustness, whereas context-sensitive perturbations, such as the code-syntax perturbation in Figure <ref>, specific the distinct robustness definition highlighted in Challenge <ref>.
The distinction of these two categories allows to employ different robust training methods according to each category.

Moreover, we propose two novel approaches tailored to context-sensitive robustness: ALP with name-Dropout (ALPD) and name-level CL (ContraName).

To address Challenge <ref>, employs example-level and sequence-level pairing to enrich the training set.
These two pairing levels allow 1) a masking mechanism to mask unnatural perturbed tokens and 2) a careful alignment between the original and perturbed token sequences, addressing the crucial challenge of applying data augmentation and ALP to CLM.
Additionally, we propose a novel token-level CL (ContraToken) inspired by Jain et al., 2023.
ContraName and ContraToken empower CLM to discriminate finer-grained representations.

We utilize to extensively evaluate various strategies, mixing data augmentation, batch augmentation, ALP, and CL.
Our best approach mixing batch augmentation with the masking mechanism, ALP, and ALPD significantly enhances the model robustness, surpassing the sub-optimal results achieved by data augmentation.
Notably, the pass rate drop due to code-syntax perturbations is improved from 95.04% to 53.35%.
Our ablation studies show that ContraSeq, the CL objective used in previous work for MLM, has negligible robustness improvements on CLM.

We summarize our contributions:
1) a framework, , for improving the robustness of code generation models trained by CLM, addressing the unique challenges of distinct robustness definition and designing robust training approaches,
2) designs of robust training approaches, ALP, ALPD, ContraToken, and ContraName tailored to CLM,
3) an extensive evaluation of different robust training approaches,

4) a surprising finding that the ContraSeq CL
objective, which is known to be beneficial for improving robustness of other code related tasks, has negligible robustness improvements on CLM.

§ RELATED WORK

Adversarial Attacks on Code-Related Tasks
Numerous adversarial attacks [Henkel et al., 2022, Zhang et al., 2020, Yefet et al., 2020, Jha & Reddy, 2023, Srikant et al., 2021, Anand et al., 2021, Gao et al., 2023] have targeted encoder-decoder models in code-related tasks, including classification (e.g., vulnerability prediction, clone detection) and generation (e.g., code summarization, comment generation). Key methods include CODA [Tian et al., 2023], which exploits syntactic differences for adversarial example generation; CARROT [Zhang et al., 2022], employing a lightweight hill climbing for optimization in attacks; and ALERT [Yang et al., 2022], which creates naturalness-aware attacks using pre-trained models.
Unlike these approaches, we focus on improving the robustness of code generation models trained using casual language modeling. We assess our approaches' effectiveness in code generation through ReCode [Wang et al., 2023], a benchmark for evaluating robustness via semantic-preserving program transformations.

Robust Training on Code-Related Tasks
Adversarial attacks typically enhance model robustness through data augmentation and adversarial training [Madry et al., 2018].
Bielik & Vechev, 2020 refine model representations by feeding only pertinent program parts to the model;
Suneja et al., 2023 use curriculum learning and data augmentation with simplified programs.
They all tend to improve robustness in classification tasks. Unlike these, our focus is on code generation robustness.

While Zhou et al., 2022 propose random input token masking to lessen dependence on non-robust features, our method selectively masks perturbed tokens during loss calculation to avoid the model generating unnatural perturbations. In contrast to ContraCode [Jain et al., 2021] and ContraBERT [Liu et al., 2023], which apply contrastive learning to classification and code translation tasks by improving robustness in masked language modeling,
we focus on the efficacy of contrastive learning in decoder-only code generation models.
Although ContraCLM [Jain et al., 2023] enhances the discrimination of CLM's representations, it does not specifically target robustness improvement.

§ PROBLEM DEFINITION

We address the robustness challenge in a code generation model $f$ trained using Causal Language Modeling (CLM).

CLM predicts the next token in a sequence, and the model can only attend to tokens on the left.
Formally, given a sequence of tokens $\bfx = x_1, \ldots, x_{n}$, the generation model $f$ captures $p_f(\cdot \mid \bfx_{:i})$, representing the conditional probabilities of the $i$-th token given the preceding tokens $\bfx_{:i} = x_1, \ldots, x_{i-1}$. The model is trained on a dataset $D=\{\trpoint{\bfx}{j}\}_{j=1}^m$ using cross-entropy loss $\lossclm(\bfx) = -\sum_{i=1}^{n} \log p_f(x_i \mid \bfx_{:i})$.

The generation model $f$ determines the most likely next token $\hat{x}_i$ by selecting the token with the highest probability in $p_f(\cdot \mid \bfx_{:i})$ from the entire vocabulary $\alphabet$,
$\hat{x_i} = \mathrm{argmax}_{v\in \alphabet} p_f(v \mid \bfx_{:i})$.

Utilizing a given decoding strategy, such as greedy or temperature sampling [Holtzman et al., 2020], the generation model $f$ produces a sequence of tokens by iteratively predicting the next tokens until a specified stop criterion is reached.
We denote $f(\bfx_{:i}) = \hat{\bfx}_{i:}$ as the generated token sequence by $f$.

The terms prompt, completion, and ground truth refer to the input $\bfx_{:i}$, the output $\hat{\bfx}_{i:}$, and the original completion $\bfx_{i:}=x_i, \ldots, x_{n}$, respectively.

In code generation, the token sequence $\bfx$ represents a code snippet.

Figure <ref> shows a problem in HumanEval [Chen et al., 2021].
Each prompt $\bfx_{:i}$ in a problem contains a function signature and a corresponding docstring description, and each ground-truth completion $\bfx_{i:}$ is the correct function implementation.
A code generation model $f$ is asked to generate a completion, denoted as $\hat{\bfx}_{i:}$.
If the completed function $\bfx_{:i} + \hat{\bfx}_{i:}$ passes all the hidden test cases, the generation $\hat{\bfx}_{i:}$ is deemed correct, denoted as $\cor{\bfx_{:i}}{\hat{\bfx}_{i:}} = \emph{true}$. Otherwise, if any of the tests fail, $\cor{\bfx_{:i}}{\hat{\bfx}_{i:}} = \emph{false}$.

§.§ Code Perturbations

ReCode [Wang et al., 2023] is a comprehensive robustness evaluation benchmark for code generation models containing semantic-preserving code perturbations across four classes: docstring, function-name, code-syntax, and code-format perturbations.
We present examples from these four classes and we encourage readers to refer to the original paper for more detailed descriptions.

Docstring Perturbations rewrite natural language in docstrings and comments, including edits like adding typos and substituting synonyms
(See Figure <ref>).
Function-Name Perturbations refactor some function names, e.g., changing from snake_case to camelCase and adding typos (See Figure <ref>).
Code-Syntax Perturbations apply perturbations related to code syntax, involving changes such as inserting deadcode and renaming variables (See Figure <ref>).
Code-Format Perturbations change the code snippets' format, e.g., adding newlines and splitting a line into two (See Figure <ref>).

We define a code perturbation, denoted by $\perturbation = \{\trans_1, \trans_2, \ldots\}$, as a collection of string transformations.
Each transformation $\trans: \mathcal{X} \mapsto \mathcal{X}$ operates on a token sequence from the input domain $\mathcal{X}$, altering them to produce a perturbed sequence.
Within $\perturbation$, each transformation specifies different positions and replacements for perturbation.

The VarRenamer perturbation, shown in Figure <ref>, is a code perturbation $\pi$.
It contains an infinite set of string transformations that specify 1) which variable name to change and 2) the new variable name.

The former contains two choices: n and i.
And the latter contains infinite choices of valid variable names.

We introduce two categories, context-free and context-sensitive perturbations, which serve as a high-level interface  for generating perturbed datasets and facilitating robust training.

We formalize the distinctive characteristics of these two categories in the following sections.

§.§.§ Context-Free Perturbations

A code perturbation $\perturbation$ is a context-free perturbation if all perturbed prompts generated by $\perturbation$ should not affect the ground-truth completion.
Formally, for all $\trans \in \perturbation$, the concatenation of the prompt perturbed by $\trans$ and the ground-truth completion remains a correct function:

\begin{align}
\forall \trans \in \perturbation, \cor{\trans(\bfx_{:i})}{\bfx_{i:}}\label{eq: context-free-def}
\end{align}

In Figure <ref>, the SynonymSubstitution perturbation in the docstring will not affect the ground-truth completion.

§.§.§ Context-Sensitive Perturbations

A code perturbation $\perturbation$ is a context-sensitive perturbation if any perturbed prompt generated by $\perturbation$ results in coherent changes to the ground-truth completion.
Formally, for all $\trans \in \perturbation$, the concatenation of the prompt perturbed by $\trans$ and its ground-truth completion perturbed correspondingly is a correct function, while the concatenation of the perturbed prompt and the original ground-truth completion is not.

\begin{align}
\forall \trans \in \perturbation, \cor{\trans(\bfx_{:i})}{\trans(\bfx_{i:})} \wedge \neg \cor{\trans(\bfx_{:i})}{\bfx_{i:}} \label{eq: context-sensitive-def}
\end{align}

In Figure <ref>, the VarRenamer perturbation requires the ground-truth completion to change coherently because all the variable i should be renamed to b.

§.§ Robustness of Code Generation Models

To define model robustness, we say a model $f$ is robust to a perturbation $\perturbation$, if

\begin{align}
\forall \trans \in \perturbation, \cor{\trans(\bfx_{:i})}{f(\trans(\bfx_{:i}))} \label{eq: robustness}
\end{align}

Notice that the robustness against context-free perturbations is similar to the traditional robustness definition, in which the perturbation should not change the model results (Eq <ref>).
However, the robustness against context-sensitive perturbations differs from the traditional robustness definition, as the context-sensitive robustness requires the model output to change coherently with the perturbed prompt (Eq <ref>).

§ : A ROBUST TRAINING FRAMEWORK

enhances the training set with a paired dataset generation module, as detailed in Section <ref>. This process involves tokenizing code snippets before and after perturbations into pairs of original and perturbed token sequences, thereby creating a paired dataset. Each token in the original sequence is matched with its equivalent in the perturbed sequence. Section <ref> outlines our robust training strategies.

§.§ Paired Dataset Generation

The paired dataset generation method offers two types of pairings: example-level and sequence-level.
Example-level pairing matches each original training example with its perturbed counterpart. Sequence-level pairing provides more detail by matching each token segment from the original example to its equivalent in the perturbed example. These pairing mechanisms are essential for the robust training strategies discussed in Section <ref>.

Example-level Pairing
Given a training set $D=\{\trpoint{\bfx}{j}\}_{j=1}^m$, where each training example is a code snippet, the paired dataset generation module returns a set of paired of training samples
$\{(\trpoint{\bfx}{j}, \trpoint{\pt{\bfx}}{j})\}$, where $\trpoint{\pt{\bfx}}{j}$ is the code snippet perturbed by some code perturbations.

To obtain $\trpoint{\pt{\bfx}}{j}$, we first randomly choose $t$ code perturbations $\{\perturbation_1, \ldots, \perturbation_t\}$ in ReCode.
Then, we randomly choose one string transformation from each code perturbation and apply it to the original code snippet $\bfx$,

\begin{align}
\pt{\bfx} = T_t(T_{t-1}(\ldots T_1(\bfx)\ldots)), \quad T_i \in \perturbation_i, \forall 1 \le i \le t. \label{eq: train_ptb}
\end{align}

Sequence-level Pairing
Given a pair of code snippets, $(\bfx, \pt{\bfx})$,

the paired dataset generation module further provides finer-granularity paring for this pair.

Consider the following pair of tokens,
Index:      0 1 2 3 4 5 6 7 8 9
Original:   A @@C@@ D E F   G H   @@C@@ @@I@@
Perturbed:  A   B   @@X@@ D E F   Y   @@X@@ @@Z Z@@

In this example, the code perturbations perform two context-free perturbations, insert “B” and substitute “G H” with “Y”, and two context-sensitive perturbations, substitute all “C” with “X” and substitute all “I” with “Z Z”.

To create sequence-level pairing, we introduce a mask sequence $\mask$ ($\pt{\mask}$) for the original sequence $\bfx$ (and the perturbed sequence $\pt{\bfx}$, respectively).
Each mask value indicates which kind of perturbation is applied to the corresponding token,  for unperturbed,  for context-free, and  for context-sensitive.

We show the two masks of Example <ref>.
Index:          0 1 2 3 4 5 6 7 8 9
Original Mask:  U @@S@@ U U U   F     F   @@S@@ @@S@@
Perturbed Mask: U   F   @@S@@ U U U   F   @@S@@ @@S@@ @@S@@

This two-level pairing design is the key to enabling some of the robust training approaches, which will be introduced subsequently.

§.§ Designing Robust Training Approaches

This section introduces four lightweight robust training approaches: data augmentation, batch augmentation, adversarial logits pairing, and contrastive learning, all carefully designed to tailor to CLM training.

§.§.§ Data Augmentation

Data augmentation is a widely used approach to improve the robustness of machine learning models. A common practice is replacing a certain portion, denoted as $p$, of the original training examples with their perturbed counterparts in each training batch.
Formally, for a training batch $\{\trpoint{\bfx}{j}\}_{j=1}^b$ and its paired perturbed batch $\{\trpoint{\pt{\bfx}}{j}\}_{j=1}^b$, the objective function for data augmentation is expressed as follows,

\begin{align}
\lossdataaug = \sum_{j=1}^b a_j\lossclm(\trpoint{\pt{\bfx}}{j}) + (1-a_j)\lossclm(\trpoint{\bfx}{j}), \label{eq: lossdataaug}
\end{align}

where  $a_j\overset{\text{i.i.d}}{\sim} \bernoulli{p}$ is a Bernoulli variable

indicating whether the $j$-th training example will be perturbed or not.

Masking Unnatural Perturbed Tokens
Some context-free perturbations introduce unnatural tokens, such as DeadCode Insertion adding an artificial code segment and ButterFingers introducing typos.
Referring back to the robustness property in Eq <ref>, our goal is for the model to learn to respond to these perturbations rather than to generate them.
Learning to generate these unnatural perturbed tokens could adversely affect the original model performance, leading to issues like generating dead code or typos.
We propose masking the CLM loss of these unnatural perturbed tokens to address these issues.

We define the CLM loss for the example $\bfx$ after masking out the unnatural perturbed tokens

\begin{align*}
\lossclm(\bfx, \mask) = -\sum_{i=1}^{|\bfx|} \mathds{1}_{\{m_i \neq \mathrm{\freeptb}\}} \log p_f(x_i \mid \bfx_{:i}),
\end{align*}

where $m_i \neq \mathrm{\freeptb}$ means that the $i$-th token is not perturbed by a context-free perturbation (see Example <ref>). We design the masked data augmentation loss $\lossmdataaug$ by replacing the term $\lossclm(\pt{\bfx})$ in Eq <ref> with the masked loss $\lossclm(\pt{\bfx}, \mask)$.

§.§.§ Batch Augmentation

Batch augmentation [Hoffer et al., 2019] duplicates a portion of training examples within the same batch with different perturbations.
It differs slightly from data augmentation, where a batch contains $p$ perturbed and $1-p$ original data.
In batch augmentation, the entire batch is augmented with $p$ perturbed rather than replacing $p$ original data with perturbed data as in data augmentation.
Given a training batch and its paired perturbed batch, the objective function of batch augmentation is defined as follows,

\begin{align*}
\lossmbatchaug = \sum_{j=1}^b a_j\lossclm(\trpoint{\pt{\bfx}}{j}, \trpoint{\pt{\mask}}{j}) + \lossclm(\trpoint{\bfx}{j})
% \label{eq: lossmbatchaug}
\end{align*}

Note that we apply the masking mechanism to batch augmentation as well. When batch augmentation was originally proposed, its primary goal was not to improve the robustness of the model.
However, we hypothesize that it can further improve the robustness over data augmentation, as indicated in some multilingual cases [Ahmed & Devanbu, 2022, Wang et al., 2023].

§.§.§ Adversarial Logits Pairing

Adversarial Logits Pairing (ALP) [Kannan et al., 2018] improves the robustness of classification models by minimizing the KL divergence between the original input's prediction distribution and the perturbed input's prediction distribution.
However, adapting ALP from classification models to generation models trained by CLM is challenging.
One straightforward approach is decomposing the generation task into multiple next-token prediction tasks.
However, the original and the perturbed token sequences can have different lengths due to some transformations adding or removing tokens.
This length discrepancy prevents a direct match of each token's prediction between the two sequences.

To address this challenge, we leverage the sequence-level pairing provided by our paired dataset generation module.
We apply ALP only to the unperturbed segments of the two sequences, marked by $\mathrm{\unptb}$ in Example <ref>.
All unperturbed segments have the same length, allowing us to apply ALP to the predictions of these unperturbed tokens.
We use $\bfu$ and $\pt{\bfu}$ to denote the ordered indices for all unperturbed tokens in the original and perturbed sequences.
The ALP objective is defined as follows,

\begin{align*}
\lossalp = \sum_{j=1}^b \sum_{i=1}^{|\trpoint{\bfu}{j}|} \KL\left(p_f(\cdot \mid \trpoint{\pt{\bfx}}{j}_{:\trpoint{\pt{u}}{j}_i})\parallel p_f(\cdot \mid \trpoint{\bfx}{j}_{:\trpoint{u}{j}_i})\right) %\label{eq: lossalp}
\end{align*}

In Example <ref>, $\bfu=(0,2,3,4)$ and $\pt{\bfu}=(0,3,4,5)$.

ALP with name-Dropout (ALPD)
We design another ALP approach specifically tailored to variable and function rename transformations among context-sensitive perturbations.
We propose to reduce the model's reliance on specific variable and function names by setting the attention masks of a portion of these names to zero.
ALPD can be seen as a dropout mechanism specific to entity names. We use $\drop{\bfx}$ to denote the input sequence after name-specific dropout.

\begin{align*}
\lossalpdrop = \sum_{j=1}^b \sum_{i=1}^{|\trpoint{\bfu}{j}|} &\KL\left(p_f(\cdot \mid \drop{\trpoint{\bfx}{j}}_{:\trpoint{u}{j}_i})\parallel p_f(\cdot \mid \trpoint{\bfx}{j}_{:\trpoint{u}{j}_i})\right) \nonumber \\
+ &\KL\left(p_f(\cdot \mid \drop{\trpoint{\pt{\bfx}}{j}}_{:\trpoint{\pt{u}}{j}_i})\parallel p_f(\cdot \mid \trpoint{\pt{\bfx}}{j}_{:\trpoint{\pt{u}}{j}_i})\right)
%\label{eq: lossalpdrop}
\end{align*}

$\lossalpdrop$ sums two KL divergence losses: the first over the original sequence after and before dropout, and the second over the perturbed sequence.

§.§.§ Contrastive Learning

Contrastive learning (CL) maximizes the cosine similarity between positive (similar) pairs and minimizes the distance between negative (dissimilar) pairs.
The granularity of pairs leads to different designs of CL objectives.
This section introduces three designs of CL objectives tailored to CLM.
ContraSeq and ContraToken objectives, inspired by ContraCLM [Jain et al., 2023], focus on the levels of sequences and tokens.
A novel ContraName objective focuses on the level of variable and function names.

ContraSeq The ContraSeq objective operates at the sequence level, where each pair consists of summarizations of two input sequences. We note that this setting is also adopted in ContraBERT [Liu et al., 2023] and ContraCode [Jain et al., 2021] for improving the robustness of the encoder model trained on masked language modeling (MLM).
Since CLM does not have the [CLS] token used in MLM, we compute the average of the hidden states in the last layer as the summarization.

Given a batch $\batch=\{\bfh_1, \ldots, \bfh_b, \pt{\bfh}_1, \ldots, \pt{\bfh}_b\}$ with $2b$ summarizations of original and perturbed sequences,
ContraSeq treats the $b$ corresponding original and perturb pairs as positive pairs and other pairs in the batch as negatives.

Denoting the temperature
hyper-parameter as $\tau$ and cosine similarity as $\diamond$, we define the ContraSeq objective as follows,

\begin{align*}
\losscontraseq = \sum_{j=1}^b g(\trpoint{\bfh}{j}, \trpoint{\pt{\bfh}}{j}, B) + g(\trpoint{\pt{\bfh}}{j}, \trpoint{\bfh}{j}, B), %\label{eq: losscontraseq}
\end{align*}

where $g(x, y, B)$ is defined as

\begin{align*}
g(x, y, B) = -\log{\frac{\exp(x \diamond y/\tau)}{\sum_{h\in B} \exp(x \diamond h/\tau) - \exp(1/\tau)}}.
%\label{eq: the_g}
\end{align*}

ContraSeq represents the coarsest granularity among the three objectives.
While ContraSeq is shown to be effective for the goal of MLM by ContraBERT and ContraCode, it may not fully cater to CLM's objective, which involves discriminating representations at a finer level, beyond just sequence representations, to predict the next token in each prefix.
Additionally, ContraSeq poses scalability challenges, as it demands a large batch size to compute a meaningful InfoNCE loss. This challenge restricts ContraSeq's feasibility to large language models.

ContraToken The ContraToken objective operates at the token level, providing a much finer granularity than ContraSeq.
ContraToken aims to discriminate the representation of each prefix.
However, as we mentioned in Section <ref>, directly treating $\bfx_{:i}$ and $\pt{\bfx}_{:i}$ as a positive pair does not work due to the potential sequence length difference between original and perturbed sequences.
To address this, ContraToken considers two prefixes ending at the same unperturbed token as a positive pair, with other prefix pairs designated as negatives.

For the $j$-th training example, let $\trpoint{\bfh}{j}_i$ denote the representation of the prefix up to the $i$-th token, and $\trpoint{u}{j}_i$ denote the index of the $i$-th unperturbed token. We define ContraToken objective as:

\begin{align*}
\losscontratoken = \sum_{j=1}^b\sum_{i=1}^{|\trpoint{\bfu}{j}|} g(\trpoint{\bfh}{j}_{\trpoint{u}{j}_i}, \trpoint{\pt{\bfh}}{j}_{\trpoint{\pt{u}}{j}_{i}}, \trpoint{H}{j})
g(\trpoint{\pt{\bfh}}{j}_{\trpoint{\pt{u}}{j}_{i}}, \trpoint{\bfh}{j}_{\trpoint{u}{j}_i}, \trpoint{H}{j})
%\label{eq: losscontratoken}
\end{align*}

where $\trpoint{H}{j}$ contains all the representations of prefixes ending at unperturbed tokens for the $j$-th training example, i.e., $\trpoint{H}{j}=\{\trpoint{\bfh}{j}_{\trpoint{u}{j}_{1}},\ldots, \trpoint{\bfh}{j}_{\trpoint{u}{j}_{|\trpoint{\bfu}{j}|}},\trpoint{\pt{\bfh}}{j}_{\trpoint{\pt{u}}{j}_{1}},\ldots, \trpoint{\pt{\bfh}}{j}_{\trpoint{\pt{u}}{j}_{|\trpoint{\pt{\bfu}}{j}|}}\}$.

To address variable and function rename transformations in context-sensitive perturbations, we design a novel name-level CL objective, ContraName. This objective aims to enhance the discrimination of representations for different variable and function names.

In ContraName, we group representations of variables or functions according to their names.
For a name spanning multiple tokens, we use the average of these tokens as its representation.
ContraName treats representations within the same group as positive pairs and those across different groups as negative pairs.
Notice that the negative pairs in ContraName have explicit semantic differences, i.e., different names should yield different representations.
This explicit semantic difference of negative pairs has been shown to improve the effectiveness of CL [Ding et al., 2023].

Suppose in the sequence $\bfx$, we identify $g$ groups of name representations $G_1, G_2, \ldots, G_g$, with $G=\bigcup
_{i=1}^g G_i$ being their union. We define the ContraName objective on the input $\bfx$ as follows,

\begin{align*}
\losscontraname(\bfx) = -\log\left( \dfrac{\sum_{i=1}^g \sum_{\bfh, \bfh' \in G_i} \exp(\bfh \diamond \bfh' / \tau)}{\sum_{\bfh, \bfh' \in G} \exp(\bfh \diamond \bfh' / \tau)}  \right)
\end{align*}

Consider the original example in Example <ref> with $G_1 = \{\bfh_1, \bfh_7\}$ and $G_2=\{\bfh_8\}$.
The perturbed example contains $\pt{G_1} = \{\pt{\bfh}_2, \pt{\bfh}_7\}$ and $\pt{G_2}=\{\frac{\pt{\bfh}_8 + \pt{\bfh}_9}{2}\}$.

The final ContraName objective is the sum of losses over the original sequences and their perturbed counterparts.

\begin{align}
\losscontraname = \sum_{j=1}^m \losscontraname(\trpoint{\bfx}{j}) +  \losscontraname(\trpoint{\pt{\bfx}}{j})  \label{eq: losscontraname}
\end{align}

§ EVALUATION

§.§ General Experimental Setup

Models We use different robust training approaches to fine-tune different sizes of mono-lingual CodeGen models [Nijkamp et al., 2023]: CodeGen-6B, CodeGen-2B, and CodeGen-350M.
We provide fine-tuning settings in Appendix <ref>.

Datasets and Benchmarks We use the stack dataset (v1.2) [Kocetkov et al., 2022] as our raw training dataset $D$. Our dataset generation module uses ReCode [Wang et al., 2023] to augment the dataset by introducing different code perturbations as $\pt{D}$. We set $t=2$ in Eq <ref>, i.e., we apply at most two code perturbations to each original code snippet.
We set $p=25\%$ for all robust training approaches.

To evaluate the robustness, we use the ReCode benchmark, which is based on HumanEval [Chen et al., 2021] and MBPP [Austin et al., 2021].
The docstring and function-name classes in ReCode are perturbed based on the original prompt.
The code-syntax and code-format classes are perturbed based on a modified version, where each prompt is appended with half of the ground truth completion.

We use the following three metrics to assess the nominal and robustness performance of models.

. We use Pass@k following Chen et al., 2021 to assess the nominal code generation performance.
To separate from the Pass@k used for robustness metrics, we name the Pass@k on unperturbed data to be Nominal Pass@k (NP@k).

This metric approximates the probability of any $k$ samples passes all the test case, if we randomly choose $k$ samples out of $n$ samples generated by the model for each problem.
We use $n=5$ due to computational constraints.
Additionally, as demonstrated in the Recode paper [Wang et al., 2023], the difference between $n=10$ and $n=100$ is already small.

. To evaluate the robustness of models, we use the same metric introduced in ReCode, the Robust Pass$_s$@k (RP$_{s}$@k). It measures the worst-case Pass@k on $s$ perturbed variance for each perturbation type and each sample. Here, we use $s=10$ to harden the robustness gain for training and differentiate performance gaps.

. Following ReCode, we also report Robust Drop%. It measures the percentage drop from Robustness Pass@k (RP$_{s}$@k) from Nominal Pass@k (NP@k), indicating the relative robustness changes given perturbations. Lower means better robustness.

§.§ Effectiveness of Proposed Approaches

[sharp corners, boxsep=0pt,left=3pt,right=3pt,top=2pt,bottom=2pt]

Summary of the Results
Our approach significantly enhances the robustness of code generation models, surpassing the results of data augmentation. Notably, our approach exhibits the most substantial improvement in robustness against Syntax perturbations, achieving a remarkable 17.83 enhancement.

Table <ref> summarizes the robust evaluation results for CodeGen models.
We use $\lossclm$ to denote the baseline method that fine-tunes on the stack dataset (unseen by CodeGen models) without any robust training approaches.
Comparing $\lossclm$ and the original model (Ori) of CodeGen, we find that fine-tuning on unseen data can already improve model robustness on the Docstring, Function, and Format perturbations, except the Syntax perturbation.

When averaging across four perturbation classes, our approach demonstrates significant improvements in —9.37, 6.49, and 4.99 for CodeGen-6B, CodeGen-2B, and CodeGen-350M, respectively, compared to the baseline $\lossclm$. In contrast, data augmentation achieves sub-optimal results with improvements of 7.87, 6.24, and 3.43.

Averaging over all three models, our approach enhances by 3.29, 0.88, 17.94, and 5.68 for Docstring, Function, Syntax, and Format perturbations, respectively, compared to the baseline $\lossclm$.
Surprisingly, our approach exhibits the most substantial improvement in robustness against Syntax perturbations. This emphasis on strengthening robustness to Syntax perturbations is crucial for ensuring the reliability of code models in handling diverse syntactic variations.

We conducted statistical analyses using paired-t tests to compare our approach with baseline $\lossclm$ and data augmentation $\lossdataaug$ across four perturbation classes.
Our approach significantly outperforms the baseline $\lossclm$ with $p<0.05$ on all perturbation classes and all models with exceptions of function-name perturbations on CodeGen-6B and CodeGen-2B.
We hypothesize that the less pronounced results on function-name perturbations are due to the imbalanced perturbed data, as the percentages of function-name perturbations are much smaller compared to other perturbation types.
When comparing our approach with data augmentation $\lossdataaug$ (shown in Table <ref>), we found that our approach significantly outperforms $\lossdataaug$ with $p<0.01$ on six cases, while $\lossdataaug$ outperforms our approach with $p<0.05$ on one case.

§.§ Ablation Studies

[sharp corners, boxsep=0pt,left=3pt,right=3pt,top=2pt,bottom=2pt]

Summary of the Results
The ablation studies confirm the effectiveness of masked batch augmentation, ALP, and ALPD.
ContraSeq provides negligible improvements compared to the baseline ($\lossclm$).
ContraToken and ContraName yield mixed results in different settings.

This section presents the ablation results of different approaches outlined in Section <ref> applied to the CodeGen-350M model.
We conduct our experiments in two settings.
The first setting (Table <ref>) focuses on context-free perturbations, applying the original data augmentation ($\lossdataaug$) loss to context-sensitive perturbations while varying different approaches for the context-free perturbations.
In the second setting (Table <ref>), we vary approaches for context-sensitive perturbations while maintaining the $\lossdataaug$ loss for context-free perturbations.
We report the overall average of , , and across four perturbation classes, with more details reported in Appendix <ref>.

Effectiveness on Masked Batch Augmentation
The masked batch augmentation loss $\lossmbatchaug$ consists of two components: (1) a masking mechanism that masks unnatural perturbed tokens and (2) batch augmentation.
Comparing the results of masked data augmentation ($\lossmdataaug$) and data augmentation ($\lossdataaug$) in Table <ref> validates the effectiveness of the masking mechanism because $\lossmdataaug$ achieves better and than $\lossdataaug$.
To assess the effectiveness of batch augmentation, we cannot directly compare the results of $\lossmdataaug$ ($[2]$, Table <ref>) and $\lossmbatchaug$ ($[4]$, Table <ref>) because $\lossmdataaug$ is trained on $p=25\%$ perturbed data, while $\lossmbatchaug$ is trained on $\frac{p}{1+p}=20\%$ perturbed data.
For a fair comparison, we train $\lossdataaug$ with $p=20\%$ perturbed data and report the result at $[3]$ in Table <ref>.
Comparing the results of $\lossmdataaug(p=20\%)$ and $\lossmbatchaug$ confirms the effectiveness of batch augmentation because $\lossmbatchaug$ achieves better , , and than $\lossmdataaug(p=20\%)$.

Effectiveness of ALP and ALPD
ALP and ALPD are shown to be effective because $\lossalp$ and $\lossalpdrop$ both improve the and ($[5]$ vs $[4]$ in Table <ref> and $[1]$ vs $[2]$ in Table <ref>).
We further investigate different designs of ALP and ALPD in Appendix <ref>.

Discussion on Contrastive Learning Objectives
ContraSeq only provides negligible improvements, as evidenced by $[6]$ vs $[0]$, and $[8]$ vs $[5]$ in Table <ref>.
ContraToken behaves differently in two ablation experiment settings.
In the context-free perturbation experiment (Table <ref>), ContraToken improves the but negatively impacts the and ($[7]$ vs $[5]$).
Conversely, in the context-sensitive perturbation experiment (Table <ref>), ContraToken hurts all metrics ($[5]$ vs $[4]$).
Adding ContraName to the masked batch augmentation loss $\lossmbatchaug$ improves the and ($[3]$ vs $[1]$).

§ CONCLUSION, LIMITATIONS, AND FUTURE WORK

Our framework, , improves the robustness of code generation models by generalizing a large variety of code perturbations to enrich the training data and enabling various robust training strategies.
We foresee many future improvements to this paper.
First, ALPD and ContraName primarily target function and variable rename perturbations but are not general enough to handle arbitrary context-sensitive perturbations.
However, these approaches can be applied to name-entities in general NLP tasks.
Second, the robustness improvement of function-name perturbation on CodeGen-6B and CodeGen-2B is insignificant compared to the baseline, necessitating unique strategies to overcome this limitation.

§ IMPACT STATEMENTS

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

[Ahmed & Devanbu, 2022]
Ahmed, T. and Devanbu, P. T.
Multilingual training for software engineering.
In 44th IEEE/ACM 44th International Conference on Software
Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp.  1443–1455. ACM, 2022.
URL <https://doi.org/10.1145/3510003.3510049>.

[Anand et al., 2021]
Anand, M., Kayal, P., and Singh, M.
On adversarial robustness of synthetic code generation.
CoRR, abs/2106.11629, 2021.
URL <https://arxiv.org/abs/2106.11629>.

[Austin et al., 2021]
Austin, J., Odena, A., Nye, M. I., Bosma, M., Michalewski, H., Dohan, D.,
Jiang, E., Cai, C. J., Terry, M., Le, Q. V., and Sutton, C.
Program synthesis with large language models.
CoRR, abs/2108.07732, 2021.
URL <https://arxiv.org/abs/2108.07732>.

[Bielik & Vechev, 2020]
Bielik, P. and Vechev, M. T.
Adversarial robustness for code.
In Proceedings of the 37th International Conference on Machine
Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of
Proceedings of Machine Learning Research, pp.  896–907. PMLR,
URL <http://proceedings.mlr.press/v119/bielik20a.html>.

[Chen et al., 2021]
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J.,
Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger,
G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S.,
Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C.,
Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes,
E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang,
J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr,
A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight,
M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei,
D., McCandlish, S., Sutskever, I., and Zaremba, W.
Evaluating large language models trained on code.
CoRR, abs/2107.03374, 2021.
URL <https://arxiv.org/abs/2107.03374>.

[Devlin et al., 2019]
Devlin, J., Chang, M., Lee, K., and Toutanova, K.
BERT: pre-training of deep bidirectional transformers for language
In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings
of the 2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, NAACL-HLT 2019,
Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp.   4171–4186. Association for Computational Linguistics, 2019.
URL <https://doi.org/10.18653/v1/n19-1423>.

[Ding et al., 2023]
Ding, Y., Chakraborty, S., Buratti, L., Pujar, S., Morari, A., Kaiser, G. E.,
and Ray, B.
CONCORD: clone-aware contrastive learning for source code.
In Just, R. and Fraser, G. (eds.), Proceedings of the 32nd
ACM SIGSOFT International Symposium on Software Testing and Analysis,
ISSTA 2023, Seattle, WA, USA, July 17-21, 2023, pp.  26–38. ACM, 2023.
URL <https://doi.org/10.1145/3597926.3598035>.

[Fried et al., 2023]
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R.,
Yih, S., Zettlemoyer, L., and Lewis, M.
Incoder: A generative model for code infilling and synthesis.
In The Eleventh International Conference on Learning
Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net,
URL <https://openreview.net/pdf?id=hQwb-lbM6EL>.

[Gao et al., 2023]
Gao, F., Wang, Y., and Wang, K.
Discrete adversarial attack to models of code.
Proc. ACM Program. Lang., 70 (PLDI), jun 2023.
URL <https://doi.org/10.1145/3591227>.

[Henkel et al., 2022]
Henkel, J., Ramakrishnan, G., Wang, Z., Albarghouthi, A., Jha, S., and Reps,
T. W.
Semantic robustness of models of source code.
In IEEE International Conference on Software Analysis,
Evolution and Reengineering, SANER 2022, Honolulu, HI, USA, March 15-18,
2022, pp.  526–537. IEEE, 2022.
URL <https://doi.org/10.1109/SANER53432.2022.00070>.

[Hoffer et al., 2019]
Hoffer, E., Ben-Nun, T., Hubara, I., Giladi, N., Hoefler, T., and Soudry, D.
Augment your batch: better training with larger batches.
CoRR, abs/1901.09335, 2019.
URL <http://arxiv.org/abs/1901.09335>.

[Holtzman et al., 2020]
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y.
The curious case of neural text degeneration.
In 8th International Conference on Learning Representations,
ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
URL <https://openreview.net/forum?id=rygGQyrFvH>.

[Jain et al., 2023]
Jain, N., Zhang, D., Ahmad, W. U., Wang, Z., Nan, F., Li, X., Tan, M.,
Nallapati, R., Ray, B., Bhatia, P., Ma, X., and Xiang, B.
Contraclm: Contrastive learning for causal language model.
In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.),
Proceedings of the 61st Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto,
Canada, July 9-14, 2023, pp.  6436–6459. Association for Computational
Linguistics, 2023.
URL <https://aclanthology.org/2023.acl-long.355>.

[Jain et al., 2021]
Jain, P., Jain, A., Zhang, T., Abbeel, P., Gonzalez, J., and Stoica, I.
Contrastive code representation learning.
In Moens, M., Huang, X., Specia, L., and Yih, S. W. (eds.),
Proceedings of the 2021 Conference on Empirical Methods in Natural
Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican
Republic, 7-11 November, 2021, pp.  5954–5971. Association for
Computational Linguistics, 2021.
URL <https://doi.org/10.18653/v1/2021.emnlp-main.482>.

[Jha & Reddy, 2023]
Jha, A. and Reddy, C. K.
Codeattack: Code-based adversarial attacks for pre-trained
programming language models.
In Williams, B., Chen, Y., and Neville, J. (eds.),
Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI
2023, Thirty-Fifth Conference on Innovative Applications of Artificial
Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in
Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14,
2023, pp.  14892–14900. AAAI Press, 2023.
URL <https://ojs.aaai.org/index.php/AAAI/article/view/26739>.

[Kannan et al., 2018]
Kannan, H., Kurakin, A., and Goodfellow, I. J.
Adversarial logit pairing.
CoRR, abs/1803.06373, 2018.
URL <http://arxiv.org/abs/1803.06373>.

[Kocetkov et al., 2022]
Kocetkov, D., Li, R., Ben Allal, L., Li, J., Mou, C., Muñoz Ferrandis, C.,
Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., Bahdanau, D., von Werra, L.,
and de Vries, H.
The stack: 3 tb of permissively licensed source code.
Preprint, 2022.

[Li et al., 2023]
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone,
M., Akiki, C., Li, J., Chim, J., Liu, Q., Zheltonozhskii, E., Zhuo, T. Y.,
Wang, T., Dehaene, O., Davaadorj, M., Lamy-Poirier, J., Monteiro, J.,
Shliazhko, O., Gontier, N., Meade, N., Zebaze, A., Yee, M., Umapathi, L. K.,
Zhu, J., Lipkin, B., Oblokulov, M., Wang, Z., V, R. M., Stillerman, J.,
Patel, S. S., Abulkhanov, D., Zocca, M., Dey, M., Zhang, Z.,
Moustafa-Fahmy, N., Bhattacharyya, U., Yu, W., Singh, S., Luccioni, S.,
Villegas, P., Kunakov, M., Zhdanov, F., Romero, M., Lee, T., Timor, N., Ding,
J., Schlesinger, C., Schoelkopf, H., Ebert, J., Dao, T., Mishra, M., Gu, A.,
Robinson, J., Anderson, C. J., Dolan-Gavitt, B., Contractor, D., Reddy, S.,
Fried, D., Bahdanau, D., Jernite, Y., Ferrandis, C. M., Hughes, S., Wolf, T.,
Guha, A., von Werra, L., and de Vries, H.
Starcoder: may the source be with you!
CoRR, abs/2305.06161, 2023.
URL <https://doi.org/10.48550/arXiv.2305.06161>.

[Liu et al., 2023]
Liu, S., Wu, B., Xie, X., Meng, G., and Liu, Y.
Contrabert: Enhancing code pre-trained models via contrastive
In 45th IEEE/ACM International Conference on Software
Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp.  2476–2487. IEEE, 2023.
URL <https://doi.org/10.1109/ICSE48619.2023.00207>.

[Luo et al., 2023]
Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q.,
and Jiang, D.
Wizardcoder: Empowering code large language models with
CoRR, abs/2306.08568, 2023.
URL <https://doi.org/10.48550/arXiv.2306.08568>.

[Madry et al., 2018]
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A.
Towards deep learning models resistant to adversarial attacks.
In 6th International Conference on Learning Representations,
ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track
Proceedings. OpenReview.net, 2018.
URL <https://openreview.net/forum?id=rJzIBfZAb>.

[Miller, 1992]
Miller, G. A.
WordNet: A lexical database for English.
In Speech and Natural Language: Proceedings of a Workshop Held
at Harriman, New York, February 23-26, 1992, 1992.
URL <https://aclanthology.org/H92-1116>.

[Nijkamp et al., 2023]
Nijkamp, E., Hayashi, H., Xiong, C., Savarese, S., and Zhou, Y.
Codegen2: Lessons for training llms on programming and natural
ICLR, 2023a.

[Nijkamp et al., 2023]
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S.,
and Xiong, C.
Codegen: An open large language model for code with multi-turn
program synthesis.
ICLR, 2023b.

[Rozière et al., 2023]
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E.,
Adi, Y., Liu, J., Remez, T., Rapin, J., Kozhevnikov, A., Evtimov, I., Bitton,
J., Bhatt, M., Canton-Ferrer, C., Grattafiori, A., Xiong, W.,
Défossez, A., Copet, J., Azhar, F., Touvron, H., Martin, L., Usunier,
N., Scialom, T., and Synnaeve, G.
Code llama: Open foundation models for code.
CoRR, abs/2308.12950, 2023.
URL <https://doi.org/10.48550/arXiv.2308.12950>.

[Srikant et al., 2021]
Srikant, S., Liu, S., Mitrovska, T., Chang, S., Fan, Q., Zhang, G., and
O'Reilly, U.
Generating adversarial computer programs using optimized
In 9th International Conference on Learning Representations,
ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
URL <https://openreview.net/forum?id=PH5PH9ZO_4>.

[Suneja et al., 2023]
Suneja, S., Zhuang, Y., Zheng, Y., Laredo, J., Morari, A., and Khurana, U.
Incorporating signal awareness in source code modeling: An
application to vulnerability detection.
ACM Trans. Softw. Eng. Methodol., may 2023.
ISSN 1049-331X.
URL <https://doi.org/10.1145/3597202>.
Just Accepted.

[Tian et al., 2023]
Tian, Z., Chen, J., and Jin, Z.
Adversarial attacks on neural models of code via code difference
CoRR, abs/2301.02412, 2023.
URL <https://doi.org/10.48550/arXiv.2301.02412>.

[Wang et al., 2023]
Wang, D., Chen, B., Li, S., Luo, W., Peng, S., Dong, W., and Liao, X.
One adapter for all programming languages? adapter tuning for code
search and summarization.
In 45th IEEE/ACM International Conference on Software
Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp.  5–16. IEEE, 2023a.
URL <https://doi.org/10.1109/ICSE48619.2023.00013>.

[Wang et al., 2023]
Wang, S., Li, Z., Qian, H., Yang, C., Wang, Z., Shang, M., Kumar, V., Tan, S.,
Ray, B., Bhatia, P., Nallapati, R., Ramanathan, M. K., Roth, D., and Xiang,
Recode: Robustness evaluation of code generation models.
In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.),
Proceedings of the 61st Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto,
Canada, July 9-14, 2023, pp.  13818–13843. Association for Computational
Linguistics, 2023b.
URL <https://aclanthology.org/2023.acl-long.773>.

[Yang et al., 2022]
Yang, Z., Shi, J., He, J., and Lo, D.
Natural attack for pre-trained models of code.
In 44th IEEE/ACM 44th International Conference on Software
Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022, pp.  1482–1493. ACM, 2022.
URL <https://doi.org/10.1145/3510003.3510146>.

[Yefet et al., 2020]
Yefet, N., Alon, U., and Yahav, E.
Adversarial examples for models of code.
Proc. ACM Program. Lang., 40 (OOPSLA):0
162:1–162:30, 2020.
URL <https://doi.org/10.1145/3428230>.

[Zhang et al., 2020]
Zhang, H., Li, Z., Li, G., Ma, L., Liu, Y., and Jin, Z.
Generating adversarial examples for holding robustness of source code
processing models.
In The Thirty-Fourth AAAI Conference on Artificial
Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of
Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium
on Educational Advances in Artificial Intelligence, EAAI 2020, New York,
NY, USA, February 7-12, 2020, pp.  1169–1176. AAAI Press, 2020.
URL <https://ojs.aaai.org/index.php/AAAI/article/view/5469>.

[Zhang et al., 2022]
Zhang, H., Fu, Z., Li, G., Ma, L., Zhao, Z., Yang, H., Sun, Y., Liu, Y., and
Jin, Z.
Towards robustness of deep program processing models - detection,
estimation, and enhancement.
ACM Trans. Softw. Eng. Methodol., 310 (3):0
50:1–50:40, 2022.
URL <https://doi.org/10.1145/3511887>.

[Zhou et al., 2022]
Zhou, Y., Zhang, X., Shen, J., Han, T., Chen, T., and Gall, H. C.
Adversarial robustness of deep code comment generation.
ACM Trans. Softw. Eng. Methodol., 310 (4):0
60:1–60:30, 2022.
URL <https://doi.org/10.1145/3501256>.

§ FINE-TUNING SETTINGS

We train with $p=25\%$ perturbed data as CodeGen models has not been fine-tuned on the stack dataset.
For CodeGen-2B and CodeGen-6B, we set batch size to $256$ and fine-tune them for 10K and 5K steps, respectively, using the AdamW optimizer and a linear schedule with 500 warmup steps and a learning rate $2\times 10^{-5}$.
For CodeGen-350M, we set batch size to $512$ and fine-tune the model on half of the stack dataset (about 266K steps) using the FusedAdam optimizer and a linear schedule with 500 warmup steps and a learning rate $2\times 10^{-5}$.

We treat all the objective functions proposed in this paper equally, i.e., summing them up without reweighing.
For the temperature hyperparameter $\tau$ in contrastive learning, we set $\tau = 0.05$ for all experiments following ContraCLM. We set the dropout rate to $0.1$ for $\lossalpdrop$.

§ DETAILED RESULTS FOR EACH PERTURBATION TYPE

Table <ref> shows a detailed breakdown of robustness gain by finetuning with our approach for each perturbation type evaluated on 350M, 2B, and 6B CodeGen models.

§ QUALITATIVE EXAMPLES

In this section, we present qualitative examples to demonstrate the robustness improvements of our robust trained models. On these MBPP examples, 6B CodeGen baseline model fails to generate correct completions after applying the perturbations. Our robust trained model, on the other hand, can still successfully complete these problems. Here, we list examples for the top four perturbation types that we have achieved the most improvements (detailed numbers for each perturbation type can be found in Table <ref>).

DeadCode Insertion. For this perturbation type, ReCode perturbation will insert a redundant code block including an , a , or a . The models will usually be distracted by the inserted blocks, causing failure of completions. Here, in Listing <ref>, ReCode perturbation inserts a redundant  block at line 12-13. Even though the semantic meaning barely changes after this insertion, the 6B baseline model fails to generate the correct answer.

[language=python, caption=DeadCoderInsertion (6B CodeGen), label=qe-deadcode]
def test_distinct(data):
Write a python function to determine whether all the numbers are different from each other are not.
>>> test_distinct([1,5,7,9])
>>> test_distinct([2,4,5,5,7,9])
>>> test_distinct([1,2,3])
if len(data) == len(set(data)):
if False:
return True

# === completion by baseline model ===
return False

# === completion by our model ===
return True
return False

NewLine Insertion. For newline insertions, ReCode will randomly insert empty lines within prompts, and code models commonly perform poor robustness on newline insertions as reported in [Wang et al., 2023]. In Listing <ref>, 6B baseline model will generate low quality outputs if we simply insert two single empty lines at line 12 and line 15. Our models remain correct under the perturbation.

[language=python, caption=NewlineInsertion (6B CodeGen), label=qe-newline]
def count_first_elements(test_tup):
Write a function to find the element count that occurs before the record in the given tuple.
>>> count_first_elements((1, 5, 7, (4, 6), 10) )
>>> count_first_elements((2, 9, (5, 7), 11) )
>>> count_first_elements((11, 15, 5, 8, (2, 3), 8) )

for count, ele in enumerate(test_tup):

# === completion by baseline model ===
if isinstance(ele, tuple):
return count_first_elements(ele)

return count

# === completion by our model ===
if isinstance(ele, tuple):
return count
return len(test_tup)

Synonym Substitution. To perform synonym substitution on docstrings, ReCode will randomly select words within docstrings or comments. It then replaces the selected words to their synonyms according to WordNet [Miller, 1992]. In Listing <ref>, “have" is replaced to be “induce" and “given" to “yield". With only these two-word changes, 6B baseline model fails. In contrast, our model is much more robust to these changes.

[language=python, caption=SynonymSubstitution (6B CodeGen),label=qe-synonym]
def find_tuples(test_list, K):
Write a function to find tuples which induce all element divisible by k from the yield list of tuples.
>>> find_tuples([(6, 24, 12), (7, 9, 6), (12, 18, 21)], 6)
'[(6, 24, 12)]'
>>> find_tuples([(5, 25, 30), (4, 2, 3), (7, 8, 9)], 5)
'[(5, 25, 30)]'
>>> find_tuples([(7, 9, 16), (8, 16, 4), (19, 17, 18)], 4)
'[(8, 16, 4)]'

# === completion by baseline model ===
return '[' + ','.join(map(str, filter(lambda x: x[0]

# === completion by our model ===
return str(list(filter(lambda x: all(x[i]

CharCase Change for Function Name. For function name perturbation, ReCode CharCase change is to randomly replace characters in function name to their upper cases. In Listing <ref>, “remove_dirty_chars"" are replaced with “remOve_dIrty_cHarS". With only four characters changed to their upper cases, 6B baseline model will fail under such perturbation. Our model demonstrates better robustness performance.

[language=python, caption=CharCase change for function name (6B CodeGen), label=qe-funcchar]
def remOve_dIrty_cHarS(string, second_string):
Write a function to remove characters from the first string which are present in the second string.
>>> remOve_dIrty_cHarS("probasscurve", "pros")
>>> remOve_dIrty_cHarS("digitalindia", "talent")
>>> remOve_dIrty_cHarS("exoticmiles", "toxic")

# === completion by baseline model ===
return string.replace(second_string, "")

# === completion by our model ===
for char in second_string:
if char in string:
string = string.replace(char, "")
return string

§ DETAILED RESULTS FOR ABLATIONS

Tables <ref> and <ref> shows detailed comparison among different approaches across four perturbation classes.

§.§ Different Designs of ALP

This section compares different designs of ALP. In terms of the KL divergence loss, two approaches are considered: (1) optimizing both original and perturbed token prefixes simultaneously, i.e., bringing their output distributions closer at the same time, denoted as $\emph{Bo}$ (both sides), and (2) optimizing only the perturbed token prefix, i.e., only bringing the output distribution of the perturbed token prefix closer to the original one, denoted as $\emph{On}$ (one side).
Another aspect involves whether to optimize all prefixes or just the ones that are correctly predicted. The instance that optimizes all prefixes is named $\emph{Al}$ (all), while the one optimizing only correctly predicted prefixes is named $\emph{CO}$ (correct only). In summary, there are four different ALP designs (two by two).
Lines [9]-[12] in Table <ref> show that $\emph{On} + \emph{Al}$ achieves the best overall among the four design. Therefore, we use this design throughout our experiments.

§.§ Different Designs of ALPD

This section compares three different designs of ALP. We conduct two additional experiments: (1) dropout of 10% arbitrary tokens, denoted as $\emph{All}$, and (2) dropout of arbitrary tokens while following the same percentage as 10% of variable and function names, denoted as $\emph{AllS}$ (all stratified).
Comparing line $[2]$ with lines $[6]$ and $[7]$ in Table <ref>, we observe that $\lossalpdrop$ with $10\%$ dropout on names achieves the best overall and . Therefore, we use this design throughout our experiments.

§.§ Effectiveness of Combining Context-Free and Context-Sensitive Perturbations

Based on the ablation results, we choose to use $\lossmbatchaug + \lossalp + \lossalpdrop$ for all the models in Section <ref>.
Our approach involves training on the combination of context-free and context-sensitive perturbations.
Comparing the results of our combined approach on CodeGen-350M in Table <ref> with those in Table <ref> line $[5]$ and in Table <ref> line $[2]$, we observe an improvement in model robustness.
Specifically, our combined approach outperforms the other two approaches that focus solely on either context-free or context-sensitive perturbations in Docstring and Format.


#  A test of linearity of the ratio of dark matter to baryonic matter in
galaxy clusters

Varenya Upadhyaya<EMAIL_ADDRESS>Shantanu Desai  E-mail:
<EMAIL_ADDRESS>Department of Physics, Indian Institute of Technology,
Hyderabad, Telangana-502284, India

###### Abstract

We search for a linearity in the ratio of dark matter to baryonic matter as a
function of radius for galaxy clusters, motivated by a recent result by Lovas
(2022), who has discovered such a linearity for a diverse suite of galaxies in
the SPARC sample. For our analysis, we used a sample of 54 non-cool core
clusters from the HIFLUGCS sample. We do not find any evidence for a linear
trend in the aforementioned ratio as a function of radius for individual
clusters. We then repeat this analysis for the stacked sample, which also does
not show this linearity. Therefore, the linear scaling found by Lovas is not a
universal property of dark matter haloes at all scales.

##  I Introduction

The dark matter problem is one of the most vexing problems in Modern Physics
and Astrophysics. Although dark matter constitutes about 25% of total energy
density of the universe and is a basic tenet of the $\Lambda$CDM model Aghanim
et al. (2020), its identity is still a mystery, despite close to 100 years of
evidence Bertone and Hooper (2018). There is also no laboratory evidence for
some of the most well motivated dark matter candidates such as WIMP or the
axion or through indirect searches which have only reported null results
Merritt (2017); Gaskins (2016); Desai et al. (2004); Boveia and Doglioni
(2018). Furthermore, the currently established $\Lambda$CDM model has also
faced some tensions at scales smaller than 1 Mpc, such as the core/cusp
problem, missing satellite problem, too big to fail problem, satellites plane
problem etc Bullock and Boylan-Kolchin (2017); Weinberg et al. (2015). Data on
galactic scales for spiral galaxies have also revealed some intriguing
deterministic scaling relations or correlations such as Radial Acceleration
Relation McGaugh et al. (2016), constancy of dark matter halo surface density
Donato et al. (2009), which cannot be trivially reproduced using the
$\Lambda$CDM model. Therefore a large number of alternatives to the standard
$\Lambda$CDM model have emerged such as Self-Interacting Dark Matter Adhikari
et al. (2022), Superfluid dark matter Berezhiani et al. (2018), Warm Dark
Matter Colín et al. (2000), Wave (or Fuzzy) Dark Matter Ferreira (2021),
Flavor Mixed Dark Matter Medvedev (2014), modified gravity which obviates the
need for dark matter Famaey and McGaugh (2012); Banik and Zhao (2022), etc.

In order to ascertain if these small scales issues need a paradigm shift from
$\Lambda$CDM or whether they can be explained using feedback processes from
baryons, one should test the aforementioned relations with a plethora of dark
matter dominated systems. We know that Radial Acceleration relation does not
hold for dwarf disk spirals, low surface brightness galaxies, elliptical
galaxies and galaxy groups Di Paolo et al. (2019); Chan et al. (2022); Gopika
and Desai (2021)

Another class of objects to test if the above correlations are universal are
galaxy clusters. Galaxy clusters are the most massive virialized objects in
the universe Allen et al. (2011); Kravtsov and Borgani (2012); Vikhlinin et
al. (2014), which have proved to be wonderful laboratories for galaxy
evolution, Cosmology and Fundamental Physics Allen et al. (2011); Vikhlinin et
al. (2014); Desai (2018); Bora and Desai (2021a); Bora and Desai (2021b); Bora
et al. (2022). Galaxy clusters were the first astrophysical source which gave
evidence for dark matter Zwicky (1933). They have been shown to rule out the
universality of Radial Acceleration Relation Pradyumna et al. (2021); Tian et
al. (2020); Chan and Del Popolo (2020); Pradyumna and Desai (2021); Eckert et
al. (2022); Chan and Law (2022) and constancy of halo surface density Chan
(2014); Gopika and Desai (2020, 2021). However, some scaling relations which
are common between galaxies and galaxy clusters have also been found Chan
(2022). Furthermore, galaxy clusters are known to exhibit many bivariate and
fundamental plane scaling relations with tight scatter (See Lovisari and
Maughan (2022) for a recent review). Some of these include gas temperature vs
total mass relation Kravtsov et al. (2006), fundamental plane between
characteristic radius, mass and temperature Fujita et al. (2018); Pradyumna
and Desai (2022), hot gas mass v/s total mass at intermediate radii Chan
(2020), etc. Therefore, they continue to be very good laboratories for testing
some of the anomalies and deterministic scaling relations obtained using
spiral galaxies.

Most recently, Lovas (L22, hereafter) Lovas (2022) has found another
intriguing property of dark matter haloes on galactic scales. Using 175
galaxies from the SPARC sample, L22 showed that the ratio of dark matter to
baryonic matter $\left(\frac{M_{dark}(r)}{M_{bar}(r)}\right)$ shows a linear
scaling as a function of radius. The residuals from the linear relation are
small with $\sigma$=0.31, where $\sigma$ was obtained from the width of the
best-fit Gaussian to the normalized histogram of residuals from the linear
relation. When the radius is scaled using the radius ($r_{1}$) at which the
aforementioned ratio is unity ($r_{norm}=mr$) where $m=1/r_{1}$, they find
that $\left(\frac{M_{dark}}{M_{bar}}\right)=r_{norm}$. L22 showed that this
linear trend was seen for galaxies of diverse morphologies in the SPARC sample
from pressure supported to rotationally supported galaxies. This linear
scaling extends up to the last available kinematic data point Lovas (2022). It
is not immediately obvious whether the linear trend for the ratio of dark
matter to baryonic matter can be reproduced from the current theory of
structure formation involving the $\Lambda$CDM model. In order to test whether
this linear trend as a function of radius is universal property of all dark
matter haloes, we test this ansatz using galaxy clusters from the HIFLUGCS
sample Chen et al. (2007).

The outline of this manuscript is as follows. The HIFLUGCS cluster sample as
well as the ensuing analysis procedure is described in Sect. II. Our results
are presented in Sect. III. We conclude in Sect. IV.

##  II Dataset and Analysis

The HIFLUGCS cluster sample consists of 106 galaxy clusters and groups based
on ROSAT and ASCA observations Chen et al. (2007); Reiprich and Böhringer
(2002). Among these, 92 clusters have temperatures determined using X-ray
spectroscopy whereas the remaining clusters had temperatures estimated using
$L_{x}-T$ correlations obtained in Ref. Markevitch (1998). The X-ray surface
brightness profiles have been obtained for 36 clusters from the RASS survey
and from pointed ROSAT observations for the remaining 70. The imaging has been
done up to a radius of $r_{500}$. This final sample consists of about 52 cool
core clusters and 54 non-cool core clusters. More details on the observations
and data reductions can be found in Refs. Chen et al. (2007); Reiprich and
Böhringer (2002).

The X-ray surface brightness profile was fit to both a single-$\beta$ and a
double-$\beta$. The number density for a single-$\beta$ profile was was given
by Cavaliere and Fusco-Femiano (1976)

$n(r)=n_{0}\left(1+\frac{r^{2}}{r_{c}^{2}}\right)^{-3\beta/2},$ (1)

where $n_{0}$ is the central density, $r_{c}$ the core radius and $\beta$ is
the index parameter. Assuming that the cluster gas is in hydrostatic
equilibrium, the total cluster mass is given by Chen et al. (2007)

$M_{tot}(r)=\frac{3\beta T_{h}r}{G\mu
m_{p}}\frac{(r/r_{c})^{2}}{1+(r/r_{c})^{2}},$ (2)

where $\mu$ is the mean molecular weight equal to 0.59 Chen et al. (2007). Eq.
2 assumes that the gas is at constant temperature given by $T_{h}$. This
assumption is true for non-cool core clusters Reiprich et al. (2013), where
the temperature gradient in the hot gas is less than 8% Hudson et al. (2010).
However, the isothermality assumption will not be applicable to the central
regions in cool core clusters. Cool core clusters are also affected by AGN
cooling and feedback near the center. Therefore, in our analysis, we only
choose non-cool core clusters. All clusters with cooling time less than Hubble
time (13 Gyr) have been identified as cool-core clusters and not used for this
work.

For the HIFLUGS sample, 49 clusters show lower $\chi^{2}$ with a
double-$\beta$ model fit to the gas density profile. However, no automated
model selection techniques (such as AIC, BIC or Bayes factors) have been used
to quantify the significance of goodness of fit of the double-$\beta$ profile
with respect to single-$\beta$ model. Therefore, it is hard to judge whether
the double-$\beta$ profile is an improvement for these clusters. Nevertheless,
for this work we restrict our analysis to clusters for which single-$\beta$
profile provides the minimum $\chi^{2}$. The best-fit values of all the
parameters (along with the best-fit statistical errors obtained from the fit)
needed to estimate the hydrostatic mass from Eq. 2 are provided in Tables 1
and 2 of Chen et al. (2007). Our final sample used for this work consists of
54 non-cool core clusters.

To test the linearity we need the total dark matter which can be obtained from
the total cluster mass as follows:

$M_{DM}=M_{tot}(r)-M_{gas}(r)-M_{stars}(r),$ (3)

where $M_{gas}$ is the total gas mass and $M_{stars}$ is the total mass due to
stars. The gas mass can be obtained by assuming spherical symmetry

$M_{gas}(r)=m_{g}\int 4\pi r^{2}n(r^{\prime})dr^{\prime},$ (4)

where $n(r)$ is the gas density given in Eq. 1, $m_{g}$ is the average mass of
each gas particle and is given by $\mu_{g}m_{p}$, where $\mu_{g}$=1.1548 is
the mean molecular weight per electron Bahar et al. (2022). Although
analytical expressions for the integral in Eq. 4 have been provided in
literature Chan and Law (2022), here we evaluate the integrals numerically to
get the gas density at a given radius ($r$). To obtain $M_{stars}$, we used
the stellar to gas mass scaling relations obtained in Ref. Chiu et al. (2018).
This relation was derived using a sample of 91 SPT-SZ clusters detected up to
a range of 1.3. This relation includes contribution from the BCG as well as
cluster member galaxies within $r_{500}$. We have also propagated the
statistical uncertainty in the stellar mass to gas mass relation provided in
Ref. Chiu et al. (2018). The total baryonic mass ($M_{bar}$) is then given by
the sum of gas and stellar mass ($M_{bar}=M_{gas}+M_{stars}$)

##  III Results

###  III.1 Single cluster analysis

We calculate the ratio of dark matter to baryonic matter mass at 10 uniformly
spaced values from $r=0$ to $r=r_{500}$. We then fit this ratio
($M_{DM}/M_{bar}$) to a simple linear multiple of $r$ ($M_{DM}/M_{bar}=kr$)
and obtain the best-fit value of $k$ using $\chi^{2}$ minimization as follows:

$\chi^{2}=\sum\left(\frac{M_{DM}/M_{bar}-kr}{\sigma_{i}}\right)^{2},$ (5)

where $\sigma_{i}$ is the error in the ratio $M_{DM}/M_{bar}$ and is obtained
by error propagation after taking into account the statistical uncertainties
in $M_{bar}$, $M_{tot}$, and $M_{stars}$. Since we are selecting 10 points per
cluster with one free parameter, we have a total of 9 degrees of freedom (dof)
for all the clusters. The best-fit $\chi^{2}$ values along with their $p$
values obtained from the $\chi^{2}$ cdf for each of the 54 non-cool core
clusters can be found in Table 1. As we can see, most clusters (34) from our
sample have $\chi^{2}$/dof much greater than one and very small $p$-values.
This shows that the linearity ansatz cannot be correct and the ratio of dark
to baryonic matter mass in galaxy clusters shows a non-linear trend as a
function of $r$, unlike the galaxies in the SPARC sample. A plot of this ratio
as a function of $r$ can be found for some sample clusters in Fig. 1. We
however have about 20 clusters with $\chi^{2}/$dof$\leq 1$. These are due to
large observational errors in the ratio, as opposed to the ratio showing a
linear trend. A plot for some of these clusters with $\chi^{2}$/dof less than
one can be found in Fig. 2. However, even for these clusters with low
$\chi^{2}$, visually we do not see a discerning linear trend as a function of
$r$.

Cluster Name | $\chi^{2}$ |  $\chi^{2}/$dof | p-value | Cluster Name | $\chi^{2}$ |  $\chi^{2}/$dof | p-value
---|---|---|---|---|---|---|---
A0119 | 144.8 | 16.09 | 0 | MKW8 | 7.3 | 0.81 | 0.7
A0399 | 10.3 | 1.14 | 0.42 | ZwCl 1215 | 9.4 | 1.04 | 0.5
A0400 | 163.8 | 18.2 | 0 | 3C 129 | 4.5 | 0.5 | 0.92
A0401 | 248.5 | 27.61 | 0 | A0548e | 130.2 | 14.47 | 0
A0576 | 2.4 | 0.27 | 0.99 | A0548w | 3.9 | 0.43 | 0.95
A0754 | 143.4 | 15.93 | 0 | A1775 | 61.8 | 6.87 | 0
A1367 | 127.2 | 14.13 | 0 | A1800 | 2.4 | 0.27 | 0.99
A1644 | 7.6 | 0.84 | 0.67 | A2319 | 195.3 | 21.7 | 0
A1650 | 8.8 | 0.98 | 0.56 | A2734 | 65.9 | 7.32 | 0
A1736 | 5.6 | 0.62 | 0.85 | A2877 | 9.6 | 1.07 | 0.48
A2065 | 1.1 | 0.12 | 1 | A3395n | 1.6 | 0.18 | 1
A2147 | 15.1 | 1.68 | 0.13 | A3528n | 73.4 | 8.16 | 0
A2163 | 84.8 | 9.42 | 0 | A3530 | 19.3 | 2.14 | 0.04
A2255 | 70.5 | 7.83 | 0 | A3532 | 50.9 | 5.66 | 0
A2256 | 76.3 | 8.48 | 0 | A3560 | 12.5 | 1.39 | 0.25
A2634 | 49.2 | 5.47 | 0 | A3627 | 39.4 | 4.38 | 0
A3158 | 147.3 | 16.37 | 0 | A3695 | 2.7 | 0.3 | 0.99
A3266 | 240.5 | 26.72 | 0 | A3822 | 6.4 | 0.71 | 0.78
A3376 | 25.1 | 2.79 | 0.01 | A3827 | 2 | 0.22 | 1
A3391 | 61.9 | 6.88 | 0 | A3888 | 6.2 | 0.69 | 0.79
A3395s | 5.3 | 0.59 | 0.87 | A3921 | 88.1 | 9.79 | 0
A3558 | 821 | 91.22 | 0 | IIZw$\sim$108 | 4.3 | 0.48 | 0.93
A3562 | 331.5 | 36.83 | 0 | OPHIUCHU | 90 | 10 | 0
A3667 | 449.7 | 49.97 | 0 | RXJ2344 | 8.6 | 0.96 | 0.57
COMA | 59.9 | 6.66 | 0 | S405 | 2.7 | 0.3 | 0.99
FORNAX | 22.2 | 2.47 | 0.01 | S636 | 5.8 | 0.64 | 0.83
IIIZw54 | 2.7 | 0.3 | 0.99 | TRIANGUL | 548.3 | 60.92 | 0
Table 1:  Table showing $\chi^{2}$ for 9 degrees of freedom along with the
$p$-value obtained using the $\chi^{2}$ cdf, while fitting the ratio of dark
to baryonic matter to a linear relation as a function of radius.

(a) $\chi^{2}=144.8$

(b) $\chi^{2}=90.0$

(c) $\chi^{2}=59.9$

(d) $\chi^{2}=22.2$

Figure 1: The mass ratio and profiles for four clusters from the HIFLUGCS
sample with large $\chi^{2}$ after fitting the ratio to linear function of the
radius. One can compare these with similar plots for the SPARC sample in Fig.
2 of Lovas (2022), where one can see a linear trend.

(a) $\chi^{2}=4.5$

(b) $\chi^{2}=4.3$

(c) $\chi^{2}=2.4$

(d) $\chi^{2}=2.7$

Figure 2: The mass ratio and profiles for four clusters with high errors and
$\chi^{2}/\text{dof}$ less than 1.

###  III.2 Stacked Analysis

In order to beat down the uncertainties in the individual cluster analysis, we
stack the data for all the non-cool core clusters. The stacking of profiles
from different clusters assumes that the profile is universal, which implies
that combining the data from the clusters in a consistent way would increase
the significance of any putative correlation. Since clusters are expected to
be self-similar objects Kaiser (1986), this is a reasonable assumption even
though the ratio in each cluster is non-linear. For the stacked analysis, we
scale the radius by $r_{500}$ and calculate the ratio of dark to baryonic
matter at 10 equally spaced values of $r/r_{500}$ between 0 and 1. The mean
ratio $\bar{R}$ at each rescaled radius is given by the weighted mean

$\bar{R}=\frac{\sum\limits_{i=1}^{N}R_{i}/\sigma_{i}^{2}}{\sum\limits_{i=1}^{N}1/\sigma_{i}^{2}},$
(6)

where $R_{i}$ and $\sigma_{i}$ are the associated values of the ratio and its
uncertainty at each rescaled radius. The uncertainty in $\bar{R}$
($\sigma_{R}$) is given by

$\sigma_{R}^{2}=\frac{1}{\sum\limits_{i=1}^{N}1/\sigma_{i}^{2}}$ (7)

We then stack the data for all the non-cool core clusters using 10 equally
spaced values of $r/r_{500}$ from 0 to 1. This plot can be found in Fig. 3. As
we can see, the stacked ratio does not obey a linear trend and we find
appreciable deviations from a linear trend with $\chi^{2}$/dof equal to 1512.
Therefore, we find that the dark to baryonic matter ratio for the stacked
sample does not show a linear trend.

Figure 3: The stacked data points for all the clusters along with the ”best”
fit line. As we can see the fit is quite bad with best-fit $\chi^{2}$ equal to
8397.7 for 9 degrees of freedom.

###  III.3 Discussion

For our analysis, we have propagated the statistical errors in the best-fit
parameters. We now discuss possible systematic effects which could affect our
conclusions. The main assumption used to obtain the total cluster mass and
thereby the dark matter mass is that of hydrostatic equilibrium. This involves
a number of assumptions, one of them being that the gas pressure is purely
thermal Gianfagna et al. (2021, 2023). It has been shown using simulations
that the hydrostatic mass can be underestimated with respect to the true mass
by 20-30% due to non-hydrostatic pressure support Gianfagna et al. (2021). One
would need to compare the X-ray masses with lensing masses to get a robust
estimate of the hydrostatic bias in order to evaluate its impact on our
underlying results. Another assumption we have made in calculating the gas
mass is to assume a constant temperature. Although this assumption is not true
for cool-core clusters (which we have omitted), it should be sound for non
cool-core clusters. However, it could introduce systematic effects up to 3% in
the estimation of central density Chan (2014). Finally, for some clusters
there could be systematics due to single-$\beta$ profile, instead of a
double-$\beta$ profile. The impact of this assumption could also be checked by
redoing the analysis using the gas mass obtained by fitting the pressure
profile to a double-$\beta$ profile.

Finally, another possible systematic could be due to the effect of stellar
mass. Over the years, many empirical relations have been used to model the
stellar mass contribution of the total mass budget of clusters Giodini et al.
(2009); Lin et al. (2012); Andreon (2015); Palmese et al. (2020). However, the
stellar mass is only a small part of the total baryonic mass budget, so using
the stellar mass relations from Ref. Chiu et al. (2018) should not impact our
results.

##  IV Conclusions

Recently, L22 found that the ratio of dark matter to baryonic matter shows a
linear trend with radius upto the last measurable data point for a diverse
class of galaxies in the SPARC sample with rms scatter of 0.31 Lovas (2022).
It is not obvious whether this linear trend is a prediction of CDM haloes. In
order to ascertain if this relation is a universal property of all dark matter
haloes, we carry out a similar test for galaxy clusters.

For this purpose we use a sample of 54 non-cool core clusters from the
HIFLUGCS sample, which are in hydrostatic equilibrium. We used an isothermal
temperature profile and single-$\beta$ model for the gas density (cf. Eq. 1)
to calculate the gas mass. The total mass is obtained by assuming the clusters
are in hydrostatic equilibrium (cf. Eq. 2.) The stellar mass and gas mass were
obtained using the scaling relations between the stellar and gas mass derived
in Chiu et al. (2018), and by assuming a spherical symmetry (cf. Eq. 4)
respectively. The dark mass is obtained by subtracting the gas and stellar
masses from the total mass. Using this, we can calculate the ratio of dark to
baryonic matter as function of radius from the cluster center up to $r_{500}$.

We then look for a linear trend in this ratio and quantifying any deviations
using $\chi^{2}$. The best-fit $\chi^{2}$ value for each of the 54 clusters
can be found in Table 1. As we can see most of the clusters have
$\chi^{2}$/dof much greater than one, indicating that the ratio of the dark
matter to baryonic matter does not scale linearly with radius. The plots for
some of these clusters can be found in Fig. 1. About 20 clusters have
$\chi^{2}$/dof less than one. The ratio as a function of radius for some of
these can be found in Fig. 2. The low $\chi^{2}$ for these clusters is because
of large error bars and by eye, we do not discern a linear trend. Therefore,
none of the individual clusters show evidence for linearity in the ratio of
dark to baryonic matter as a function of radius. Finally, we stack the data as
a function of radius scaled by $r_{500}$, and calculated the weighted mean of
this ratio for the entire cluster sample. This ratio can be found in Fig. 3.
We find that the stacked ratio also does not scale linearly as a function of
radius. We have also discussed some possible systematic errors which we have
not considered.

Therefore, we conclude the the linear ratio for the dark matter to baryonic
matter found by L22 for galaxies in the SPARC sample cannot be a universal
property of all dark matter dominated systems, since galaxy clusters do not
show a similar linear trend.

## Acknowledgements

We are grateful to Man-Ho Chan for helping us correct a mistake in the first
draft of the manuscript. We also acknowledge the anonymous referee for
constructive feedback and comments on our manuscript.

## References

  * Aghanim et al. (2020) N. Aghanim et al. (Planck), Astron. Astrophys. 641, A6 (2020), [Erratum: Astron.Astrophys. 652, C4 (2021)], eprint 1807.06209.
  * Bertone and Hooper (2018) G. Bertone and D. Hooper, Reviews of Modern Physics 90, 045002 (2018), eprint 1605.04909.
  * Merritt (2017) D. Merritt, Studies in the History and Philosophy of Modern Physics 57, 41 (2017), eprint 1703.02389.
  * Gaskins (2016) J. M. Gaskins, Contemporary Physics 57, 496 (2016), eprint 1604.00014.
  * Desai et al. (2004) S. Desai et al. (Super-Kamiokande), Phys. Rev. D 70, 083523 (2004), eprint hep-ex/0404025.
  * Boveia and Doglioni (2018) A. Boveia and C. Doglioni, Annual Review of Nuclear and Particle Science 68, 429 (2018), eprint 1810.12238.
  * Bullock and Boylan-Kolchin (2017) J. S. Bullock and M. Boylan-Kolchin, Ann. Rev. Astron. Astrophys. 55, 343 (2017), eprint 1707.04256.
  * Weinberg et al. (2015) D. H. Weinberg, J. S. Bullock, F. Governato, R. Kuzio de Naray, and A. H. G. Peter, Proceedings of the National Academy of Science 112, 12249 (2015), eprint 1306.0913.
  * McGaugh et al. (2016) S. S. McGaugh, F. Lelli, and J. M. Schombert, Phys. Rev. Lett. 117, 201101 (2016), eprint 1609.05917.
  * Donato et al. (2009) F. Donato, G. Gentile, P. Salucci, C. Frigerio Martins, M. I. Wilkinson, G. Gilmore, E. K. Grebel, A. Koch, and R. Wyse, Mon. Not. R. Astron. Soc. 397, 1169 (2009), eprint 0904.4054.
  * Adhikari et al. (2022) S. Adhikari, A. Banerjee, K. K. Boddy, F.-Y. Cyr-Racine, H. Desmond, C. Dvorkin, B. Jain, F. Kahlhoefer, M. Kaplinghat, A. Nierenberg, et al., arXiv e-prints arXiv:2207.10638 (2022), eprint 2207.10638.
  * Berezhiani et al. (2018) L. Berezhiani, B. Famaey, and J. Khoury, JCAP 2018, 021 (2018), eprint 1711.05748.
  * Colín et al. (2000) P. Colín, V. Avila-Reese, and O. Valenzuela, Astrophys. J. 542, 622 (2000), eprint astro-ph/0004115.
  * Ferreira (2021) E. G. M. Ferreira, Astron and Astrophys Reviews 29, 7 (2021), eprint 2005.03254.
  * Medvedev (2014) M. V. Medvedev, Phys. Rev. Lett. 113, 071303 (2014), eprint 1305.1307.
  * Famaey and McGaugh (2012) B. Famaey and S. S. McGaugh, Living Reviews in Relativity 15, 10 (2012), eprint 1112.3960.
  * Banik and Zhao (2022) I. Banik and H. Zhao, Symmetry 14, 1331 (2022), eprint 2110.06936.
  * Di Paolo et al. (2019) C. Di Paolo, P. Salucci, and J. P. Fontaine, Astrophys. J. 873, 106 (2019), eprint 1810.08472.
  * Chan et al. (2022) M. H. Chan, S. Desai, and A. Del Popolo, PASJ 74, 1441 (2022), eprint 2205.07515.
  * Gopika and Desai (2021) K. Gopika and S. Desai, Physics of the Dark Universe 33, 100874 (2021), eprint 2106.07294.
  * Allen et al. (2011) S. W. Allen, A. E. Evrard, and A. B. Mantz, Ann. Rev. Astron. Astrophys. 49, 409 (2011), eprint 1103.4829.
  * Kravtsov and Borgani (2012) A. V. Kravtsov and S. Borgani, Ann. Rev. Astron. Astrophys. 50, 353 (2012), eprint 1205.5556.
  * Vikhlinin et al. (2014) A. A. Vikhlinin, A. V. Kravtsov, M. L. Markevich, R. A. Sunyaev, and E. M. Churazov, Physics Uspekhi 57, 317-341 (2014).
  * Desai (2018) S. Desai, Physics Letters B 778, 325 (2018), eprint 1708.06502.
  * Bora and Desai (2021a) K. Bora and S. Desai, JCAP 2021, 012 (2021a), eprint 2008.10541.
  * Bora and Desai (2021b) K. Bora and S. Desai, JCAP 2021, 052 (2021b), eprint 2104.00974.
  * Bora et al. (2022) K. Bora, R. F. L. Holanda, S. Desai, and S. H. Pereira, European Physical Journal C 82, 17 (2022), eprint 2106.15805.
  * Zwicky (1933) F. Zwicky, Helvetica Physica Acta 6, 110 (1933).
  * Pradyumna et al. (2021) S. Pradyumna, S. Gupta, S. Seeram, and S. Desai, Physics of the Dark Universe 31, 100765 (2021), eprint 2011.06421.
  * Tian et al. (2020) Y. Tian, K. Umetsu, C.-M. Ko, M. Donahue, and I. N. Chiu, Astrophys. J. 896, 70 (2020), eprint 2001.08340.
  * Chan and Del Popolo (2020) M. H. Chan and A. Del Popolo, Mon. Not. R. Astron. Soc. p. 218 (2020), eprint 2001.06141.
  * Pradyumna and Desai (2021) S. Pradyumna and S. Desai, Physics of the Dark Universe 33, 100854 (2021), eprint 2107.05845.
  * Eckert et al. (2022) D. Eckert, S. Ettori, E. Pointecouteau, R. F. J. van der Burg, and S. I. Loubser, Astron. & Astrophys. 662, A123 (2022), eprint 2205.01110.
  * Chan and Law (2022) M. H. Chan and K. C. Law, Phys. Rev. D 105, 083003 (2022), eprint 2203.15217.
  * Chan (2014) M. H. Chan, Mon. Not. R. Astron. Soc. 442, L14 (2014), eprint 1403.4352.
  * Gopika and Desai (2020) K. Gopika and S. Desai, Physics of the Dark Universe 30, 100707 (2020), eprint 2006.12320.
  * Chan (2022) M. H. Chan, arXiv e-prints arXiv:2212.01018 (2022), eprint 2212.01018.
  * Lovisari and Maughan (2022) L. Lovisari and B. J. Maughan, arXiv e-prints arXiv:2202.07673 (2022), eprint 2202.07673.
  * Kravtsov et al. (2006) A. V. Kravtsov, A. Vikhlinin, and D. Nagai, Astrophys. J. 650, 128 (2006), eprint astro-ph/0603205.
  * Fujita et al. (2018) Y. Fujita, K. Umetsu, E. Rasia, M. Meneghetti, M. Donahue, E. Medezinski, N. Okabe, and M. Postman, Astrophys. J. 857, 118 (2018), eprint 1801.03943.
  * Pradyumna and Desai (2022) S. Pradyumna and S. Desai, JCAP 2022, 058 (2022), eprint 2112.10478.
  * Chan (2020) M. H. Chan, Physics of the Dark Universe 28, 100478 (2020), eprint 2001.08863.
  * Lovas (2022) S. Lovas, Mon. Not. R. Astron. Soc. 514, L56 (2022), eprint 2206.11431.
  * Chen et al. (2007) Y. Chen, T. H. Reiprich, H. Böhringer, Y. Ikebe, and Y. Y. Zhang, Astron. & Astrophys. 466, 805 (2007), eprint astro-ph/0702482.
  * Reiprich and Böhringer (2002) T. H. Reiprich and H. Böhringer, Astrophys. J. 567, 716 (2002), eprint astro-ph/0111285.
  * Markevitch (1998) M. Markevitch, Astrophys. J. 504, 27 (1998), eprint astro-ph/9802059.
  * Cavaliere and Fusco-Femiano (1976) A. Cavaliere and R. Fusco-Femiano, Astron. & Astrophys. 49, 137 (1976).
  * Reiprich et al. (2013) T. H. Reiprich, K. Basu, S. Ettori, H. Israel, L. Lovisari, S. Molendi, E. Pointecouteau, and M. Roncarelli, Space Science Revs. 177, 195 (2013), eprint 1303.3286.
  * Hudson et al. (2010) D. S. Hudson, R. Mittal, T. H. Reiprich, P. E. J. Nulsen, H. Andernach, and C. L. Sarazin, Astron. & Astrophys. 513, A37 (2010), eprint 0911.0409.
  * Bahar et al. (2022) Y. E. Bahar, E. Bulbul, N. Clerc, V. Ghirardini, A. Liu, K. Nandra, F. Pacaud, I. N. Chiu, J. Comparat, J. Ider-Chitham, et al., Astron. & Astrophys. 661, A7 (2022), eprint 2110.09534.
  * Chiu et al. (2018) I. Chiu, J. J. Mohr, M. McDonald, S. Bocquet, S. Desai, M. Klein, H. Israel, M. L. N. Ashby, A. Stanford, B. A. Benson, et al., Mon. Not. R. Astron. Soc. 478, 3072 (2018), eprint 1711.00917.
  * Kaiser (1986) N. Kaiser, Mon. Not. R. Astron. Soc. 222, 323 (1986).
  * Gianfagna et al. (2021) G. Gianfagna, M. De Petris, G. Yepes, F. De Luca, F. Sembolini, W. Cui, V. Biffi, F. Kéruzoré, J. Macías-Pérez, F. Mayet, et al., Mon. Not. R. Astron. Soc. 502, 5115 (2021), eprint 2010.03634.
  * Gianfagna et al. (2023) G. Gianfagna, E. Rasia, W. Cui, M. De Petris, G. Yepes, A. Contreras-Santos, and A. Knebe, Mon. Not. R. Astron. Soc. 518, 4238 (2023), eprint 2211.08372.
  * Giodini et al. (2009) S. Giodini, D. Pierini, A. Finoguenov, G. W. Pratt, H. Boehringer, A. Leauthaud, L. Guzzo, H. Aussel, M. Bolzonella, P. Capak, et al., Astrophys. J. 703, 982 (2009), eprint 0904.0448.
  * Lin et al. (2012) Y.-T. Lin, S. A. Stanford, P. R. M. Eisenhardt, A. Vikhlinin, B. J. Maughan, and A. Kravtsov, Astrophys. J. Lett. 745, L3 (2012), eprint 1112.1705.
  * Andreon (2015) S. Andreon, Astron. & Astrophys. 575, A108 (2015), eprint 1501.01814.
  * Palmese et al. (2020) A. Palmese, J. Annis, J. Burgad, A. Farahi, M. Soares-Santos, B. Welch, M. da Silva Pereira, H. Lin, S. Bhargava, D. L. Hollowood, et al., Mon. Not. R. Astron. Soc. 493, 4591 (2020), eprint 1903.08813.