# Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning

Naibin Gu1,2, Peng Fu1,2, Xiyu Liu1,2, Bowen Shen1,2, Zheng Lin1,2, Weiping
Wang1
1Institute of Information Engineering, Chinese Academy of Sciences, Beijing,
China
2School of Cyber Security, University of Chinese Academy of Sciences, Beijing,
China
<EMAIL_ADDRESS>
Corresponding author: Peng Fu.

###### Abstract

Parameter-efficient fine-tuning (PEFT) has emerged as the predominant
technique for fine-tuning in the era of large language models. However,
existing PEFT methods still have inadequate training efficiency. Firstly, the
utilization of large-scale foundation models during the training process is
excessively redundant for certain fine-tuning tasks. Secondly, as the model
size increases, the growth in trainable parameters of empirically added PEFT
modules becomes non-negligible and redundant, leading to inefficiency. To
achieve task-specific efficient fine-tuning, we propose the Light-PEFT
framework, which includes two methods: Masked Early Pruning of the Foundation
Model and Multi-Granularity Early Pruning of PEFT. The Light-PEFT framework
allows for the simultaneous estimation of redundant parameters in both the
foundation model and PEFT modules during the early stage of training. These
parameters can then be pruned for more efficient fine-tuning. We validate our
approach on GLUE, SuperGLUE, QA tasks, and various models. With Light-PEFT,
parameters of the foundation model can be pruned by up to over 40%, while
still controlling trainable parameters to be only 25% of the original PEFT
method. Compared to utilizing the PEFT method directly, Light-PEFT achieves
training and inference speedup, reduces memory usage, and maintains comparable
performance and the plug-and-play feature of PEFT111Our code is available at
https://github.com/gccnlp/Light-PEFT..

Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning

Naibin Gu1,2, Peng Fu1,2††thanks:  Corresponding author: Peng Fu., Xiyu
Liu1,2, Bowen Shen1,2, Zheng Lin1,2, Weiping Wang1 1Institute of Information
Engineering, Chinese Academy of Sciences, Beijing, China 2School of Cyber
Security, University of Chinese Academy of Sciences, Beijing, China
<EMAIL_ADDRESS>

##  1 Introduction

Large-scale pre-trained language models have demonstrated outstanding
performance in various natural language processing domains (Liu et al., 2019;
Brown et al., 2020; Touvron et al., 2023; OpenAI, 2023). Along with the
performance improvements, the scale of model parameters continues to grow,
making the cost of fine-tuning models increasingly expensive. Moreover, the
practice of maintaining a separate copy of the large model for each task in
conventional fine-tuning incurs substantial storage costs.

To address these challenges, parameter-efficient fine-tuning (PEFT) has been
proposed: freezing most parameters of the foundation model and fine-tuning
only a small number of parameters (Houlsby et al., 2019; Li and Liang, 2021;
Liu et al., 2022a; Hu et al., 2022), thereby reducing the computational
resource requirements during training and performing nearly full-parameter
fine-tuning. In addition, this technique eliminates the need to save an entire
model copy for each task. During inference, task-specific models can be
obtained by switching directly to the appropriate parameter-efficient module
for the given task.

However, the training efficiency of existing PEFT methods still needs
improvement. The first problem lies in the excessive redundancy of using a
large-scale foundation model during fine-tuning for specific tasks, which
results in substantial computational costs. A typical strategy is to integrate
PEFT with quantization (Dettmers et al., 2023; Kim et al., 2023). Nonetheless,
these methods only quantize parameters to low-bit in memory, without reducing
the number of parameters and they still need to be dequantized to high-bit
during training, leading to wasted training time. Another more direct approach
for reducing parameters is model structured pruning (Hedegaard et al., 2022;
Zhao et al., 2023). However, most methods mainly focus on the inference
efficiency of the model, which means they may result in higher training costs.

The second problem is that as the size of the foundation model increases, the
number of parameters in added trainable modules also increases significantly.
This introduces a lot of redundancy in trainable parameters, leading to
inefficiency in fine-tuning. For instance, the commonly used methods LoRA (Hu
et al., 2022) and QLoRA (Dettmers et al., 2023) empirically insert the low-
rank modules onto fixed weight. However, there is no need to uniformly add
trainable modules of the same rank to all weights for fine-tuning each task.
An improved approach is the dynamic rank method (Zhang et al., 2023; Valipour
et al., 2023; Ding et al., 2023), which adaptively allocates module parameters
by progressively calculating the importance of the rank during training.
However, these methods require continuous estimation during training and show
limited improvement in actual training efficiency.

In this paper, we introduce a novel framework named Light-PEFT, which aims to
enhance the efficiency of the PEFT technique during fine-tuning. The framework
consists of two methods: Masked Early Pruning of Foundation Model and Multi-
Granularity Early Pruning of PEFT. In the early training stage, Light-PEFT
estimates redundant parameters in both the foundation model (heads and
intermediate dimensions) and the PEFT modules (module importance and rank
importance) simultaneously. Structured pruning is used to eliminate this
redundancy, resulting in a lighter foundation model and PEFT module for more
efficient fine-tuning.

To validate the effectiveness of our Light-PEFT framework, we conduct
extensive evaluations on various foundation models (RoBERTa, OPT-1.3B,
OPT-6.7B), different PEFT structures (LoRA, Adapter), and on diverse
benchmarks (GLUE, SuperGLUE, and question-answering tasks). The empirical
results indicate that the proposed Light-PEFT framework outperforms other
baseline methods in performance. It significantly improves training efficiency
that reduces training memory usage by 39% and accelerates training to
1.6$\times$. Additionally, the Light-PEFT framework improves inference
efficiency that reduces inference memory by 48% and increases inference speed
to 1.6$\times$.

##  2 Related Works

###  2.1 Parameter-Efficient Fine-Tuning

Parameter-Efficient Fine-Tuning has been proposed to reduce the computational
cost of fine-tuning entire model parameters (Houlsby et al., 2019; Li and
Liang, 2021; Hu et al., 2022). Following works aim to further improve the
efficiency of PEFT.

Improvements to the PEFT module. The motivation behind of this category of
methods is that previous works often insert trainable modules empirically,
resulting in uniform ranks for all inserted modules that are not parameter-
efficient. AdaLoRA (Zhang et al., 2023) proposes obtaining the optimal rank
for each module by iteratively pruning ranks during training. DyLoRA (Valipour
et al., 2023) achieves this through dynamic training on a range of ranks.
AutoPEFT (Zhou et al., 2023) automatically selects PEFT configurations through
Bayesian optimization. Recently, SoRA (Ding et al., 2023) introduces masks on
the ranks and gradually makes each module sparse. However, all of these
methods gradually calculate the rank allocation during training, which does
not improve the actual training efficiency in fine-tuning. Our method
estimates the rank allocation for each module in the early stage of training
and utilizes the pruned parameter-efficient modules to improve training
efficiency during fine-tuning.

Improvements to the training paradigm of PEFT. To enhance training efficiency,
one idea is to further reduce the memory footprint during training. QLoRA
(Dettmers et al., 2023) and PEQA (Kim et al., 2023) reduce memory usage by
quantizing the foundation model, while LST (Sung et al., 2022) and MEFT (Liao
et al., 2023), respectively alleviate the memory footprint of intermediate
activations in the foundation model through methods ladder side-tuning and
reversible structures. Our approach is orthogonal to these methods from a
memory perspective and can be combined with them. We explore early-stage
pruning of the foundation model to reduce memory usage. Moreover, our approach
can lower computational costs, speed up training, and improve inference
efficiency.

Combining PEFT with pruning, most of works focus on improving inference
efficiency. PST (Li et al., 2022) and DSEE (Chen et al., 2023) propose
combining unstructured pruning and PEFT, which hardly achieves acceleration on
practical hardware. SPAs (Hedegaard et al., 2022) integrates structured
pruning of the foundation model with PEFT, while CPET Zhao et al. (2023)
proposes distilling knowledge into PEFT modules simultaneously with pruning to
reduce performance degradation. Concurrently to our works, APT (Zhao et al.,
2024) reduces the training cost of the CPET method, presenting more efficient
distillation and pruning. However, these methods, including APT, still require
higher training time and memory costs compared to the original PEFT methods.
Our approach aims to reduce the original PEFT training costs, including speed
and memory, by employing early-stage structured pruning to train a non-
redundant PEFT model efficiently, while improving inference efficiency
simultaneously.

###  2.2 Structured Pruning of Models

Model pruning has been proposed to compress redundant parameters in models
(LeCun et al., 1989; Kurtic et al., 2022; Liu et al., 2022b; Ma et al., 2023),
with structured pruning being the most straightforward method to achieve
acceleration on actual hardware. For the structured pruning of Transformer
models, the focus lies in pruning components of the model, such as attention
heads and feed-forward dimensions (Liu et al., 2021; Xia et al., 2022; Tao et
al., 2023; Xia et al., 2024). However, most structured pruning works require
additional costs during training to obtain smaller and more accurate models
for inference efficiency. In terms of training efficiency, You et al. (2020)
base on the lottery ticket hypothesis (Frankle and Carbin, 2019) and discover
the existence of early winning tickets in DNN models, allowing early pruning
to enhance subsequent training efficiency. Subsequently, Chen et al. (2021)
identify early tickets in BERT models (Devlin et al., 2019) to enhance the
efficiency of BERT’s pre-training and fine-tuning. We follow these works and
explore early pruning in parameter-efficient fine-tuning and generative
foundation models.

##  3 Preliminaries

###  3.1 Parameter-Efficient Fine-Tuning

In our framework, we choose two of the most widely used methods: Adapter
(Houlsby et al., 2019) and LoRA (Hu et al., 2022) to validate our approach.

Adapter. For each layer in the foundation model, including the attention sub-
layer and the feed-forward sub-layer, Adapter inserts a trainable MLP module
after each sub-layer. It consists of a down-projection layer
$W_{down}\in{R^{d\times r}}$, followed by a non-linear activation function
$f$, and finally an up-projection layer $W_{up}\in{R^{r\times d}}$, where $d$
is the hidden size of the foundation model, and $r$ is the bottleneck
dimension in the trainable module, with $r\ll d$. The Adapter method can be
formulated as follows:

$h\leftarrow h+f(hW_{down})W_{up}$ (1)

where $h$ is the output of the inserted sub-layer.

LoRA. For each linear weight matrix $W\in{R^{d\times d}}$ in the foundation
model, the LoRA method adds trainable MLP modules in parallel to $W$. The
trainable module includes a down-projection layer $W_{down}$ and an up-
projection layer $W_{up}$. The LoRA method can be be formulated as follows:

$h\leftarrow h+s\cdot XW_{down}W_{up}$ (2)

where $X$ represents the input to the linear weight matrix $W$ and $s$ is a
hyper-parameter scaling factor.

###  3.2 PEFT Training Efficiency

In this section, we present observations on the training efficiency of PEFT.
We utilize LoRA to observe the results on two foundation models, RoBERTa (Liu
et al., 2019) and OPT (Zhang et al., 2022). For training samples, we set the
length to 128 with a batch size of 32 and the time is the sum of 10 batches.
All tests are conducted on a single NVIDIA RTX 3090 GPU.

The impact of foundation models size. From the perspective of training speed
(Figures 1a), PEFT methods reduce the gradient computation time, so the
forward pass time gradually surpasses the backward pass time. Nonetheless, the
forward calculation is still unchanged and needs to use all model parameters
to propagate the state forward and backpropagate the loss through the entire
model, becoming slower as the model size increases. From the memory
perspective (Figure 1b), although PEFT techniques reduce the memory
consumption of optimizer states and gradients, the model weights and
intermediate activations still occupy a significant amount of memory during
training. Compressing the foundation model to a smaller size can better
alleviate it. This highlights the importance of reducing the parameter
redundancy of the foundation model for training efficiency.

(a) Training Time (b) Memory Usage

Figure 1: Impact of the foundation model size on training efficiency. The
experiments are conducted on OPT models (FP16). As the size of foundation
models increases, the time for forward and backward pass during training and
the required memory significantly increase.

The impact of PEFT modules. We explore the impact of intra-module rank and the
number of PEFT modules on training efficiency. From the perspective of
training speed, Figure 2a presents experiments where we keep same modules and
only increase the rank. Figure 2b shows experiments where we keep the same
trainable parameter, adding structured PEFT modules to different weights. It
can be observed that when increasing the number of PEFT modules compared to
varying the rank, both forward and backward times significantly increased.
This indicates that, during training, the impact on speed of adding more
structured PEFT modules is significantly larger than that of increasing in
rank of a single structured module. From a memory perspective, the trainable
parameters affect the memory consumption of optimizer states and gradients
during training. As the size of the foundation model increases, the redundancy
introduced by empirically adding trainable parameter modules impacts training
efficiency.

(a) Module rank (b) Number of modules

Figure 2: Impact of intra-module rank and the number of PEFT modules on
training speed. The experiments are conducted on RoBERTa-Large (FP32). Q, K,
V, O denote the Query, Key, Value and Output matrices in the foundation
model’s attention sub-layer. (a) Keeping the same number of modules and
increasing the rank results in a relatively small change in pass time. (b)
Increasing the number of modules while keeping the same trainable parameters
leads to a significant change in pass time.

##  4 Methodology

###  4.1 Overview of Light-PEFT

Our goal is to eliminate parameters redundancies in the early stage, thereby
reducing the computational costs of fine-tuning. Thus, we propose the Light-
PEFT framework as shown in Figure 3, which consists of two methods: Masked
Early Pruning of Foundation Model to reduce the redundancy of the foundation
model and Multi-Granularity Early Pruning of PEFT to reduce the redundancy of
the trainable parameters. First, both methods in our framework simultaneously
estimate redundancies during the early stage of training, where the total
training steps are denoted as $t$, and the estimation for early pruning steps
denoted as $t^{\prime}$, $t^{\prime}\ll t$. After estimation, we prune
redundancies in both, thus obtain a non-redundant foundation model and PEFT
modules for more efficient fine-tuning. Besides the PEFT parameters, we only
need to additionally save mask vectors, which are much smaller than PEFT
modules, to record the pruning index of the foundation model. During
inference, our method allows the masks and PEFT modules to be easily changed,
maintaining the plug-and-play feature.

Figure 3: Illustration of Light-PEFT. The left side shows the two methods in
Light-PEFT. On the right side is an illustration of the paradigm. Firstly,
both methods simultaneously estimate redundancies during the early-stage of
training. After estimation, Light-PEFT prunes redundancies in both, obtaining
a non-redundant foundation model and PEFT modules for more efficient fine-
tuning.

###  4.2 Masked Early Pruning of Foundation Model

A typical Transformer model (Vaswani et al., 2017) consists of $L$ layers,
each with a multi-head attention (MHA) sub-layer and a feed-forward network
(FFN) sub-layer. A MHA sub-layer contains $N_{H}$ attention heads and weight
matrices $W_{Q}^{(i)},W_{K}^{(i)},W_{V}^{(i)}\in\mathbb{R}^{d\times d_{H}}$,
$W_{O}\in\mathbb{R}^{d\times d}$ are used for query, key, value and output,
where $d$ is the hidden size and $d_{H}=d/N_{H}$ is the hidden size of a head.
In parameter-efficient fine-tuning, the weights of the foundation model are
frozen, and we add the PEFT module’s $\Delta W$ to these matrices. Taking the
LoRA module as an example, for an input $X$ the output of the MHA is
calculated as follows:

$\displaystyle head^{(i)}$ $\displaystyle=(W_{Q}^{(i)}+{\Delta W}_{Q}^{(i)},$
(3) $\displaystyle W_{K}^{(i)}+{\Delta W}_{K}^{(i)},W_{V}^{(i)}+{\Delta
W}_{V}^{(i)},X)$ $\displaystyle{\rm MHA}(X)$
$\displaystyle=Concat(head^{(1)},...,$ (4) $\displaystyle
head^{(N_{H})})(W_{O}+{\Delta W}_{O})$

To identify redundancy in attention heads, we introduce a trainable scalar
mask $m_{A}$ in each layer’s MHA sub-layer. Now the MHA becomes:

$\displaystyle head^{(i)}$ $\displaystyle=m_{A}^{(i)}\cdot(W_{Q}^{(i)}+{\Delta
W}_{Q}^{(i)},$ (5) $\displaystyle W_{K}^{(i)}+{\Delta
W}_{K}^{(i)},W_{V}^{(i)}+{\Delta W}_{V}^{(i)},X)$

For a FFN sub-layer, which contains activation function ${\rm Act}(\cdot)$ and
weight matrices $W_{fc1}$ and $W_{fc2}$ which denote up-projection and down-
projection. With PEFT modules, for an input $X$ the output of the FFN is
calculated as follows:

$\displaystyle{\rm FFN}(X)=$ $\displaystyle{\rm Act}(X(W_{fc1}+{\Delta
W_{fc1}}))$ (6) $\displaystyle\cdot(W_{fc2}+{\Delta W_{fc2}})$

We also introduce a trainable scalar mask $m_{F}$ in each layer’s FFN sub-
layer to eliminate redundancy in intermediate dimension. Now the FFN become:

$\displaystyle{\rm FFN}(X)=$ $\displaystyle{\rm Act}(X(W_{fc1}+{\Delta
W_{fc1})})\cdot{m_{F}}$ (7) $\displaystyle\cdot(W_{fc2}+{\Delta W_{fc2}})$

Inspired by Liu et al. (2017), we then use L1 regularization to learn masks
$m_{A}$ and $m_{F}$. During the mask learning, the PEFT module and the mask
are trained jointly using gradient descent, which allows the mask to better
present the impact of PEFT to the foundation model training on the target
task. The loss function is as follows:

$\mathcal{L}_{mask}=\mathcal{L}+\lambda_{A}\|m_{A}\|_{1}+\lambda_{F}\|m_{F}\|_{1}$
(8)

where $\mathcal{L}$ is the original loss in fine-tuning, $\lambda_{A}$ and
$\lambda_{F}$ are hyper-parameters to control the penalty of regularization
(see Appendix A.4 for details). The masks are initialized to 1 at the
beginning of training.

After estimating, we perform structured pruning on attention heads with
pruning ratio $\rho_{A}$ layer-wise and intermediate dimensions with
$\rho_{F}$ globally based on the magnitudes of $m_{A}$ and $m_{F}$.

###  4.3 Multi-Granularity Early Pruning of PEFT

In comparison to the fine-grained sparsity (i.e. rank allocation) that is the
focus of most previous works (Zhang et al., 2023; Valipour et al., 2023), our
preliminary observation also confirms the significance of coarse-grained
module pruning for training speed. Therefore, we propose multi-granularity
PEFT pruning to consider both aspects simultaneously. Furthermore, we perform
pruning PEFT in the early stage to maximize efficiency during training.

####  4.3.1 Modules Pruning

To achieve coarse-grained module pruning, we begin with the original design of
PEFT, where we believe that the importance of a module is primarily determined
by the change it brings to the original information. Specifically, for the
LoRA method, we add a trainable module $W_{down}W_{up}$ on the weight $W$.
Thus, given an input $X$, the importance ratio $I_{M}$ is defined as:

$I_{M}=\frac{\|{X\cdot W_{down}W_{up}}\|_{2}}{\|{X\cdot W}\|_{2}}$ (9)

where ${\|{\cdot}\|_{2}}$ represents the L2 norm, measuring the magnitude of
the vector output from the PEFT module. Because one of the weight matrices in
the PEFT module, such as $W_{up}$ in the LoRA method, is typically initialized
to zero. Therefore, during training, the ratio of the output magnitude of the
LoRA module to the weight $W$’s output magnitude indicates the importance of
the changes required by the module added at that position.

For the Adapter method, a trainable module is added after a sub-layer. Given
the output $h$ of the previous sub-layer, the importance ratio $I_{M}$ is
defined as:

$I_{M}=\frac{\|{f(hW_{down})W_{up}}\|_{2}}{\|{h}\|_{2}}$ (10)

where $I_{M}$ represents the change in information of the Adapter module on
the output information $h$ of the previous sub-layer.

In the implementation, to better estimate the importance of all added
positions for the LoRA method, we add LoRA modules on all weights of the
foundation model. This may result in higher costs compared to the original
LoRA in the short term, but our early estimation steps are significantly
smaller than the total training steps, allowing for a substantial reduction in
total costs. For the Adapter method, we follow the original approach by adding
them after both the MHA and FFN sub-layers. After estimation, we use $I_{M}$
to globally prune the entire PEFT modules with the pruning rate $\rho_{M}$.

####  4.3.2 Ranks Pruning

In addition to coarse-grained pruning, we further perform fine-grained pruning
on the rank of the modules. This allows us to reduce more trainable parameters
and enhance training efficiency. Our motivation is based on the fact that not
all modules require the same rank allocation. To eliminate redundant ranks, we
use the first-order Taylor expansion (Molchanov et al., 2017) to estimate the
importance $I_{W_{i,j}}$ of each parameter connected to the rank in the PEFT
module:

$I_{W_{i,j}}=\left|{\frac{\partial{\mathcal{L}}}{\partial{W_{i,j}}}{W_{i,j}}}\right|$
(11)

where $W_{i,j}$ represents the i-th row and j-th column of parameters in
$W_{down}$ or $W_{up}$ of the PEFT module. The importance of the rank $I_{R}$
is the sum of the importance $I_{W_{i,j}}$ of all parameters corresponding to
the rank in the column of $W_{down}$ and the row of $W_{up}$. After
estimation, we globally prune the unimportant ranks with pruning rate
${\rho}_{R}$.

##  5 Experiments

###  5.1 Experimental Setup

Datasets and evaluation. We conduct experiments on eight natural language
understanding (NLU) tasks from GLUE Wang et al. (2019b) and SuperGLUE Wang et
al. (2019a) and six question-answering (QA) tasks. Because our goal is to
enhance training efficiency, training on small datasets does not hold much
significance. As a result, we choose four larger datasets from GLUE including
MNLI Williams et al. (2018), QNLI Rajpurkar et al. (2016),
QQP222https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs, and
SST-2 Socher et al. (2013), and four larger datasets from SuperGLUE comprising
ReCord Zhang et al. (2018), WiC Pilehvar and Camacho-Collados (2019), BoolQ
Clark et al. (2019), and MultiRC Khashabi et al. (2018). For MNLI, we report
accuracy on the matched validation set. For QNLI, QQP, SST-2, WiC and BoolQ we
report accuracy. For ReCord we report F1 and for MultiRC we report F1 over all
answer-options (F1a). The QA tasks include OpenBookQA Mihaylov et al. (2018),
PIQA Bisk et al. (2020), ARC-Easy and ARC-Challenge Clark et al. (2018), SciQ
Welbl et al. (2017) and WebQuestions Berant et al. (2013). We report accuracy
on all QA tasks by lm-evaluation-harness Gao et al. (2023).

Baselines. We use RoBERTa-Large for NLU tasks, OPT-1.3B and OPT-6.7B for QA
tasks as foundation models. We choose several baselines to verify the
effectiveness of our method. Full-FT is the conventional approach for fine-
tuning. Adapter Houlsby et al. (2019) and LoRA Hu et al. (2022) are original
structures we used in our framework. LayerDrop Fan et al. (2020) is a strong
baseline method that enhances training efficiency by dynamically dropout
layers during training. We re-implement it combining with LoRA method. LST
Sung et al. (2022) improves model training efficiency by avoiding
backpropagation in the foundation model. Offsite-Tuning Xiao et al. (2023)
uses a emulator derived from the foundation model for efficient training, and
replaces the emulator’s layers back into the foundation model for inference.
LLM-Pruner Ma et al. (2023) prunes model on small amount of task-agnostic
corpora and restores performance using LoRA, thereby improving training
efficiency. We re-implement their original task-agnostic pruning and add a
task-specific pruning implementation using 1k random samples from task data.

Implementation. For the GLUE benchmark, we control the estimation steps for
early pruning to be around 5% of the total training steps. For the more
challenging SuperGLUE benchmark, we set the estimation steps to be within 10%.
For QA tasks, we uniformly use 10% of the training steps. Please refer to the
Appendix A.1 for detailed pruning settings, as well as other training details.

###  5.2 Experimental Results

Method |  #Trainable Params |  #Foundation Model Params | GLUE |  Training Speed up
---|---|---|---|---
MNLI | QNLI | QQP | SST-2 | Avg.
Full-FT | 355.0M | 100% | 90.4 | 94.7 | 92.2 | 96.4 | 93.4 | 0.7$\times$
Adapter | 0.8M | 100% | 90.8 | 94.7 | 91.5 | 96.3 | 93.3 | 1$\times$
LoRA | 0.8M | 100% | 90.6 | 94.9 | 91.6 | 96.2 | 93.3 | 1$\times$
LayerDrop | 0.5M | 67% | 87.4 | 91.7 | 88.3 | 94.7 | 90.5 | 1.4$\times$
LST | 8.6M | 100% | 86.7 | 90.2 | 89.7 | 95.1 | 90.4 | 1.4$\times$
Ours (Adapter) | 0.3M | 72% | 88.3 | 93.2 | 89.8 | 95.6 | 91.7 | 1.4$\times$
Ours (LoRA) | 0.3M | 72% | 89.4 | 93.6 | 89.7 | 95.9 | 92.2 | 1.4$\times$
Ours (Adapter) | 0.3M | 67% | 87.6 | 93.1 | 89.1 | 95.4 | 91.3 | 1.6$\times$
Ours (LoRA) | 0.3M | 67% | 89.0 | 93.5 | 89.2 | 95.8 | 91.9 | 1.6$\times$

Table 1: Results of GLUE benchmark. The training speed is measured on a single
NVIDIA TITAN RTX 24GB GPU with batch size=32 and sequence length=128. Note
that the speed computed here also includes the time required for estimation
before pruning.

####  5.2.1 Experiments on NLU Tasks

We first evaluate our method on the GLUE benchmark. As shown in Table 1, we
achieve comparable performance with the original method while using 72% of the
foundation model parameters (pruning 5/16 of the heads and 1/3 of the FFN
intermediate dimensions) and 0.3M trainable parameters by pruning PEFT modules
and ranks. This results in a 1.4$\times$ training speedup and improvements in
memory usage due to pruning. Furthermore, our method outperforms the baseline
methods with the same speed, having fewer trainable parameters. When
increasing the pruning rate and retaining 67% of the parameters in the
foundation model, Light-PEFT achieves a 1.6$\times$ training speedup while
still ensuring slightly better performance than the baselines. On the more
challenging SuperGLUE benchmark, as shown in Table 2, we prune 4/16 of the
heads and 30% of the FFN intermediate dimensions, retaining 76% of the
parameters in the foundation model and 0.3M trainable parameters. This
achieves performance comparable to the original PEFT method , demonstrating
the effectiveness of our method Masked Early Pruning of Foundation Model.

Method | #T.P. |  #F.P. | SuperGLUE
---|---|---|---
ReCord | WiC | BoolQ | MultiRC | Avg.
Adapter | 0.8M | 100% | 89.5 | 71.0 | 84.3 | 82.4 | 81.8
Ours | 0.3M | 76% | 86.0 | 70.1 | 81.2 | 76.0 | 78.3
LoRA | 0.8M | 100% | 88.3 | 72.7 | 84.1 | 82.7 | 82.0
Ours | 0.3M | 76% | 86.6 | 70.2 | 83.3 | 78.0 | 79.5

Table 2: Results of SuperGLUE Benchmark. #T.P. denotes the trainable
parameters. #F.P. denotes the proportion of parameters retained after pruning
the foundation model.

####  5.2.2 Experiments on QA Tasks

For the QA tasks (Table 3), we first conduct experiments on OPT-1.3B. We prune
parameters (12/32 heads and 2/5 intermediate dimensions), retaining 64% of the
foundation model parameters and 1.5M trainable parameters, achieving
comparable performance to the original method. When the trainable parameter in
the original LoRA method is set to 1.57M (r=8), our method outperforms the
original LoRA under fewer foundation model parameters, which demonstrates the
effectiveness of our method Multi-Granularity Early Pruning of PEFT.

Compared to Offsite-Tuning, our method achieves better performance without the
high training costs of the distillation. Compared to LLM-Pruner, our method
outperforms both task-agnostic and specific implementations, and our pruning
process does not require the large model’s gradients, leading to significantly
reduced computational costs. Even when pruning it to 54%, we maintain better
performance than the baselines.

On the larger OPT-6.7B model, pruning more foundation model parameters than
OPT-1.3B and using 5.2M trainable parameters, we achieve performance
comparable to the original method. When reducing trainable parameters to 2M,
our method still demonstrates good performance. These experimental results
demonstrate that in QA tasks, we can use the Light-PEFT framework to remove
more redundant parameters from the foundation model and trainable modules,
improving training efficiency while ensuring performance.

Method |  #Trainable Params |  #Foundation Model Params | QA Tasks
---|---|---|---
OpenBookQA | PIQA | ARC-E | ARC-C | SciQ | WebQs | Avg.
OPT-1.3B
Full-FT | 1.3B | 100% | 31.4 | 75.2 | 61.3 | 27.7 | 92.5 | 31.2 | 53.2
Offsite-Tuning | - | 100% | 29.0 | 74.5 | 59.4 | 27.8 | 92.9 | 26.2 | 51.6
LoRA (r=64) | 12.6M | 100% | 33.6 | 74.7 | 59.5 | 29.5 | 92.0 | 29.8 | 53.2
LoRA (r=8) | 1.6M | 100% | 29.6 | 74.6 | 59.9 | 29.1 | 93.0 | 28.7 | 52.5
LLM-Pruner (ag.) | 10.6M | 70% | 29.0 | 72.4 | 54.0 | 24.7 | 89.2 | 20.7 | 48.3
LLM-Pruner (sp.) | 10.6M | 70% | 30.4 | 72.9 | 55.9 | 27.6 | 88.7 | 26.5 | 50.3
Ours (LoRA) | 1.5M | 64% | 33.2 | 74.1 | 59.0 | 28.4 | 92.7 | 28.6 | 52.7
Ours (LoRA) | 1.9M | 54% | 33.2 | 72.6 | 57.6 | 27.5 | 91.8 | 28.2 | 51.8
OPT-6.7B
Offsite-Tuning | - | 100% | 33.8 | 77.7 | 66.8 | 33.9 | 91.9 | 23.9 | 54.7
LoRA (r=64) | 33.6M | 100% | 39.2 | 78.5 | 67.5 | 36.7 | 94.0 | 38.5 | 59.1
Ours (LoRA) | 5.2M | 52% | 39.4 | 74.9 | 63.4 | 32.7 | 92.9 | 35.8 | 56.5
Ours (LoRA) | 2.0M | 52% | 37.2 | 76.0 | 64.4 | 31.7 | 93.3 | 34.7 | 56.2

Table 3: Results of QA Tasks. Full-FT and Offsite-Tuning results are from Xiao
et al. (2023). For the original LoRA method, we add modules (rank=64) to the
Query and Value matrices to achieve results similar to Full-FT. For the LLM-
Pruner method, We re-implement their original task-agnostic pruning (ag.) and
add a task-specific pruning (sp.) implementation using 1k random samples from
task data.

PEFT Pruning Strategy | LoRA | Adapter
---|---|---
QNLI | SST-2 | QNLI | SST-2
all | 93.5 | 95.8 | 93.1 | 95.4
w/o module p. | 93.8 | 96.1 | 92.9 | 95.5
w/o rank p. | 93.8 | 95.8 | 93.2 | 95.2
w/o all | 93.6 | 95.6 | 93.0 | 95.1

Table 4: Ablation Study of Multi-Granularity Early Pruning of PEFT. We
investigate the results of not using coarse-grained module pruning (w/o module
p.), not using fine-grained rank pruning (w/o rank p.), and not using any PEFT
pruning (w/o all).

###  5.3 Analysis

####  5.3.1 Ablation Study

In the Section 5.2, we have demonstrated the performance of foundation model
pruning (more experiments in Appendix A.2). Here, we conduct ablation study to
examine two PEFT pruning strategies, module pruning and rank pruning (Table
4). Compared to not using any PEFT pruning, using module pruning or rank
pruning generally improves generalization and thus enhances performance in
most cases, indicating the effectiveness of the two proposed pruning
strategies. Moreover, by combining the two pruning strategies, the model
maintains a comparable level of performance despite having more pruned
trainable parameters.

(a) Training Time (b) Memory Usage

Figure 4: Training Efficiency. The experiments are conducted on RoBERTa-Large,
and we set batch size=32 and sequence length=128. Our method retains 67% of
foundation model parameters and 0.3M trainable parameters.

####  5.3.2 Training and Inference Efficiency

We validate the training and inference efficiency of our method on NVIDIA RTX
3090. In terms of training efficiency (Figure 4), we conduct experiments on
RoBERTa-Large, retaining 67% of foundation model parameters and 0.3M trainable
parameters that resulted in 32% reduction in model weight memory, 40%
reduction in activations memory, and 39% reduction in peak memory. Calculating
the total time for 10 batches, we achieve 2.2$\times$ speedup in forward and
backward pass time compared to the original LoRA.

In terms of inference efficiency (Table 5), we conduct experiments on
OPT-6.7B, representing widely used generative LLMs. Compared to the common
practice of adding LoRA modules onto all matrices in the fine-tuning of LLMs
(Vanilla), our proposed foundation model pruning and PEFT module pruning can
effectively increase inference speed by up to 1.6$\times$. Additionally,
foundation model pruning can effectively reduce the model loading memory usage
by up to 48%.

Method |  #Foundation Model Params |  $N_{M}$ ($\rho_{M}$) |  Inference Speed Up |  Load Memory
---|---|---|---|---
Vanilla | 100% | 192 (-0%) | 1$\times$ | 12.5G
Light-PEFT | 76% | 192 (-0%) | 1.1$\times$ | 9.5G
52% | 192 (-0%) | 1.2$\times$ | 6.5G
52% | 96 (-50%) | 1.4$\times$ | 6.5G
52% | 48 (-75%) | 1.6$\times$ | 6.4G

Table 5: Inference Efficiency. The experiments are conducted on OPT-6.7B.
$\rho_{M}$ denotes the PEFT module pruning rate, where 0% indicates inserting
LoRA modules (r=8) onto all matrices of the foundation model. And $N_{M}$
denotes the remaining number of LoRA modules after PEFT module pruning. We set
batch size=96 and max length=100.

##  6 Conclusion

This paper introduces Light-PEFT, a novel framework designed to improve the
efficiency of the PEFT technique during fine-tuning. The framework comprises
two methods: Masked Early Pruning of Foundation Model and Multi-Granularity
Early Pruning of PEFT. The Light-PEFT framework estimates redundant parameters
in both the foundation model and PEFT modules during the early stage of
training and prunes them to achieve more efficient fine-tuning. We validate
our approach on GLUE, SuperGLUE, and QA tasks using various models. The
experiments demonstrate that Light-PEFT achieves training and inference
speedup, reduces memory usage, and maintains comparable performance.

## Limitations

Although Light-PEFT has achieved improved training and inference efficiency
along with good performance, our work primarily focuses on the single-task
fine-tuning scenario. A future direction worth exploring is the estimation and
early pruning of redundant parameters on the multi-task learning scenario,
enabling efficient fine-tuning across multiple tasks.

## Ethics Statement

The goal of our Light-PEFT framework is to enhance training efficiency and
reduce computational resource costs, which has positive impacts.

## Acknowledgements

The authors thank Yuanxin Liu from Peking University and Jiaxuan Zhao from
Institute of Information Engineering for the help and the anonymous reviewers
for their valuable feedback on our paper.

## References

  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on Freebase from question-answer pairs.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1533–1544, Seattle, Washington, USA. Association for Computational Linguistics.
  * Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020.  PIQA: reasoning about physical commonsense in natural language.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 7432–7439. AAAI Press.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Chen et al. (2021) Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. 2021.  EarlyBERT: Efficient BERT training via early-bird lottery tickets.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2195–2207, Online. Association for Computational Linguistics.
  * Chen et al. (2023) Xuxi Chen, Tianlong Chen, Weizhu Chen, Ahmed Hassan Awadallah, Zhangyang Wang, and Yu Cheng. 2023.  DSEE: Dually sparsity-embedded efficient tuning of pre-trained language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8208–8222, Toronto, Canada. Association for Computational Linguistics.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the AI2 reasoning challenge.  _CoRR_ , abs/1803.05457.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  QLoRA: Efficient finetuning of quantized LLMs.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ding et al. (2023) Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. 2023.  Sparse low-rank adaptation of pre-trained language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 4133–4145, Singapore. Association for Computational Linguistics.
  * Fan et al. (2020) Angela Fan, Edouard Grave, and Armand Joulin. 2020.  Reducing transformer depth on demand with structured dropout.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Frankle and Carbin (2019) Jonathan Frankle and Michael Carbin. 2019.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  In _International Conference on Learning Representations_.
  * Gao et al. (2023) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023.  A framework for few-shot language model evaluation.
  * Hedegaard et al. (2022) Lukas Hedegaard, Aman Alok, Juby Jose, and Alexandros Iosifidis. 2022.  Structured pruning adapters.  _CoRR_ , abs/2211.10155.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799. PMLR.
  * Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  Lora: Low-rank adaptation of large language models.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Hu et al. (2023) Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. 2023.  LLM-adapters: An adapter family for parameter-efficient fine-tuning of large language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5254–5276, Singapore. Association for Computational Linguistics.
  * Khashabi et al. (2018) Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018.  Looking beyond the surface: A challenge set for reading comprehension over multiple sentences.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 252–262, New Orleans, Louisiana. Association for Computational Linguistics.
  * Kim et al. (2023) Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. 2023.  Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Kurtic et al. (2022) Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. 2022.  The optimal BERT surgeon: Scalable and accurate second-order pruning for large language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 4163–4181, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * LeCun et al. (1989) Yann LeCun, John S. Denker, and Sara A. Solla. 1989.  Optimal brain damage.  In _Advances in Neural Information Processing Systems 2, [NIPS Conference, Denver, Colorado, USA, November 27-30, 1989]_ , pages 598–605. Morgan Kaufmann.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597, Online. Association for Computational Linguistics.
  * Li et al. (2022) Yuchao Li, Fuli Luo, Chuanqi Tan, Mengdi Wang, Songfang Huang, Shen Li, and Junjie Bai. 2022.  Parameter-Efficient Sparsity for Large Language Models Fine-Tuning.  In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22_ , pages 4223–4229. International Joint Conferences on Artificial Intelligence Organization.
  * Liao et al. (2023) Baohao Liao, Shaomu Tan, and Christof Monz. 2023.  Make pre-trained model reversible: From parameter to memory efficient fine-tuning.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Liu et al. (2022a) Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022a.  P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 61–68, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Liu et al. (2021) Yuanxin Liu, Zheng Lin, and Fengcheng Yuan. 2021.  ROSITA: refined BERT compression with integrated techniques.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 8715–8722. AAAI Press.
  * Liu et al. (2022b) Yuanxin Liu, Fandong Meng, Zheng Lin, Peng Fu, Yanan Cao, Weiping Wang, and Jie Zhou. 2022b.  Learning to win lottery tickets in BERT transfer via task-agnostic mask training.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5840–5857, Seattle, United States. Association for Computational Linguistics.
  * Liu et al. (2017) Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017.  Learning efficient convolutional networks through network slimming.  In _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_ , pages 2755–2763. IEEE Computer Society.
  * Ma et al. (2023) Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023.  LLM-pruner: On the structural pruning of large language models.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics.
  * Molchanov et al. (2017) Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2017.  Pruning convolutional neural networks for resource efficient inference.  In _International Conference on Learning Representations_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774.
  * Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019.  WiC: the word-in-context dataset for evaluating context-sensitive meaning representations.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1267–1273, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Sung et al. (2022) Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022.  LST: Ladder side-tuning for parameter and memory efficient transfer learning.  In _Advances in Neural Information Processing Systems_.
  * Tao et al. (2023) Chaofan Tao, Lu Hou, Haoli Bai, Jiansheng Wei, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2023.  Structured pruning for efficient generative pre-trained language models.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 10880–10895, Toronto, Canada. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.  _ArXiv_ , abs/2302.13971.
  * Valipour et al. (2023) Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. 2023.  DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 3274–3287, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Wang et al. (2019a) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 3261–3275.
  * Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Welbl et al. (2017) Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017.  Crowdsourcing multiple choice science questions.  In _Proceedings of the 3rd Workshop on Noisy User-generated Text_ , pages 94–106, Copenhagen, Denmark. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xia et al. (2024) Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2024.  Sheared LLaMA: Accelerating language model pre-training via structured pruning.  In _The Twelfth International Conference on Learning Representations_.
  * Xia et al. (2022) Mengzhou Xia, Zexuan Zhong, and Danqi Chen. 2022.  Structured pruning learns compact and accurate models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1513–1528, Dublin, Ireland. Association for Computational Linguistics.
  * Xiao et al. (2023) Guangxuan Xiao, Ji Lin, and Song Han. 2023.  Offsite-tuning: Transfer learning without full model.  _CoRR_ , abs/2302.04870.
  * You et al. (2020) Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Yingyan Lin, Zhangyang Wang, and Richard G. Baraniuk. 2020.  Drawing early-bird tickets: Toward more efficient training of deep networks.  In _International Conference on Learning Representations_.
  * Zhang et al. (2023) Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023.  Adaptive budget allocation for parameter-efficient fine-tuning.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
  * Zhang et al. (2018) Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018.  Record: Bridging the gap between human and machine commonsense reading comprehension.  _CoRR_ , abs/1810.12885.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  Opt: Open pre-trained transformer language models.  _ArXiv_ , abs/2205.01068.
  * Zhao et al. (2024) Bowen Zhao, Hannaneh Hajishirzi, and Qingqing Cao. 2024.  APT: adaptive pruning and tuning pretrained language models for efficient training and inference.  _CoRR_ , abs/2401.12200.
  * Zhao et al. (2023) Weilin Zhao, Yuxiang Huang, Xu Han, Zhiyuan Liu, Zhengyan Zhang, and Maosong Sun. 2023.  CPET: effective parameter-efficient tuning for compressed large language models.  _CoRR_ , abs/2307.07705.
  * Zhou et al. (2023) Han Zhou, Xingchen Wan, Ivan Vulic, and Anna Korhonen. 2023.  Autopeft: Automatic configuration search for parameter-efficient fine-tuning.  _CoRR_ , abs/2301.12132.

##  Appendix A Appendix

###  A.1 Details of Experimental Setup

Hardware. We use NVIDIA TITAN RTX and NVIDIA RTX 3090 for NLU experiments and
experiments using OPT-1.3B in QA Tasks. Additionally, we use NVIDIA A800 for
experiments using OPT-6.7B in QA Tasks.

Implementation. The implementation of Light-PEFT is based on Transformers
(Wolf et al., 2020), LLM-Adapters (Hu et al., 2023), and EarlyBERT (Chen et
al., 2021). The data processing for SuperGLUE and QA tasks follows Liu et al.
(2022a) and Xiao et al. (2023), respectively.

Hyper-parameters. We use AdamW as the optimizer for training. Other detailed
settings for NLU tasks are provided in Table 7, while the settings for QA
tasks can be found in Table 8 and Table 9.

###  A.2 The impact of the pruning rate on the foundation model.

We analyze the impact of different foundation model pruning rates on
performance on the WiC dataset (Figure 5). It is observed that within a
certain range (above 62.5%), pruning results in a relatively minor decrease in
performance. However, once this threshold is exceeded, a significant
performance decline occurs, demonstrating that pruning within this range
removes redundant parameters.

Figure 5: The impact of the pruning rate on the foundation model.

###  A.3 The impact of the estimation steps of early pruning

We analyze the impact of the early pruning estimation steps on performance
using the BoolQ dataset (Figure 6). It is observed that once the estimation
steps exceed 6.8% of the total training steps, further estimation does not
lead to performance improvement. This demonstrates that our method can
effectively identify redundant parameters in both the foundation model and
PEFT modules during the early stage of training.

Figure 6: The impact of the estimation steps of early pruning.

###  A.4 The settings of mask learning penalty

In practice, we keep $\lambda_{A}$ and $\lambda_{F}$ consistent and assess the
impact of these hyper-parameters in pilot experiments (Table 6). Based on this
result, we uniformly set $\lambda_{A}$ and $\lambda_{F}$ to $1\times 10^{-4}$
and achieve good task performance in our main experiments.

$\lambda_{A}$, $\lambda_{F}$ | SST-2 | QNLI | Avg.
---|---|---|---
$1*10^{-2}$ | 95.8 | 91.9 | 93.85
$1*10^{-3}$ | 95.9 | 93.5 | 94.70
$1*10^{-4}$ | 95.9 | 93.6 | 94.75
$1*10^{-5}$ | 95.6 | 91.9 | 93.75
Table 6: The impact of $\lambda_{A}$ and $\lambda_{F}$ on the performance of
tasks.

Method | Dataset | MNLI | QNLI | QQP | SST-2 | ReCord | WiC | BoolQ | MultiRC
---|---|---|---|---|---|---|---|---|---
LoRA | Estimation Steps | 1000 | 1000 | 1000 | 800 | 2000 | 680 | 400 | 600
Rank | 8
$\rho_{M}$ | 75%
$\rho_{R}$ | 50%
Estimation lr | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4
Fine-Tuning lr | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4
Batch Size | 32 | 32 | 32 | 32 | 32 | 16 | 32 | 16
Sequence Length | 128 | 128 | 128 | 128 | 256 | 128 | 128 | 384
# Epochs | 5 | 5 | 5 | 10 | 5 | 50 | 20 | 20
Adapter | Estimation Steps | 1000 | 1000 | 1000 | 800 | 2000 | 680 | 400 | 1000
Rank | 8
$\rho_{M}$ | 25%
$\rho_{R}$ | 50%
Estimation lr | 6e-4 | 8e-4 | 3e-4 | 6e-4 | 6e-4 | 3e-4 | 6e-4 | 7e-4
Fine-Tuning lr | 4e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 1e-4 | 6e-4 | 5e-4
Batch Size | 32 | 32 | 32 | 32 | 32 | 16 | 32 | 16
| Sequence Length | 128 | 128 | 128 | 128 | 256 | 128 | 128 | 384
| # Epochs | 5 | 5 | 5 | 10 | 5 | 50 | 20 | 20

Table 7: Hyperparameters for NLU Tasks.

Method | Dataset | OpenBookQA | PIQA | ARC-E | ARC-C | SciQ | WebQs
---|---|---|---|---|---|---|---
LoRA | Estimation Steps | 1 Epoch | 1 Epoch | 1 Epoch | 1 Epoch | 1 Epoch | 1 Epoch
Rank | 8
$\rho_{M}$ | 50%/50%
$\rho_{R}$ | 50%/25%
Estimation lr | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4
Fine-Tuning lr | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4
Batch Size | 64 | 64 | 64 | 64 | 64 | 64
Sequence Length | 128 | 128 | 128 | 128 | 128 | 128
# Epochs | 10 | 10 | 10 | 10 | 10 | 10

Table 8: Hyperparameters for QA Tasks on OPT-1.3B.

Method | Dataset | OpenBookQA | PIQA | ARC-E | ARC-C | SciQ | WebQs
---|---|---|---|---|---|---|---
LoRA | Estimation Steps | 1 Epoch | 1 Epoch | 1 Epoch | 1 Epoch | 1 Epoch | 1 Epoch
Rank | 8
$\rho_{M}$ | 50%/75%
$\rho_{R}$ | 25%/50%
Estimation lr | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4
Fine-Tuning lr | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4 | 3e-4
Batch Size | 32 | 32 | 32 | 32 | 32 | 32
Sequence Length | 128 | 128 | 128 | 128 | 128 | 128
# Epochs | 10 | 10 | 10 | 10 | 10 | 10

Table 9: Hyperparameters for QA Tasks on OPT-6.7B.


11institutetext: Leibniz-Institut für Astrophysik Potsdam (AIP), An der
Sternwarte 16, 14482 Potsdam, Germany 11email<EMAIL_ADDRESS>22institutetext:
European Southern Observatory, Karl-Schwarzschild-Str. 2, 85748 Garching,
Germany 33institutetext: Gemini Observatory / NSF’s NOIRLab, Casilla 603, La
Serena, Chile 44institutetext: Instituto de Astrofísica de Canarias, C. Vía
Láctea s/n, 38205 La Laguna, Santa Cruz de Tenerife, Spain 55institutetext:
Dpto. Astrofísica, Universidad de La Laguna, Av. Astrofísico Francisco
Sánchez, 38206 La Laguna, Santa Cruz de Tenerife, Spain 66institutetext: MIT
Kavli Institute and Department of Physics, 77 Massachusetts Avenue, Cambridge,
MA 02139, USA

# Detection of extragalactic magnetic massive stars

S. Hubrig 11 M. Schöller 22 S. P. Järvinen 11 A. Cikota 33 M. Abdul-Masih 4455
A. Escorza 4455 R. Jayaraman 66

(Received MM DD, 2024; accepted MM DD, 2024)

###### Abstract

Context.  Studies of the magnetic characteristics of massive stars have
recently received significant attention because they are progenitors of highly
magnetised compact objects. Stars initially more massive than about
$8\,M_{\odot}$ leave behind neutron stars and black holes by the end of their
evolution. The merging of binary compact remnant systems produces
astrophysical transients detectable by gravitational wave observatories.
Studies of magnetic fields in massive stars with low metallicities are of
particular interest because they provide important information on the role of
magnetic fields in the star formation of the early Universe.

Aims.  While several detections of massive Galactic magnetic stars have been
reported in the last few decades, the impact of a low-metallicity environment
on the occurrence and strength of stellar magnetic fields has not yet been
explored. Because of the similarity between Of?p stars in the Magellanic
Clouds (MCs) and Galactic magnetic Of?p stars, which possess globally
organised magnetic fields, we searched for magnetic fields in Of?p stars in
the MCs. Additionally, we observed the massive contact binary Cl$*$ NGC 346
SSN7 in the Small Magellanic Cloud to test the theoretical scenario that the
origin of magnetic fields involves a merger event or a common envelope
evolution.

Methods.  We obtained and analysed measurements of the magnetic field in four
massive Of?p stars in the MCs and the binary Cl$*$ NGC 346 SSN7 using the
ESO/VLT FORS2 spectrograph in spectropolarimetric mode.

Results.  We detected kilogauss-scale magnetic fields in two Of?p-type stars
and in the contact binary Cl$*$ NGC 346 SSN7. These results suggest that the
impact of low metallicity on the occurrence and strength of magnetic fields in
massive stars is low. However, because the explored stellar sample is very
small, additional observations of massive stars in the MCs are necessary.

###### Key Words.:

stars: magnetic field – stars: binaries: spectroscopic – stars: massive –
stars: oscillations – stars: variables: general – stars: individual: OGLE SMC-
SC6 237339, BI 57, AzV 220, UCAC4 115-008604, Cl$*$ NGC 346 SSN7

##  1 Introduction

The discovery of gravitational wave transients in high-redshift galaxies
caused by mergers of black holes and neutron stars in binary systems has
radically changed astronomical research, with focus now on advancing our
understanding of the life cycle of massive binary and multiple systems.
Importantly, magnetism is considered to be a key component in massive star
evolution, with a far-reaching impact on their ultimate fate. Highly
magnetised neutron stars with very strong magnetic fields, the magnetars, are
highly relevant in the field of gravitational wave astronomy (e.g. Regimbau &
de Freitas Pacheco 2006). A magnetic mechanism for the collimated explosion of
massive stars, relevant for long-duration gamma-ray bursts, X-ray flashes, and
asymmetric core collapse supernovae, was proposed in several theoretical
studies (e.g. Uzdensky & MacFadyen 2006). Such energetic physical processes
affect the structure of entire galaxies and chemically enrich the interstellar
medium.

Magnetic massive stars are unique sites to observe the combined effect of
stellar winds, rotation, and magnetism. In particular, studies of massive
stars with low metallicities are of primary interest because these stars are
especially valuable proxies of the early Universe and of contemporary star-
forming galaxies. With low metallicities, massive stars are hotter and more
luminous than their metallic counterparts, and their radii are smaller.
Furthermore, they have very different wind characteristics, which play a
dominant role in the evolution of massive stars (Ekström et al., 2019). Yet,
the impact of lower metallicities on the occurrence and strength of stellar
magnetic fields in massive stars has not been explored, neither theoretically
nor observationally.

The previously reported incidence rate of stable, predominantly dipolar
magnetic fields in Galactic O stars, with field strengths in the range from
hundreds of gauss to tens of kilogauss, was about 7% (e.g. Grunhut et al.
2017; Schöller et al. 2017). More recently, it was reported that massive stars
in binary and multiple systems are more likely to possess magnetic fields
(Hubrig et al., 2023). The origin of the magnetic fields in massive stars is
still under debate: it has been argued that magnetic fields could be fossil,
generated by dynamos, or generated by a strong binary interaction (i.e. in
stellar mergers, during a mass transfer, or during a common envelope
evolutionary phase). Recent theoretical research shows that protostellar
mergers might offer a tentative explanation for surface magnetic fields in
massive stars (Schneider et al., 2019, 2020).

A fraction of the known Galactic magnetic O-type stars belong to the rare
class of Of?p-type stars. Of?p stars are identified by the presence of C iii
$4650$ Å emission that is in comparable strength to the neighbouring N iii
$4634$ and $4642$ Å lines and recurrent spectral variability, notably in their
Balmer and He lines (Walborn, 1972). Five Galactic Of?p stars are currently
known: HD 108, NGC 1624-2, CPD$-$28∘ 2561, HD 148937, and HD 191612 (Walborn
et al., 2010), and all of them show evidence of the presence of magnetic
fields (Hubrig et al., 2008, 2011; Grunhut et al., 2017). Obviously, the
detection of magnetic fields in all five Galactic Of?p stars implies a tight
relation between the spectral characteristics of the Of?p star group and the
presence of a magnetic field.

The origin of Of?p stars and why only five Galactic stars with this
classification have been identified so far is, however, not understood. A
useful hint came from the combination of ten years of spectroscopic and
interferometric data for the Of?p binary HD 148937, which has an orbital
period of 29 yr (Frost et al., 2021). It was concluded that the magnetic
primary, although more massive, appears younger, suggesting that a merger or
mass transfer took place in this system.

Importantly, massive stars with the Of?p classification have also been
identified in the Small and Large Magellanic Clouds (SMC and LMC): three stars
each in the SMC (SMC 159-2, OGLE SMC-SC6 237339, and AzV 220) and the LMC (BI
57, LMC 64-2, and UCAC4 115-008604) (e.g. Walborn et al. 2015; Neugent et al.
2018). The SMC and LMC are known to have significantly lower metallicities
compared to our Galaxy, $Z_{\rm SMC}=0.2\,Z_{\odot}$ and $Z_{\rm
LMC}=0.5\,Z_{\odot}$, respectively, and can therefore be used as close proxies
for the early Universe. Evidently, due to the similarity with the well-studied
Galactic magnetic Of?p stars that possess globally organised magnetic fields,
the Of?p stars in the Magellanic Clouds (MCs) can be regarded as the best
candidate extragalactic magnetic O-type stars. Therefore, to investigate the
impact of low metallicities on the presence and strength of stellar magnetic
fields, we applied for observing time with the ESO/VLT FOcal Reducer low
dispersion Spectrograph (FORS2; Appenzeller et al. 1998) in
spectropolarimetric mode. Further, to test theoretical scenarios that suggest
that the origin of magnetic fields in massive stars involves a strong binary
interaction (e.g. Schneider et al. 2016; Pelisoli et al. 2022), we included in
our target list the ON3 If$+$O5.5 V((f)) contact system Cl$*$ NGC 346 SSN 7
(SSN 7 henceforth), located within the core of the most massive star-forming
region in the SMC, NGC 346 (Dufton et al., 2019). NGC 346 is usually
considered a counterpart of 30 Doradus in the LMC, and observations of this
cluster are frequently used to study the violent feedback effects of massive
stars on the formation and evolution of protostars. SSN 7 was recently
characterised as a short-period ($\sim$3.07 d) contact binary by Rickard &
Pauli (2023).

In Sect. 2 we describe the obtained observations and their reduction, and we
present the measurement procedure in Sect. 3. In Sect. 4 we discuss the
results of the magnetic field measurements.

##  2 Observations and data reduction

Table 1:  Logbook of our observations.

Object | $m_{V}$ | Spectral | Date | MJD | DIT | S/N | Wavelength
---|---|---|---|---|---|---|---
|  | Type |  |  | [s] |  | [Å]
OGLE SMC-SC6 237339 | 14.0 | O6.5?p | 2023-10-03 | 60220.0931 | 12500 | 970 | 4600
BI 57 | 14.0 | O8?p | 2023-10-03 | 60220.2334 | 9600 | 680 | 4600
AzV 220 | 14.5 | O6.5?p | 2023-10-04 | 60221.0657 | 14400 | 910 | 4600
UCAC4 115-008604 | 14.7 | O6.5?p | 2023-10-04 | 60221.2405 | 12900 | 720 | 4220
Cl $*$NGC 346 SSN 7 | 12.6 | ON3 If+O5.5 V((f)) | 2023-10-04 | 60221.3532 | 4200 | 1340 | 4730
$\xi^{1}$ CMa | 4.3 | B0.7 IV | 2023-10-03 | 60220.3053 | 32 | 3700 | 4530
HD 45166 | 9.9 | WRpec+B7 V | 2023-10-04 | 60221.3993 | 90 | 860 | 4600

Notes: Column 1 gives the object name, followed by the visual magnitude
($m_{V}$) and the spectral type in Columns 2 and 3. In Columns 4 and 5, we
list the date of the observation and the corresponding modified Julian date
(MJD). Finally, Column 6 shows the detector integration time (DIT) and Columns
7 and 8 the peak S/N values per Å at the wavelengths corresponding to the
maximum of the flux distribution for each target.

Two nights were allocated for our observing programme, which was carried out
in visitor mode on 2023 October 3 and 4 using the ESO/VLT FORS2 instrument
(Appenzeller et al., 1998), which is capable of imaging, polarimetry, long-
slit, and multi-object spectroscopy. FORS2 is equipped with polarisation
analysing optics, comprising super-achromatic halfwave and quarterwave phase
retarder plates, and a Wollaston prism with a beam divergence of 22′′ in
standard resolution mode. As we did not know beforehand exactly when our
visitor programme would be scheduled, for the observations of the mean
longitudinal magnetic field we selected the four Of?p stars with the longest
rotation periods, OGLE SMC-SC6 2373392 (=2dFS936), BI 57, AzV 220, and UCAC4
115-008604 (=LMCe 136-1), which have periods of 1370 d, 787 d, $>$16 yr, and
18.7 d, respectively (Bagnulo et al., 2017, 2020). Because the maximum of the
longitudinal magnetic field in Galactic massive stars usually occurs at the
same time as the maximum of the light curve (Munoz et al., 2020), the idea
behind such a selection was to avoid FORS2 spectropolarimetric observations at
unfavourable rotation phases when the light curve is at its minimum. For SSN
7, however, the rotation periods of both components are unknown. Rotational
phases are particularly important in longitudinal magnetic field measurements
since this field component is defined as the component of the magnetic field
averaged over the visible stellar disc. Hence, the ability to detect
longitudinal magnetic fields critically depends not only on the intrinsic
strength of the magnetic field but also on the geometrical view of the stellar
magnetic field structure at the time of the observation. To verify the
performance of the FORS2 Zeeman analyser, we observed in addition two Galactic
targets: the B0.7 IV $\xi^{1}$ CMa (=HD 46328), with a magnetic field of the
order of a few hundred gauss (Hubrig et al., 2006), and the O4 If$+$O5-6
system HD 45166, with a kilogauss-scale field (Shenar et al., 2023). We
observed each target once.

We used GRISM 600B and a slit width of 0.5′′ to achieve a spectral resolving
power of about 1650. This instrumental configuration allowed us to cover a
larger wavelength range, from 3250 to 6215 Å, which includes all Balmer lines
except H$\alpha$. The detector was a 2k$\times$4k MIT CCD with a pixel size of
15 $\mu$m $\times$ 15 $\mu$m. The MIT CCD has a nominal gain of 1.25 and a
readout noise of 2.70 electrons. To minimise the cross-talk effect, the
position angle of the retarder waveplate was changed from $+45^{\circ}$ to
$-45^{\circ}$ and vice versa every second exposure (Hubrig et al., 2004a, b).
We executed the sequence $-$45∘, $+$45∘, $+$45∘, $-$45∘ twice to increase the
final signal-to-noise ratio (S/N). The exposure times ranged from 2.7 h for
the brightest Of?p star in the sample, BI 57, with $m_{V}=14.0,$ to about 4.0
h for the Of?p star AzV 220 with $m_{V}=14.5$. The weather conditions were
mediocre on our first visitor night, with a seeing of up to 2.1′′ and strong
winds. This explains the rather low S/N of 680 achieved for the target BI 57.
The second night was better, with seeing around 1.2′′. Our observation of OGLE
SMC-SC6 237339 took place at the rotation phase $\phi=0.545,$ close to the
maximum of the Optical Gravitational Lensing Experiment (OGLE) light curve
($\phi=0.5$). The same is also true for the observations of UCAC4 115-008604
at $\phi=0.296$, which is close to the light curve maximum ($\phi=0.4$). Only
for the target BI 57 was our observation obtained at a phase between maximum
and minimum light, $\phi=0.250$. The rotation phase of our observation of AzV
220 is unknown due to missing information on its rotation period.

The obtained data were reduced in two ways using a set of completely
independent tools and routines. First, from the raw FORS2 data, the spectra
recorded in left- and right-hand polarised light (i.e. the ordinary and
extraordinary beams) were extracted using a pipeline written in the MIDAS
environment (henceforth the MIDAS pipeline). A description of the assessment
of the longitudinal magnetic field measurements using FORS1/2
spectropolarimetric observations was presented in our previous work (Hubrig et
al., 2004a, b). The MIDAS pipeline reduction by default includes sky
background subtraction. The wavelength calibration was carried out using He-
Ne-Ar arc lamp exposures. A unique wavelength calibration frame was used for
each night. The effects of improper flat field correction in the presence of
polarisation optics were minimised by taking advantage of the redundant number
of quarterwave positions (see Sect. 4.6.1 in the FORS2 User Manual).

The second reduction was performed using standard Image Reduction and Analysis
Facility (IRAF111https://iraf.noirlab.edu/; Tody 1993) procedures (Cikota et
al., 2017). After bias subtraction, the extraction of the spectra
corresponding to ordinary and extraordinary beams was carried out in an
unsupervised way using the PYRAF apextract.apall procedure, with a fixed
aperture size of 50 pixels. IRAF’s apall function was applied to subtract the
sky background using small apertures above and below the spectra. To avoid
spectrum-tracing problems, the input frames were properly trimmed to exclude
the low S/N at the edges of the spectral range. Since the exposure times were
relatively long, the polarimetric spectra are affected by cosmics. They were
removed using L.A.Cosmic (Laplacian Cosmic Ray Identification; van Dokkum
2001). We did not detect any differences between the spectra extracted using
the MIDAS pipeline and those from the IRAF procedures.

For each target, the information on the time of observation, the length of the
exposure, and the achieved S/N is presented in Table 1.

##  3 Measurement procedure

The mean longitudinal magnetic field was determined from the FORS2
observations as the slope of a weighted linear regression through the measured
Stokes $V$ values. The linear regression method uses a technique similar to
that developed earlier by Angel & Landstreet (1970) and is based on the
measurement of the difference between the circular polarisation observed in
the red and blue wings of spectral lines. The $V/I$ spectrum was calculated as

$\frac{V}{I}=\frac{1}{2}\left\\{\left(\frac{f^{\rm o}-f^{\rm e}}{f^{\rm
o}+f^{\rm e}}\right)_{-45^{\circ}}-\left(\frac{f^{\rm o}-f^{\rm e}}{f^{\rm
o}+f^{\rm e}}\right)_{+45^{\circ}}\right\\},$ (1)

where $+45^{\circ}$ and $-45^{\circ}$ indicate the position angle of the
retarder waveplate, and $f^{\rm o}$ and $f^{\rm e}$ are the ordinary and
extraordinary beams, respectively. To allow quality checks, diagnostic null
profiles ($N_{V}$) were computed as pairwise differences from all available
$V$ spectra, which caused the real polarisation signal to be cancelled out.
More details on our measurement procedures are presented in numerous previous
publications (e.g. Hubrig et al. 2014; Schöller et al. 2017; Hubrig et al.
2020).

For the determination of the stellar mean longitudinal magnetic field, we
usually considered two sets of spectral lines: (i) the entire spectrum that
includes all available absorption and emission lines and (ii) exclusively
hydrogen lines. For massive stars we always assume a Landé factor $g_{\rm
eff}=1.0$ for the hydrogen lines and $g_{\rm eff}=1.2$ for all other lines.
Notably, in our regression analysis we did not differentiate between
absorption and emission lines, since the relation between the Stokes $V$
signal and the slope of the spectral line wing (unlike with high-resolution
spectropolarimetric observations) holds for both type of lines, so the signals
of emission and absorption lines add up rather than cancel out.

Figure 1:  FORS2 observations of the reference magnetic massive star HD 45166.
In the left panel, the solid lines show the recorded Stokes $I$ and Stokes $V$
spectra and the diagnostic $N_{V}$ spectra, from top to bottom. In the right
panel, we show the regression detection of the mean longitudinal magnetic
field with $\left<B_{\rm z}\right>_{\rm all}=-1107\pm 80$ G.

Figure 2:  Same as Fig. 1 but for the reference magnetic massive star
$\xi^{1}$ CMa, with $\left<B_{\rm z}\right>_{\rm all}=-362\pm 44$ G.

Figure 3:  Same as Fig. 1 but for the Of?p star OGLE SMC-SC6 237339, with a
longitudinal magnetic field of $\left<B_{\rm z}\right>_{\rm all}=794\pm 209$ G
detected at a 3.8$\sigma$ significance level.

Figure 4:  Same as Fig. 1 but for the Of?p star UCAC4 115-008604 with a
longitudinal magnetic field of $\left<B_{\rm z}\right>_{\rm all}=-902\pm 271$
G detected at a 3.3$\sigma$ significance level.

Figure 5:  Same as Fig. 1 but for the Of?p star BI 57, with a mean
longitudinal magnetic field measured at a low significance: $\left<B_{\rm
z}\right>_{\rm all}=-837\pm 440$ G.

Figure 6:  Same as Fig. 1 but for the Of?p star AzV220, with a mean
longitudinal magnetic field measured at a low significance: $\left<B_{\rm
z}\right>_{\rm all}=-472\pm 202$ G.

Figure 7:  Same as Fig. 1 but for the contact binary Cl$*$ NGC 346 SSN 7, with
a mean longitudinal magnetic field of $\left<B_{\rm z}\right>_{\rm all}=908\pm
206$ G detected at a 4.4$\sigma$ significance level.

The $\left<B_{\rm z}\right>$ is defined by the slope of the weighted linear
regression line through the measured data points, where the weight of each
data point is given by the squared S/N of the Stokes $V$ spectrum. For the
magnetic field determination, we used both the spectra extracted using the
MIDAS pipeline and those from the IRAF procedures. As the measured magnetic
field strengths and measurement accuracies are similar for the two reductions,
we took the average of the two values as the final field strength. Linear
regression plots for each observed target are presented in Figs. 1–7. The
formal 1$\sigma$ error of $\left<B_{\rm z}\right>$ was obtained from the
standard relations for weighted linear regression. This error is inversely
proportional to the Stokes $V$ rms S/N. Furthermore, to derive robust
estimates of standard errors (Steffen et al., 2014; Schöller et al., 2017), we
carried out Monte Carlo bootstrapping tests. The measurement uncertainties
obtained before and after the Monte Carlo bootstrapping tests were found to be
in close agreement, indicating the absence of reduction flaws.

The results of our longitudinal magnetic field measurements, those for the
entire spectrum or only the hydrogen lines, are presented in Table 2. We also
list the corresponding known rotation periods, the orbital period for SSN 7 –
for which the rotation periods of both components are unknown – and the
rotation phases at the time of observation. The phases for the Of?p stars were
calculated according to the respective periods and ephemerides provided in
previous studies (Nazé et al., 2015; Bagnulo et al., 2020) using OGLE light
curves, with $\phi=0$ corresponding to the photometric minima.

Table 2:  Results of our magnetic field measurements.

Object | $\left<B_{\rm z}\right>_{\rm all}$ | $\sigma_{\rm all}$ | $\left<B_{\rm z}\right>_{\rm hyd}$ | $\sigma_{\rm hyd}$ | $\left<B_{\rm z}\right>_{\rm N}$ | $P_{\rm rot}$ | $P_{\rm orb}$ | Ref. | Phase
---|---|---|---|---|---|---|---|---|---
| [G] |  | [G] |  | [G] |  |  |  |
OGLE SMC-SC6 237339 | 794$\pm$ | 209 | 3.8 | 843$\pm$ | 253 | 3.3 |  $-$62$\pm$ | 224 | 1370 d |  | 1 | 0.545
BI 57 |  $-$837$\pm$ | 440 | 1.9 |  $-$190$\pm$ | 498 | 0.4 | 363$\pm$ | 452 | 787 d |  | 1 | 0.250
AzV220 |  $-$472$\pm$ | 202 | 2.3 |  $-$531$\pm$ | 262 | 2.0 | 109$\pm$ | 199 |  $>$16 yr |  | 1 |
UCAC4 115-008604 |  $-$902$\pm$ | 271 | 3.3 |  $-$1013$\pm$ | 401 | 2.5 | 118$\pm$ | 282 | 18.706 d |  | 2 | 0.296
Cl$*$ NGC 346 SSN 7 | 908$\pm$ | 206 | 4.4 | 801$\pm$ | 274 | 2.9 |  $-$122$\pm$ | 216 |  | 3.07359 d | 3 | 0.348
$\xi^{1}$ CMa |  $-$362$\pm$ | 44 | 8.3 |  $-$369$\pm$ | 76 | 4.8 | 96$\pm$ | 42 |  $\sim$30 yr |  | 4 |
HD 45166 |  $-$1107$\pm$ | 80 | 13.8 |  $-$1452$\pm$ | 254 | 5.7 |  $-$36$\pm$ | 76 | 124.8 d |  | 5 |

Notes: The first column lists the object name, followed by the measurement of
the longitudinal magnetic field using the full spectrum (Column 2) and only
the hydrogen lines (Column 4), with the corresponding $\sigma$ significance of
the measurement given in Columns 3 and 5. Column 6 lists the measurements
carried out using the null spectrum. In Column 7 we present the rotation
period ($P_{\rm rot}$) for each object and in Column 8 the orbital period
($P_{\rm orb}$), with the corresponding reference in Column 9 ((1) Nazé et al.
(2015); (2) Bagnulo et al. (2020); (3) Rickard & Pauli (2023); (4) Erba et al.
(2021); (5) Shenar et al. (2023)). The rotation or orbital phase for each
measurement is listed in Column 10.

##  4 Discussion

In Figs. 1 and 2 we present the observed Stokes $I$ and $V$ spectra, the
diagnostic $N_{V}$ spectra, and the corresponding weighted linear regression
lines through the measured data points for the reference magnetic massive
stars HD 45166 and $\xi^{1}$ CMa. Similar plots for the Of?p stars OGLE SMC-
SC6 237339 and UCAC4 115-008604, both of which have detected magnetic fields,
are shown in Figs. 3 and 4, and in Figs. 5 and 6 we present the plots for the
two Of?p stars with no magnetic field detection, BI 57 and AzV 220. Plots for
the contact binary with a detected magnetic field, SSN 7, are shown in Fig. 7.
The longitudinal magnetic field $\left<B_{\rm z}\right>_{\rm all}=-362\pm 44$
G measured for our reference target $\xi^{1}$ CMa, which has an extremely long
rotation period of about 30 yr, is in full agreement with the expected
negative extremum of the field predicted to occur in 2024 (Erba et al., 2021).
The measurement of HD 45166, reported to possess an extraordinary longitudinal
magnetic field of 13.5 kG (Shenar et al., 2023), now shows, for the first
time, a negative longitudinal kilogauss-scale field. For this system, the
stripped, helium-strong, quasi-Wolf–Rayet component may be due to multiple
interactions and a merging event.

Of the four observed Of?p stars, the presence of a mean longitudinal magnetic
field, $\left<B_{\rm z}\right>_{\rm all}$, is only detected in OGLE SMC-SC6
237339 (in the SMC), at a significance level of 3.8$\sigma$, and in UCAC4
115-008604 (in the LMC), at a significance level of 3.3$\sigma$. Minimum
dipole strengths ($B_{\rm d}$) of 2.4 kG and 2.7 kG were estimated for these
two stars using the relation $B_{\rm d}\geq 3\left|\left<B_{\rm z}\right>_{\rm
all}\right|$ (Babcock, 1958). A longitudinal magnetic field of $\left<B_{\rm
z}\right>_{\rm all}=908\pm 206$ G at a significance level of 4.4$\sigma$ is
detected in the contact binary system SSN 7. The corresponding minimum $B_{\rm
d}$ value for this system is 2.7 kG. Considering the spectral line content and
the radial velocities observed in our FORS2 spectrum of SSN 7, we conclude
that the magnetic field is located in the primary component. Due to the very
weak contribution of the secondary to the composite spectrum, no conclusion
can be drawn about the magnetic nature of this component. As the orbital and
stellar parameters for this binary are already known (Rickard & Pauli, 2023),
we were able to calculate the configuration and orientation of this system at
the time of the FORS2 observations using the PHOEBE code (PHysics Of Eclipsing
BinariEs222http://phoebe-project.org), which is an open source modelling code
for computing theoretical light and radial velocity curves (Prša et al.,
2016). It is a Wilson-Devinney-like eclipsing binary light curve modelling
code that uses Roche geometry to model stellar surfaces. The obtained PHOEBE
surface mesh model is presented in Fig. 8.

With regard to the significance levels of 3.8$\sigma$ and 3.3$\sigma$ for the
magnetic field detections in the Of?p stars OGLE SMC-SC6 237339 and UCAC4
115-008604, we note that the two clearly magnetic Galactic Of?p stars HD
148937 and CPD$-$28∘ 2561 were for the first time detected as magnetic in
FORS1 and FORS2 observations, at significance levels of 3.1$\sigma$ and
3.2$\sigma$, respectively, using the same instrumental setup (Hubrig et al.,
2008, 2011). This indicates that both Of?p stars in the MCs very likely
possess kilogauss-scale magnetic fields.

Figure 8:  PHOEBE surface mesh model showing the configuration and plane-of-
sky orientation of SSN 7 during the recently obtained FORS2 observations,
based on the previously reported orbital solution (Rickard & Pauli, 2023). The
face colour represents the temperature across the surface (see the colour
bar), and the axes show the $U$ and $V$ plane-of-sky orientation in units of
solar radius.

The detection of magnetic fields in Of?p stars – suggested to be merger
products in Galactic Of?p stars (Frost et al., 2021) – was more or less
expected but was probably hampered by the faintness of the targets. Several
years ago, Bagnulo et al. (2017, 2020) carried out similar FORS2 observations,
but in different rotation phases and with a different instrumental setup,
using GRISM 1200B, a slit width of 1′′, and a binning of 2, therefore covering
the wavelength region 3700 to 5120 Å. In contrast, our instrumental setup
using GRISM 600B allowed us to cover a larger wavelength range, from 3250 to
6215 Å, and allowed us to measure a polarisation signal in additional He i, He
ii, and metal lines, improving the measurement accuracy. Furthermore, only
absorption lines were used for the magnetic field measurements by Bagnulo et
al. (2017, 2020) despite the fact that Of?p stars present numerous lines that
are in emission or display P Cyg-type profiles. None of the obtained
measurements of the mean longitudinal magnetic field in that study showed a
definite 3$\sigma$ detection.

The measurement accuracies achieved by Bagnulo et al. (2017, 2020) were of the
order of several hundred gauss: 575 G and 305 G respectively for the two
observations of AzV 220 in 2015 and 2017, compared to our measurement accuracy
of 202 G; 530 G for the single observation of OGLE SMC-SC6 237339 in 2015
compared to our accuracy of 209 G; 345 G for the single observation of BI 57
in 2016 compared to our accuracy of 440 G; and 235 G and 275 G for the two
observations of UCAC4 115-008604 in 2017 compared to our accuracy of 271 G.

It is not clear whether the non-detections reported by Bagnulo et al. (2017,
2020) were due to the observations being carried out at unfavourable
rotational phases. As we report in Sect. 2, because the longitudinal magnetic
field is strongly dependent on the viewing angle of the observer (i.e. on the
rotation angle of the star), we tried to avoid rotation phases when the light
curves for the studied targets were at a minimum. Our observations that
yielded field detections were carried out at very different rotational phases
than those covered by Bagnulo et al. (2017, 2020). This applies for our
observations of OGLE SMC-SC6 237339 at a rotation phase of 0.545 (close to the
maximum of the OGLE light curve phase of 0.5), whereas the observations by
Bagnulo et al. (2017, 2020) took place at a rotational phase of 0.42. We
observed the target UCAC4 115-008604 at a rotational phase of 0.296, which is
close to the light curve maximum (~0.4), whereas the two previous observations
of this target were carried out at rotational phases of 0.74 and 0.85,
respectively. Also, our observations of the other two Of?p stars, AzV 220 and
BI 57, that yielded non-detections were carried out at different rotational
phases than those by Bagnulo et al. (2017, 2020), and they yield very
different longitudinal magnetic field strengths. It is possible that the
differences between the different field measurements are caused by rotational
modulation, but the number of measurements is currently too small to draw any
firm conclusion.

The detection of a magnetic field in the extragalactic contact binary SSN 7 in
a lower-metallicity environment provides an important constraint on the
strength of magnetic fields formed through mass transfer. This detection also
allowed us to robustly test theoretical predictions. According to recent
numerical simulations (e.g. Schneider et al. 2019), the magnetic field is
formed in two steps, first when the binary system begins its mass transfer and
second when the cores of the two components come into contact. In the first
step, sheer between the accreting stream and the surface of the accretor forms
a modest magnetic field, which is then bolstered considerably during the
second step when the cores merge. Therefore, magnetic fields in massive stars
would only be expected in actively interacting or post-interaction systems.
Importantly, stripped-envelope stars formed in systems with interacting
components through one or multiple phases of Roche-lobe overflow may be the
progenitors of gravitational waves (e.g. Laplace et al. 2020). It is of
interest that rather strong longitudinal magnetic fields have recently been
reported for the two well-known Galactic over-contact binaries LY Aur (=HD
35921) and MY Ser (=HD 167971) by Hubrig et al. (2023). Together with
Plaskett’s star – which is in a semi-detached configuration (Grunhut et al.,
2022) – these are the only known Galactic magnetic O-type stars in close
binary systems that fit the proposed scenario, as the over-contact phase is
the last evolutionary phase before a massive stellar merger.

It is not clear whether studies of pulsations can help discriminate between
magnetic and non-magnetic pulsating massive stars. A recent study by Kurtz et
al. (2020) reported the detection of coherent pulsation modes in the Of?p star
NGC 1624-2 based on high-cadence Transiting Exoplanet Survey Satellite (TESS;
Ricker et al. 2015) photometry. With a surface magnetic field strength of the
order of 12 kG, this star possesses the strongest magnetic field ever measured
in an O-type star (e.g. Järvinen et al. 2021). For all targets in our sample,
TESS data are available, although a careful analysis is necessary because the
signal can be affected by contamination from nearby stars in the crowded field
of view. Interestingly, our preliminary inspection of TESS data indicates the
possible presence of periods of several hours to one day in three Of?p stars,
but does not confirm the previously reported 18.7 d rotation period for UCAC4
115-008604 (Bagnulo et al., 2020) or the spectroscopic period of about 3 d for
SSN 7 (Rickard & Pauli, 2023).

Studies of magnetic fields in massive stars with lower metallicities provide
important information on the role of magnetic fields in star formation in the
early Universe in minimally polluted gas. Our magnetic field detections in two
Of?p stars and one contact binary located in the MCs suggests that the impact
of the lower-metallicity environment of the MCs on the occurrence and strength
of stellar magnetic fields in massive stars is low. However, because the
explored stellar sample is very small, additional observations of massive
stars in the MCs are needed. Notably, studies of other characteristics of
massive stars in these galaxies report that lower metallicities lead to a
lower opacity of the lines that drive the wind, which has an important effect
on the wind characteristics of a star (i.e. the terminal velocity and the
mass-loss rate; Vink et al. 2001). The modified mass loss, however, impacts
the evolution, rotation, nucleosynthesis, and ionising photon production of
massive stars. While the physical properties of the Of?p stars in our sample
have not yet been studied in detail, the spectroscopic and orbital
characteristics of the contact system SSN 7, including the evolutionary model,
are already available thanks to its location in a frequently surveyed
starburst region of the SMC. It has been suggested that SSN 7 is the most
massive Algol-like system discovered to date (Rickard & Pauli, 2023). The mass
of the primary component, $M_{1}=32\,M_{\odot}$, is surprisingly low for its
luminosity, and the mass of the secondary is $M_{2}=55\,M_{\odot}$. The low
mass of the primary indicates that the binary components must be interacting
or have interacted in the past. Indeed, their Roche radii suggest that the
stars are still in contact, currently slightly overfilling their Roche lobe
with $R_{1}/R_{\rm RL}=1.01$ and $R_{2}/R_{\rm RL}=1.03$. A characterisation
of the targets with detected magnetic fields – including their atmospheric
parameters, wind properties, and magnetic field geometry – is urgently needed
to anchor stellar models appropriate for a lower-metallicity environment.

###### Acknowledgements.

We thank the anonymous referee for constructive comments and suggestions.
Based on observations collected at the European Southern Observatory under ESO
programme 111.24JY.001. This paper includes data collected by the TESS
mission. Funding for the TESS mission is provided by the NASA’s Science
Mission Directorate. NOIRLab IRAF is distributed by the Community Science and
Data Center at NSF NOIRLab, which is managed by the Association of
Universities for Research in Astronomy (AURA) under a cooperative agreement
with the U.S. National Science Foundation. This project received the support
of two fellowships from “La Caixa” Foundation (ID 100010434). The fellowship
codes are LCF/BQ/PI23/11970031 (AE) and LCF/BQ/PI23/11970035 (MAM). The
authors would like to thank the Paranal team for the excellent support during
the execution of the programme.

## References

  * Angel & Landstreet (1970) Angel, J. R. P. & Landstreet, J. D. 1970, ApJ, 160, L147
  * Appenzeller et al. (1998) Appenzeller, I., Fricke, K., Fürtig, W., et al. 1998, The Messenger, 94, 1
  * Babcock (1958) Babcock, H. W. 1958, ApJS, 3, 141
  * Bagnulo et al. (2017) Bagnulo, S., Nazé, Y., Howarth, I. D., et al. 2017, A&A, 601, A136
  * Bagnulo et al. (2020) Bagnulo, S., Wade, G. A., Nazé, Y., et al. 2020, A&A, 635, A163
  * Cikota et al. (2017) Cikota, A., Patat, F., Cikota, S., & Faran, T. 2017, MNRAS, 464, 4146
  * Dufton et al. (2019) Dufton, P. L., Evans, C. J., Hunter, I., Lennon, D. J., & Schneider, F. R. N. 2019, A&A, 626, A50
  * Ekström et al. (2019) Ekström, S., Meynet, G., Georgy, C., et al. 2019, in International Astronomical Union Symposia, Vol. 344, Dwarf Galaxies: From the Deep Universe to the Present, ed. K. B. W. McQuinn & S. Stierwalt, 153–160
  * Erba et al. (2021) Erba, C., Shultz, M. E., Petit, V., et al. 2021, MNRAS, 506, 2296
  * Frost et al. (2021) Frost, A. J., Mahy, L., Sana, H., et al. 2021, in OBA Stars: Variability and Magnetic Fields, 19
  * Grunhut et al. (2022) Grunhut, J. H., Wade, G. A., Folsom, C. P., et al. 2022, MNRAS, 512, 1944
  * Grunhut et al. (2017) Grunhut, J. H., Wade, G. A., Neiner, C., et al. 2017, MNRAS, 465, 2432
  * Hubrig et al. (2006) Hubrig, S., Briquet, M., Schöller, M., et al. 2006, MNRAS, 369, L61
  * Hubrig et al. (2023) Hubrig, S., Järvinen, S. P., Ilyin, I., Schöller, M., & Jayaraman, R. 2023, MNRAS, 521, 6228
  * Hubrig et al. (2004a) Hubrig, S., Kurtz, D. W., Bagnulo, S., et al. 2004a, A&A, 415, 661
  * Hubrig et al. (2020) Hubrig, S., Schöller, M., Cikota, A., & Järvinen, S. P. 2020, MNRAS, 499, L116
  * Hubrig et al. (2011) Hubrig, S., Schöller, M., Kharchenko, N. V., et al. 2011, A&A, 528, A151
  * Hubrig et al. (2014) Hubrig, S., Schöller, M., & Kholtygin, A. F. 2014, MNRAS, 440, 1779
  * Hubrig et al. (2008) Hubrig, S., Schöller, M., Schnerr, R. S., et al. 2008, A&A, 490, 793
  * Hubrig et al. (2004b) Hubrig, S., Szeifert, T., Schöller, M., Mathys, G., & Kurtz, D. W. 2004b, A&A, 415, 685
  * Järvinen et al. (2021) Järvinen, S. P., Hubrig, S., Schöller, M., et al. 2021, MNRAS, 501, 4534
  * Kurtz et al. (2020) Kurtz, D. W., Hubrig, S., Järvinen, S. P., & Schöller, M. 2020, Research Notes of the American Astronomical Society, 4, 157
  * Laplace et al. (2020) Laplace, E., Götberg, Y., de Mink, S. E., Justham, S., & Farmer, R. 2020, A&A, 637, A6
  * Munoz et al. (2020) Munoz, M. S., Wade, G. A., Nazé, Y., et al. 2020, MNRAS, 492, 1199
  * Nazé et al. (2015) Nazé, Y., Walborn, N. R., Morrell, N., Wade, G. A., & Szymański, M. K. 2015, A&A, 577, A107
  * Neugent et al. (2018) Neugent, K. F., Massey, P., & Morrell, N. 2018, ApJ, 863, 181
  * Pelisoli et al. (2022) Pelisoli, I., Dorsch, M., Heber, U., et al. 2022, MNRAS, 515, 2496
  * Prša et al. (2016) Prša, A., Conroy, K. E., Horvat, M., et al. 2016, ApJS, 227, 29
  * Regimbau & de Freitas Pacheco (2006) Regimbau, T. & de Freitas Pacheco, J. A. 2006, A&A, 447, 1
  * Rickard & Pauli (2023) Rickard, M. J. & Pauli, D. 2023, A&A, 674, A56
  * Ricker et al. (2015) Ricker, G. R., Winn, J. N., Vanderspek, R., et al. 2015, Journal of Astronomical Telescopes, Instruments, and Systems, 1, 014003
  * Schneider et al. (2020) Schneider, F. R. N., Ohlmann, S. T., Podsiadlowski, P., et al. 2020, MNRAS, 495, 2796
  * Schneider et al. (2019) Schneider, F. R. N., Ohlmann, S. T., Podsiadlowski, P., et al. 2019, Nature, 574, 211
  * Schneider et al. (2016) Schneider, F. R. N., Podsiadlowski, P., Langer, N., Castro, N., & Fossati, L. 2016, MNRAS, 457, 2355
  * Schöller et al. (2017) Schöller, M., Hubrig, S., Fossati, L., et al. 2017, A&A, 599, A66
  * Shenar et al. (2023) Shenar, T., Wade, G. A., Marchant, P., et al. 2023, Science, 381, 761
  * Steffen et al. (2014) Steffen, M., Hubrig, S., Todt, H., et al. 2014, A&A, 570, A88
  * Tody (1993) Tody, D. 1993, in Astronomical Society of the Pacific Conference Series, Vol. 52, Astronomical Data Analysis Software and Systems II, ed. R. J. Hanisch, R. J. V. Brissenden, & J. Barnes, 173
  * Uzdensky & MacFadyen (2006) Uzdensky, D. A. & MacFadyen, A. I. 2006, ApJ, 647, 1192
  * van Dokkum (2001) van Dokkum, P. G. 2001, PASP, 113, 1420
  * Vink et al. (2001) Vink, J. S., de Koter, A., & Lamers, H. J. G. L. M. 2001, A&A, 369, 574
  * Walborn (1972) Walborn, N. R. 1972, AJ, 77, 312
  * Walborn et al. (2015) Walborn, N. R., Morrell, N. I., Nazé, Y., et al. 2015, AJ, 150, 99
  * Walborn et al. (2010) Walborn, N. R., Sota, A., Maíz Apellániz, J., et al. 2010, ApJ, 711, L143