# INVESTIGATING PERSONALIZATION METHODS IN TEXT TO MUSIC GENERATION

###### Abstract

In this work, we investigate the personalization of text-to-music diffusion
models in a few-shot setting. Motivated by recent advances in the computer
vision domain, we are the first to explore the combination of pre-trained
text-to-audio diffusers with two established personalization methods. We
experiment with the effect of audio-specific data augmentation on the overall
system performance and assess different training strategies. For evaluation,
we construct a novel dataset with prompts and music clips. We consider both
embedding-based and music-specific metrics for quantitative evaluation, as
well as a user study for qualitative evaluation. Our analysis shows that
similarity metrics are in accordance with user preferences and that current
personalization approaches tend to learn rhythmic music constructs more easily
than melody. The code, dataset, and example material of this study are open to
the research community.

Index Terms—  text-to-music, diffusion, personalization

##  1 Introduction

Creating customized music and sound effects to meet individualized
specifications can have significant impact across diverse application domains,
including music production, augmented and virtual reality, and game
development applications. In recent years, there has been a growing number of
text-to-music generative models [1, 2]. These models are versatile, capable of
generating a diverse range of audio, including music, based on a textual
prompt.

Guiding such models to a desirable output sound is not straightforward,
requiring considerable prompt engineering [3]. This means that there is no way
to finely control the generation process to consistently produce sounds based
on a specific example. This challenge arises either because the model cannot
produce any instance of a class of sounds (e.g. an obscure ethnic instrument)
or because the desired sound is a specific instance of a known class (e.g.
producing a user’s guitar playing style) that cannot be yielded even with the
most detailed textual description. For instance, can one generate a rock song
using their personal guitar playing style or a specific ethnic instrument?

Fig. 1: An overview of text-to-music personalization. With just a few audio
clips, we implant a novel musical concept into a pre-trained text-to-audio
model, enabling its manipulation with textual prompts.

In the image domain, this problem is addressed by personalization methods that
expand the language-vision dictionary of the model so that it binds new words
with user-specific concepts. This enables the generation of a user-specific
concept in different contexts and stylistic variations while maintaining its
distinct characteristics [4]. Given a few examples, e.g. ${\sim}3-5$ images of
a dog in different backgrounds and views, the objective of personalization
methods is to inject it into the model such that it can be synthesized with a
unique identifier (e.g. a pseudoword).

Recently, several approaches based on pre-trained text-to-image diffusion
models have been proposed [5, 4, 6]. Textual Inversion [5] adds a new word
embedding for the novel concept and associates it with a pseudoword $V_{*}$.
The embedding is trained with prompts of the form ”a photo a $V_{*}$” via the
standard denoising objective [7] while the model is kept frozen. In DreamBooth
[4] the full weights of the model are finetuned while a prior preservation
loss prevents the model from catastrophic forgetting and language drift [8].
CustomDiffusion [6] and SVDiff [9] reduce the amount of fine-tuning parameters
by only training the cross-attention layers. While text-to-image
personalization has been widely explored, the adoption of such methods for
controllable music generation has not been addressed.

In this work, we start from a pre-trained text-to-audio diffusion model, i.e.
AudioLDM [1], and to the best of our knowledge are the first to investigate
the ability to personalize its outputs for newly learned musical concepts in a
few-shot manner. Motivated by the computer vision literature, we explore the
application of two established methods, i.e. Textual Inversion [5] and
Dreambooth [4]. We adapt these methods for music personalization and
experiment with different training configurations. We evaluate the capacity of
the model to learn new concepts along two dimensions, _reconstruction_ , i.e.
the ability to faithfully reconstruct the novel concept, and _editability_ ,
i.e. the ability to manipulate it through textual prompts. To this end, we
construct a new dataset of various instruments and playing styles. Our
evaluation protocol consists of a) embedding distance-based metrics, b) music-
specific metrics, and c) an A/B testing user study comparing the two
adaptation approaches. Finally, we adapt AudioLDM to perform text-guided style
transfer for newly learned concepts.

Our key contributions are a) the personalization of AudioLDM’s generation and
style-transfer abilities for new concepts, b) the exploration of audio-
specific augmentations and evaluation metrics, and c) the construction of a
new dataset for text-to-music personalization methods. Our code and data, as
well as generated music samples, are publicly available
111https://zelaki.github.io/.

##  2 Methods

Text-to-Audio Latent Diffusion Models: Diffusion Models [10] are probabilistic
generative models that learn a data distribution by gradually denoising a
latent variable sampled from a Gaussian distribution. This corresponds to
learning the reverse process of a fixed-length Markovian forward process.

In Latent Diffusion Models (LDMs) [7], the denoising process occurs in the
latent space of an encoder-decoder architecture ($\mathcal{E}$,$\mathcal{D}$)
trained on a large collection of samples. Given an audio sample $x$, a text-
guided latent diffusion model is conditioned on a text-embedding model
$c_{\tau}$. The LDM loss is then given by:

$\mathcal{L}_{LDM}=\mathbb{E}_{z\sim\mathcal{E}(x),\epsilon\sim\mathcal{N}(0,I),y,t}\left[\|\epsilon-\hat{\epsilon}_{\phi}(z_{t},t,c_{\tau}(y)\|_{2}^{2}\right]$
(1)

where $\phi$,$\tau$ are the parameters of the denoising network
$\hat{\epsilon}$ and the text encoder $c$ respectively, $t$ is the time step,
$z_{t}$ is the latent representation of $x$ noised to time $t$ and $\epsilon$
is the sample noise. While training, the parameters $\theta=\phi\;\cup\;\tau$
are jointly optimized to minimize the LDM loss. Intuitively, the objective
aims to correctly remove the noise added to a latent representation of an
audio. At inference, a random noise tensor is sampled and iteratively denoised
to produce a new audio latent, $z_{0}$, which is transformed into audio
through the pre-trained decoder $\hat{x}=\mathcal{D}(z_{0})$.

Fig. 2: We illustrate the proposed methods for personalized text-to-music
generation. In Textual Inversion only the novel embedding $v_{*}$ is optimized
while in DreamBooth the full Denoising Network is finetuned.

Personalization of Text-to-Audio Models: Initially, a placeholder string,
$S_{*}$, representing the novel concept is associated with a unique embedding
vector $v_{*}$ that can be retrieved via an embedding lookup as in Fig.2. Thus
the parameter space of the text encoder $c$ becomes
$\tau^{\prime}=\tau\;\cup\;v_{*}$ and the trainable parameters of the model
become $\theta^{\prime}=\phi\;\cup\;\tau\cup\;v_{*}$. The generation is
conditioned to a constructed neutral text prompt $y$ e.g. ”A recording of a
$S_{*}$”.

By directly minimizing the LDM loss (1) over the small training set
representing the concept and choosing different subsets of the parameter space
of the model $\theta^{\prime}$ for training, different methods for learning
the novel concept can be derived. In Dreambooth (DB) the weights of the
denoising network $\phi$ are optimized while in Textual Inversion (TI) $\phi$
and $\tau$ are kept frozen and the only learnable parameters are the weights
of the embedding $v_{*}$.

Personalized Style Transfer: Given an input audio sample $x_{in}$, we can
calculate its noisy latent representation $z_{t}$ with a predefined time step
$t\leq N$ according to the forward process [1, 10]. By utilizing $z_{t}$ as
the starting point of the reverse process of a pre-trained AudioLDM model, we
enable the manipulation of $x_{in}$ with text input y with a shallow reverse
process:

$p_{\theta^{\prime}}(z_{0:t}|c(y)=p(z_{t})\prod_{n=1}^{t}p_{\theta^{\prime}}(z_{n-1}|z_{n},c(y)),$
(2)

where $t$ controls the transfer strength. To infuse the style of the input
sample $x_{in}$ with the characteristics of an acquired concept, we set
$y=S_{*}$, where $S_{*}$ is the placeholder string linked to the newly learned
concept.

##  3 Experiments

Dataset. We collect a dataset of 32 musical concepts including Percussion
Instruments and Beats, Solo Melodic Instruments, and Multi-Instrument Pieces,
from a wide array of musical cultures and playing styles. Each concept
includes five 10-second audio clips. All audio clips are either recorded by
the authors or sourced from Freesound and YouTube. We also collect 20
editability prompts that aim to manipulate the genre, recording conditions,
accompaniments, and background sounds. The full list of prompts can be found
in the provided example page.

Experimental Setup. As a backbone for all our experiments we utilize AudioLDM-
Medium 222https://huggingface.co/cvssp/audioldm-m-full. We conduct our
experiments using a single NVIDIA RTX-3090 GPU with a batch size of 4,
employing learning rates of $2\times 10^{-2}$ and $4\times 10^{-6}$ for TI and
DB, respectively, and running 150 optimization steps for TI and 1500 for DB.

Following the original papers for both methods, a placeholder token $S_{*}$ is
reserved for the novel concept. In the case of TI, $S_{*}$ is a new token,
inserted into the tokenizer, while for DB, $S_{*}$ = [identifier] [class
noun], where [identifier] is an existing rare word in the tokenizer and [class
noun] is a coarse descriptor of the musical concept [4]. We experiment with
different training configurations including training with 1 or 3 audio clips
and randomly mixing the training audio with environmental sounds sourced from
AudioSet [11] with SNR$=20\;$dB. We will refer to these experiments as 1-AC,
3-AC, and MIX respectively.

We further conduct ablation experiments specific to each method. For DB we
include the text encoder in training, denoted as TE. For TI, we explore two
possible initializations for the learnable embedding $v_{*}$. As a baseline,
we consider the initialization of $v*$ from the mean of all word embeddings in
the vocabulary. Alternatively, we initialize $v*$ from the mean of the
embeddings of [class noun]. We refer to the baseline and mean word
initialization as BL and MW. For evaluation, we generate four 10-second music
clips per concept and per prompt, totaling 2560 clips.

Evaluation Metrics. We evaluate the audio similarity between the training set
and the music clips generated with a reconstruction prompt ”a recording of a
”$S_{*}$”, by measuring  CLAP-A and FAD [12] scores. CLAP-A is the average
pairwise cosine similarity between CLAP [13] audio embeddings of generated and
training clips. FAD is the Frechet distance between the embeddings of a pre-
trained VGGish audio classifier of the training set and the generated music
clips. We further evaluate the model’s capacity to manipulate a learned audio
concept via the editability prompts. For this, we calculate the average cosine
similarity between the embedding of the editability prompt and the audio CLAP
embeddings and denote this metric as CLAP-T.

Fig. 3: Audio and Text similarities for all experiments. TI methods (in
orange) have roughly the same CLAP audio similarity but consistently lower
text similarity than the DB ones.

Finally, we employ a set of automatically extracted music-specific metrics, in
order to assess the capability of the model to retain certain musical
properties, such as rhythm, harmony, and dynamics. To measure rhythmic
similarity we compute the Beats per Minute (BPM) of the concepts that have a
constant BPM and consider generated audio clips similar if they are within a 5
BPM tolerance [14]. To measure similarity in dynamics, we use the European
Broadcasting Union (EBU) R 128 Loudness Scale [15], and consider clips similar
in absolute loudness if they are within 2.5 LUFS of the mean training set
loudness [16]. Finally, to measure harmonic similarity, we use a key detection
algorithm based on Harmonic Pitch Class Profiles (HPCP) [17], which detects
the fundamental tone of the clip’s key, as well as whether it is major or
minor. We compute the scale and key for all concepts that contain harmonic
instruments and compare the generated clips’ key and scale to the most common
key and scale in the training set. All features are calculated using Essentia
[18].

##  4 Results

Quantitative Analysis: In Fig. 3, we summarize our experimental results for
the CLAP-A and CLAP-T metrics. To gain an intuition for the scale of the
results, we add three references. The Audio Similarity ceiling, $\lceil
A\rceil$ is the mean CLAP-A between the training samples of the concepts, the
Text Similarity ceiling $\lceil T\rceil$ is the CLAP-T between the generated
audio and the editability prompts without $S_{*}$ and the Text Similarity
floor $\lfloor T\rfloor$ is the CLAP-T between the training audio and the
editability prompts. We include $\lceil T\rceil$ and $\lfloor T\rfloor$ scores
to emphasize that the editability of the learned concepts is constrained by
the prior manipulation capabilities of AudioLDM.

We observe that DB outperforms TI in terms of both similarity metrics. Similar
to the computer vision literature, we observe a “Pareto front” formed along
the text and audio similarity axes, especially for DB [19, 20]. When analyzing
different training configurations, we observe that training using only one
audio clip yields worse results both in terms of reconstruction (audio
similarity) and in terms of editability (text similarity). Additionally, we
observe that the MIX strategy provides a good balance between reconstruction
and editability. Further, MW outperforms the baseline TI, leveraging the prior
of the embeddings of [class word]. Finally, training the text encoder overfits
audio reconstruction while impeding the manipulation capacity through textual
prompts.

In Table 1, we see a detailed view of the effect of different training
configurations and include results on FAD score. We observe that the audio
reconstruction capability is strongly emphasized by the FAD score. While the
MIX strategy has a significant impact on DB, it does not improve TI, since the
regularization performed due to the data augmentation has a larger impact on
the fully fine-tuned DB architecture.

Method | Setup |  CLAP-A ($\mathbf{\uparrow}$) |  FAD ($\mathbf{\downarrow}$) |  CLAP-T ($\mathbf{\uparrow}$)
---|---|---|---|---
DB | BL | $0.80$ | $12.37$ | $0.30$
1-AC | $0.78$ | $12.5$ | $0.29$
3-AC | $0.78$ | $13.04$ | $0.29$
TE | $\mathbf{0.88}$ | $\mathbf{8.1}$ | $0.26$
MIX | $0.84$ | $11.47$ | $\mathbf{0.33}$
TI | BL | $0.81$ | $17.17$ | $0.25$
1-AC | $0.810$ | $19.32$ | $0.21$
3-AC | $0.86$ | $17.45$ | $0.20$
MW | $0.83$ | $16.61$ | $0.27$
MIX | $0.816$ | $17.24$ | $0.25$

Table 1: Quantitative metrics comparing the different evaluation setups, for
each method and training configuration. Fig. 4: Audio Similarity for the input
source and the target concept for personalized style transfer, with samples
generated with transfer strength $t\in[0,1]$. The source is a 10-second clip
from Stairway to Heaven and the target concept is a Middle Eastern string
instrument called Oud.

Human Preference Study: We conduct a human preference study comparing DB and
TI, in the form of an A/B testing setup, by creating an online survey
consisting of $20$ questions. Half of the questions aim to evaluate the audio
reconstruction ability, i.e. “which of the two generated clips better matches
the reference audio clip”. The other half, aim to evaluate the editability of
DB and TI, by presenting the user with a textual prompt, including the novel
concept class name, and then asking which of the generated clips better
matches the prompt. The possible preference answers for all questions are “A”,
“B”, “None”, and “Cannot decide”. The study was completed by $34$ users. In
Table 2 we can see that there is a preference towards DB instead of TI, in
terms of reconstruction and editability in alignment with quantitative
metrics. Furthermore, we observe that a significant amount of users do not
prefer either DB or TI in the editability questions, indicating that text-to-
music personalization still has room for improvement.

| DB | TI | None | Undecided
---|---|---|---|---
Reconstruction | $\mathbf{58}\textbf{\%}$ |  $24$% |  $9$% |  $9$%
Editability |  $\mathbf{37}$% |  $31$% |  $24$% |  $8$%

Table 2: Human preference study results.

Music-Specific Evaluation: In Fig. 5 we illustrate the results for DB in three
training configurations and the baseline TI on the proposed music metrics.
Initially, we observe that DB can effectively retain tempo, while TI cannot.
Additionally, while both methods are able to reconstruct the scale to some
extent they fail to reconstruct the key. Finally, both methods cannot generate
clips with loudness comparable to the training set. We hypothesize that this
is due to the model’s normalizing effect on the generated audio.

Fig. 5: Low-level music audio features for some of the training
configurations.

Text-to-Audio Style-Transfer: In Fig. 4 we perform personalized style-transfer
for a novel concept, using TI+MW. We present that by increasing transfer
strength the generated audio progressively becomes similar to the target clip,
and dissimilar to the source clip. Furthermore, the range $0.4-0.6$ for the
transfer strength appears to be a sweet spot for performing style-transfer,
while maintaining the source audio properties.

##  5 Conclusion and Future Work

In this work, we conduct a preliminary study for the personalization of text-
to-music generation models adapting them to user-specific needs. We explore
the application of two established methods, namely Textual Inversion and
DreamBooth. Both methods are evaluated on their ability to learn and modify
new musical concepts, using quantitative metrics and a user study. We
construct a new evaluation dataset, investigate diverse training
configurations, and propose a music-specific evaluation framework. In the
future, we aim to explore multi-concept text-to-music personalization,
learning multiple music concepts from a single mixture, with a source
separation regularization objective.

## References

  * [1] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley,  “Audioldm: Text-to-audio generation with latent diffusion models,”  arXiv preprint arXiv:2301.12503, 2023.
  * [2] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez,  “Simple and controllable music generation,”  arXiv preprint arXiv:2306.05284, 2023.
  * [3] Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, and Philip Torr,  “A systematic survey of prompt engineering on vision-language foundation models,” 2023.
  * [4] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman,  “Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation,”  in In Proc. CVPR, 2023.
  * [5] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or,  “An image is worth one word: Personalizing text-to-image generation using textual inversion,”  in The Eleventh International Conference on Learning Representations, 2023.
  * [6] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu,  “Multi-concept customization of text-to-image diffusion,”  2023\.
  * [7] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer,  “High-resolution image synthesis with latent diffusion models,”  in In Proc. CVPR, 2022, pp. 10684–10695.
  * [8] Jason Lee, Kyunghyun Cho, and Douwe Kiela,  “Countering language drift via visual grounding,”  arXiv preprint arXiv:1909.04499, 2019.
  * [9] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang,  “Svdiff: Compact parameter space for diffusion fine-tuning,” 2023.
  * [10] Jonathan Ho, Ajay Jain, and Pieter Abbeel,  “Denoising diffusion probabilistic models,”  Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020.
  * [11] Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter,  “Audio set: An ontology and human-labeled dataset for audio events,”  in In Proc ICASSP, 2017, pp. 776–780.
  * [12] Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi,  “Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms.,”  2019\.
  * [13] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov,  “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,”  in In Proc. ICASSP. IEEE, 2023, pp. 1–5.
  * [14] Jongpil Lee, Nicholas J. Bryan, Justin Salamon, Zeyu Jin, and Juhan Nam,  “Disentangled multidimensional metric learning for music similarity,”  in In Proc. ICASSP. IEEE, 2020.
  * [15] European Broadcasting Union,  “Loudness normalisation and permitted maximum level of audio signals,” 2020.
  * [16] Fabian Begnert, Håkan Ekman, and Jan Berg,  “Difference between the ebu r-128 meter recommendation and human subjective loudness perception,”  in Audio Engineering Society Convention 131. Audio Engineering Society, 2011.
  * [17] Emir Demirel, Baris Bozkurt, and Xavier Serra,  “Automatic chord-scale recognition using harmonic pitch class profiles,”  in Barbancho I, Tardón LJ, Peinado A, Barbancho AM, editors. Proceedings of the 16th Sound & Music Computing Conference; 2019 May 28-31; Málaga, Spain.[Málaga]: SMC; 2019. Sound & Music Computing Conference, 2019.
  * [18] Dmitry Bogdanov, Nicolas Wack, Emilia Gómez, Sankalp Gulati, Perfecto Herrera, Oscar Mayor, Gerard Roma, Justin Salamon, José Zapata, and Xavier Serra,  “Essentia: An open-source library for sound and music analysis,”  in Proceedings of the 21st ACM International Conference on Multimedia, New York, NY, USA, 2013, MM ’13, p. 855–858, Association for Computing Machinery.
  * [19] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or,  “Encoder-based domain tuning for fast personalization of text-to-image models,”  ACM Transactions on Graphics (TOG), vol. 42, no. 4, pp. 1–13, 2023\.
  * [20] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon,  “Key-locked rank one editing for text-to-image personalization,”  in ACM SIGGRAPH 2023 Conference Proceedings, 2023, pp. 1–11.


# Learning Multi-Site Harmonization of Magnetic Resonance Images Without
Traveling Human Phantoms

Siyuan Liu1 and Pew-Thian Yap1,✉

###### Abstract

Harmonization improves data consistency and is central to effective
integration of diverse imaging data acquired across multiple sites. Recent
deep learning techniques for harmonization are predominantly supervised in
nature and hence require imaging data of the same human subjects to be
acquired at multiple sites. Data collection as such requires the human
subjects to travel across sites and is hence challenging, costly, and
impractical, more so when sufficient sample size is needed for reliable
network training. Here we show how harmonization can be achieved with a deep
neural network that does not rely on traveling human phantom data. Our method
disentangles site-specific appearance information and site-invariant
anatomical information from images acquired at multiple sites and then employs
the disentangled information to generate the image of each subject for any
target site. We demonstrate with more than 6,000 multi-site T1- and
T2-weighted images that our method is remarkably effective in generating
images with realistic site-specific appearances without altering anatomical
details. Our method allows retrospective harmonization of data in a wide range
of existing modern large-scale imaging studies, conducted via different
scanners and protocols, without additional data collection.

Department of Radiology and Biomedical Research Imaging Center (BRIC),
University of North Carolina at Chapel Hill, NC, U.S.A.

††footnotetext: ✉ Corresponding author: Pew-Thian Yap<EMAIL_ADDRESS>
## Introduction

Magnetic resonance imaging (MRI) is a non-invasive and versatile technique
that provides good soft tissue contrasts useful for diagnosis, prognosis, and
treatment monitoring. Since MRI experiments are costly and time-consuming,
modern large-scale MRI studies typically rely on multiple imaging sites to
collaboratively collect data with greater sample sizes for more comprehensive
coverage of factors that can affect study outcomes, such as age, gender,
geography, socioeconomic status, and disease subtypes. Notable examples of
multi-site studies include the Adolescent Brain Cognitive Development
(ABCD)[1], the Alzheimer’s Disease Neuroimaging Initiative (ADNI)[2], and the
Australian Imaging, Biomarkers and Lifestyle Flagship Study of Aging
(AIBL)[3].

Multi-site data collection leads inevitably to undesirable elevation of non-
biological variability introduced by differences in scanners[4] and imaging
protocols[5]. Protocols can be prospectively harmonized by selecting for the
individual sites the acquisition parameters that result in images with maximal
inter-site consistency. However, prospective harmonization requires extensive
data acquisition for parameter tuning, needs to be performed before each
study, and does not allow for correction of data collected in studies that
have already taken place. Moreover, significant inter-site variability can
still occur in data collected with harmonized acquisition protocols simply due
to irreconcilable differences between scanners[4]. Retrospective MRI
harmonization[6] overcomes these limitations by performing post-acquisition
correction and is hence applicable to existing studies for improving the
accuracy, reliability, and reproducibility of downstream processing, analyses,
and inferences.

Table 1: Existing MRI harmonization methods Category |  Method |  Summary
---|---|---
Statistics | Intensity normalization |  Histogram matching[9, 8, 7] |  Align distributions of voxel intensity values based on an image template constructed from several control subjects.
|  |  White stripe[10] |  Normalize intensity values based on patches of normal appearing white matter (NAWM). Rescaled intensity values are biologically interpretable as units of NAWM.
|  |  Multi-site image harmonization by cumulative distribution function alignment (MICA)[11] |  Transform voxel intensity values of one scan to align with the intensity cumulative distribution function (CDF) of a second scan.
| Batch effect adjustment |  Removal of Artificial Voxel Effect by Linear regression (RAVEL)[12] |  Estimate using a control region of the brain the latent factors of unwanted variation common to all voxels.
|  |  Combating batch effects (ComBat)[13, 14] |  Identify batch-specific transformation to express all data in a common space.
Learning | Machine learning |  Regression ensembles with patch learning for image contrast agreement (REPLICA)[15] |  Supervised training of a random forest for nonlinear regression of a target contrast.
|  |  NeuroHarmony[16] |  Supervised training of a random forest for nonlinear regression of a target contrast determined based on prescribed regions.
| Deep learning |  DeepHarmony[17] |  Supervised training of a U-Net to produce images with a consistent contrast.
|  |  Disentangled Latent Space (DLS)[18] |  Supervised training of a deep neural network using disentangled anatomical and contrast components.
|  |  Unlearning of dataset bias[19] |  Supervised training of a deep neural network to learn scanner-invariant features and then reducing scanner influence on network predictions in tasks of interest.
|  |  MURD |  Unsupervised harmonization of images using content and style disentanglement learned from jointly multi-site images.
Figure 1: Overview of MURD. a, Multi-site representation disentanglement.
Given image sets from $N$ sites $\\{\mathcal{X}_{i}\\}_{i=1}^{N}$, MURD
disentangles each image set $\mathcal{X}_{i}$ in a site-invariant content
space $\mathcal{C}$ and a site-specific style space $\mathcal{S}_{i}$ with the
content features aligned in the content space $\mathcal{C}$. This is achieved
using content-style disentangled cycle translation (CS-DCT). b, Site-specific
CS-DCT. An image from the $i$-th site $x_{i}$ is encoded in content space
$\mathcal{C}$ and style space $\mathcal{S}_{i}$ to obtain content features
$c_{i}$ and style features $s_{i}$. Style features of the $j$-th site $s_{j}$
($j\neq i$) are generated using style generator $G^{\text{S}}_{j}$. $s_{j}$
and $c_{i}$ decoded to generate a harmonized image for the $j$-th site
$\tilde{x}_{j}$, which is in turn encoded to generate content features
$\tilde{c}_{j}$ for reconstructing image $\hat{x}_{i}$ with style features
$s_{i}$. c, Reference-specific CS-DCT. Unlike site-specific CS-DCT in b, style
features $s_{j}$ are obtained by encoding a reference image $x_{j}$ in style
space $\mathcal{S}_{j}$. d, The MURD framework. MURD implements CS-DCT by
employing a site-shared content encoder, a site-specific style encoder, a
site-shared generator, a site-specific style generator, and a site-specific
discriminator for content encoding, style encoding, decoding, style
generation, and adversarial learning, respectively. MURD is constrained by
four types of losses: consistency losses $\mathcal{L}_{\text{cyc}}$,
$\mathcal{L}_{\text{cont}}$ and $\mathcal{L}_{\text{sty}}$, adversarial loss
$\mathcal{L}_{\text{adv}}$, content alignment loss $\mathcal{L}_{\text{ca}}$,
and identity loss $\mathcal{L}_{\text{id}}$.

Existing retrospective harmonization methods are either statistics-based or
learning-based (Table 1). Statistics-based methods align intensity
distributions via intensity normalization[7, 8, 9, 10, 11] or batch effect
adjustment[12, 13, 14]. However, these methods are typically limited to whole-
image, but not detail-specific, harmonization. Learning-based methods
translate images between sites via nonlinear mappings determined using machine
learning[15, 16] or deep learning[17, 18, 19], with or without supervision.
Machine learning methods predict harmonized images using regression models
learned, typically with supervision, with hand-crafted features. In contrast,
deep learning techniques automatically extract features pertinent to the
harmonization task. Supervised methods typically require training data
acquired from traveling human phantoms, which might not always be available
for large-scale, multi-site, or longitudinal studies. Unsupervised deep
learning techniques[20, 21, 22] determine mappings between sites using
unpaired images and therefore avoid the need for traveling human phantom data.
These methods are however unscalable to large-scale multi-site MRI
harmonization as they typically learn pair-wise mappings between sites. For
$N$ sites, these methods learn $N(N-1)$ mappings for all site pairs and
therefore require a large amount of data for learning the multitude of network
parameters. These pair-wise methods are also ineffective by not fully and
jointly utilizing complementary information available from all sites.

Here, we draw inspiration from recent advancements in multi-domain image-to-
image translation[23, 24, 25] and introduce a unified framework for
simultaneous multi-site harmonization using only a single deep learning model.
Our method, called multi-site unsupervised representation disentangler (MURD),
decomposes an image into anatomical content that is invariant across sites and
appearance style (e.g., intensity and contrast) that is dependent on each
site. Harmonized images are generated by combining the content of the original
image with styles specific to the different sites. More specifically, encoding
an image in site-invariant and site-specific feature spaces is achieved via
two encoders, i.e., a content encoder that captures anatomical structures
common across sites and a style encoder that captures style information
specific to each site. A site-harmonized image is generated via a generator
that combines the extracted content with the style associated with a target
site. The target style can be specified by a reference image from a site or by
a randomized style code generated by a style generator specific to each site.
The latter allows multiple appearances to be generated in relation to natural
style variations associated with each site. The network is trained with losses
that are designed to promote full representation disentanglement and to
maximally preserve structural details. MURD is summarized in Figure 1 and
detailed in the Methods section.

## Results

Figure 2: Harmonization of T1- and T2-weighted Images. Site-specific
harmonization of a, T1-weighted images and b, T2-weighted images. Reference-
specific harmonization of c, T1-weighted images and d, T2-weighted images. The
original images are shown in the first column and the harmonized images are
shown for GE, Philips and Siemens respectively in the second to fourth
columns. MURD is remarkably effective in harmonizing contrasts and preserving
details. Figure 3: Numerical Evaluation of Harmonization Outcomes.
Quantitative evaluation conducted for the GE, Philips, and Siemens sites using
FID and KID as metrics for a, T1-weighted images ($n=600$ slices per site) and
b, T2-weighted images ($n=600$ slices per site) from the validation dataset,
and c, T1-weighted images ($n=60000$ slices per site) and d, T2-weighted
images ($n=60000$ slices per site) from the generalizability dataset.
Bullseyes show the means and error bars show the standard errors on the means.
MURD, both site-specific (SS) and reference-specific (RS), yields lower FID
and KID values that are closer to the reference values than DRIT++[26] and
StarGAN-v2[25]. The FID and KID values for the generalizability dataset are
largely consistent with those of the validation dataset.  Figure 4:
Validation via the Traveling Human Phantom Dataset. Evaluation of harmonized
a, T1-weighted images and b, T2-weighted images from the traveling human
phantom dataset for the following harmonization tasks: GE to Philips (GP,
$n=60$ slices), Philips to GE (PG, $n=60$ slices), GE to Siemens (GS, $n=600$
slices), Siemens to GE (SG, $n=600$ slices), Philips to Siemens (PS, $n=120$
slices), and Siemens to Philips (SP, $n=120$ slices). Segmentation consistency
of c, T1-weighted images and d, T2-weighted images from the traveling human
phantom dataset with and without harmonization: GP ($n=1$ volume), PG ($n=1$
volume), GS ($n=5$ volumes), SG ($n=5$ volumes), PS ($n=2$ volumes), and SP
($n=2$ volumes). The lines show the means and the shaded regions show the
standard errors on the means. MURD yields the best performance in terms of
MAE, MS-SSIM, and PSNR. The improvement in DSCs indicates that MURD
harmonization significantly increases consistency of tissue segmentation.
Figure 5:  Continuous and Incremental Harmonization. Continuous harmonization
of a, T1-weighted images and b, T2-weighted images between two sites,
demonstrating that MURD smoothly changes image contrasts without altering
anatomical details thanks to its ability to disentangle content and style
information.

###  0.1 Materials

We demonstrate the effectiveness of MURD on brain T1- and T2-weighted images
of children 9 to 10 years of age acquired through the Adolescent Brain
Cognitive Development (ABCD) study[1] using scanners from different vendors,
including General Electric (GE), Philips, and Siemens. Informed consents were
obtained from all participants[1]. Although the imaging protocols were matched
across scanners at the different imaging sites[27], inter-scanner variability
in appearance is still significant (Figure 2). For simplicity, we group the
images and treat them as coming from three _virtual_ sites—the GE, Philips,
and Siemens sites. We trained and tested MURD separately for T1-weighted
images and T2-weighted images. For each modality, we trained MURD using a
modest sample size of 20 images per vendor. Three datasets per modality were
used for testing different aspects of MURD:

  1. (i)

Validation Dataset – A small but diverse dataset of 10 images per vendor,
carefully selected to be structurally different from the images in the
training dataset. The purpose of this testing dataset is to test the
effectiveness of MURD beyond the training dataset.

  2. (ii)

Generalizability Dataset – A large dataset of 1000 images randomly selected
for each vendor. This testing dataset is allowed to be deliberately much
larger than the training dataset to test the generalizability of MURD.

  3. (iii)

Human Phantom Dataset – A traveling human phantom dataset of 1 subject scanned
on both GE and Philips scanners, 5 subjects scanned on both GE and Siemens
scanners, and 2 subjects scanned on both Philips and Siemens scanners. This
testing dataset allows numerical evaluation to ensure that anatomical
structures are preserved after harmonization. Images for each subject were
aligned using Advanced Normalization Tools (ANTS)[28].

Note that the training dataset and the three testing datasets are mutually
exclusive with no overlapping samples. Considering both modalities and all
three vendors, more than 6000 images were used for evaluation. 60 central
axial slices were extracted from each image volume for training and testing.

###  0.2 MURD Harmonizes Contrasts and Preserves Details

We demonstrate the effectiveness of MURD harmonization on T1- and T2-weighted
images (Figure 2). Note that MURD allows harmonization with respect to either
a site or a reference image. The former amounts to harmonization with respect
to an image randomly drawn from a site image distribution. MURD is remarkably
effective in adapting contrast and preserving details when harmonizing images
from a source site with a target site (Figures 2a and b). When the source and
target sites are identical, MURD retain both contrast and details. When given
a reference image, MURD harmonizes the contrasts of images from a source site
with the reference image but preserves the anatomical details of the original
images (Figures 2c and d).

###  0.3 MURD Outperforms State-of-the-Art Methods

To further demonstrate the effectiveness and superiority of MURD, we compared
it with two closely-related state-of-the-art unsupervised methods, i.e.,
DRIT++[26] and StarGAN-v2[25], which are designed respectively for dual-domain
and multi-domain image-to-image translation. DRIT++ and StarGAN-v2 were
implemented and trained as described in the original papers[26, 25]. DRIT++
was trained once for every pair of sites. StarGAN-v2 was trained concurrently
for multiple sites, similar to MURD. We quantitatively compared the visual
quality of the harmonized images using two metrics, i.e., the Frechét
inception distance (FID)[29] and the kernel inception distance (KID)[30],
which reflect distribution discrepancy between two sets of images in a manner
that correlates well with human judgment[31]. FID and KID are respectively
computed based on the Frechét distance and maximum mean discrepancy (MMD) of
features from the last average pooling layer of the Inception-V3[32] trained
on the ImageNet[33]. FID and KID were computed at the slice level for the
harmonized images with respect to the training images of the target sites. For
site-specific harmonization, 10 target-site harmonized images were generated
for each testing image of the source site with 10 randomly generated style
codes. For reference-specific harmonization, each image in the source site was
harmonized with respect to 10 reference images randomly selected from the
testing images of a target site. The results (Figures 3a and b) indicate that
MURD outperforms DRIT++ and StarGAN-v2 for both FID and KID. For comparison,
reference FID and KID values were computed between the training and testing
images from the same site (denoted as “Reference”). The FID and KID values
given by MURD are significantly closer to the reference values than DRIT++ and
StarGAN-v2. MURD harmonization of the generalizability dataset (Figures 3c and
d) yields FID and KID values that are highly consistent with the validation
dataset and close to the reference values. This indicates that, although
trained using a modest dataset, the model is generalizable to a much larger
dataset.

###  0.4 MURD Efficacy Validated via Traveling Human Phantom Data

The human phantom dataset allows direct quantitative evaluation of the effects
of harmonization on consistency of structure and appearance. Based on multiple
metrics, including mean absolute error (MAE), multi-scale structural
similarity index (MS-SSIM), and peak signal-to-noise ratio (PSNR), MURD
significantly outperforms DRIT++ and StarGAN-v2, indicating better
harmonization of contrast and preservation of anatomical details (Figures 4a
and b).

###  0.5 MURD Improves Tissue Segmentation Consistency

Segmentation of brain tissues is sensitive to variation in image contrast and
under- or over-segmentation might happen owing to differences in acquisition
protocols. We applied Brain Extraction Tool (BET)[34] and FMRIB’s Automated
Segmentation Tool (FAST)[35] on T1- and T2-weighted images in the human
phantom dataset for brain extraction and tissue segmentation. Tissue
segmentation consistency before and after harmonization was measured using the
Dice similarity coefficient (DSC) with the tissue segmentation maps from the
target site as references. The results (Figures 4c and d) indicate that DSCs
are improved remarkably by harmonization using MURD.

###  0.6 MURD Supports Continuous and Incremental Harmonization

We further demonstrate that MURD completely disentangles content and style
information by visualization via continuous and incremental harmonization. We
generated images with between-site appearances to aid visual inspection of how
appearance changes gradually between sites and whether unwanted anatomical
alterations are introduced in the process. Intermediate style features are
calculated based on the style features of Site A, $s_{\text{A}}$, and Site B,
$s_{\text{B}}$, via $(1-\beta)s_{\text{A}}+\beta s_{\text{B}}$, where
$\beta\in[0,1]$. The intermediate style features and the content features of
an image are then used to generate an intermediate image. The results (Figures
5) indicate that MURD gradually and smoothly changes image appearance without
altering anatomical details.

###  0.7 Discussion

We have demonstrated that MURD is remarkably effective in harmonizing MR
images by removing non-biological site differences and at the same time
preserving anatomical details. MURD network training involves only site-
labeled images and requires no traveling human phantom data. This flexibility
allows data from existing large-scale studies to be harmonized retrospectively
without requiring additional data to be collected.

We have shown that MURD yields superior performance over DRIT++[26] and
StarGAN-v2[25]. For every pair of sites, DRIT++ embeds images in a site-
invariant content space capturing information shared across sites and a site-
specific style space. The encoded content features extracted from an image of
one site are combined with style features from another site to synthesize the
corresponding harmonized image. Learning is unsupervised and hence paired data
is not required. However, DRIT++ is not scalable due to the need to learn all
mappings for all site pairs. DRIT++ is also ineffective because it cannot
fully utilize the entire training data and can only learn from two sites at
each time, causing it to miss global features that can be learned from images
of all sites. Failure to fully utilize training data likely limits the quality
of generated images. Unlike DRIT++, StarGAN-v2[25] is scalable and performs
image-to-image translations for multiple sites using only a single model. It
has been applied to the problem of MRI harmonization[36] with promising
results. In addition to greater scalability, StarGAN-v2 generates images of
higher visual quality owing to its ability to jointly consider the information
offered by images from all sites. StarGAN-v2, however, does not explicitly
disentangle images into structural and appearance information. This introduces
the possibility of altering anatomical details during harmonization via style
transfer. In contrast, MURD enforces explicit disentanglement of content and
style features by jointly considering images from all sites, allowing it to
produce harmonized images with diverse appearances with significantly better
preservation of anatomical details (see Supplementary Figures 1–3).
Disentanglement safeguards harmonization against altering image anatomical
contents and allows gradual and controllable harmonization via interpolation
of style features.

The harmonization target is specified for DRIT++ and StarGAN-v2 by a reference
image. The appearance of the harmonized image is determined by the style
features extracted from the reference image. In addition to a reference image,
the harmonization target for MURD can be specified by a site label, which
determines the output branch of the style generator and the style encoder. A
latent code sampled from the standard Gaussian distribution determines an
appearance specific to the site.

A recent MRI harmonization method, called CALAMITI[37], relies on intra-site
supervised image-to-image translation and unsupervised domain adaptation for
multi-site harmonization. This requires training a disentangled representation
model with intra-site multi-contrast images (T1- and T2-weighted images) of
the same subjects and retraining the model for a new site via domain
adaptation[38]. Unlike CALAMITI, MURD requires only images from a single
contrast and can learn multi-site harmonization simultaneously without needing
fine-tuning or retraining.

We consider the multi-site harmonization problem as image-to-image translation
among multiple sites and propose an end-to-end multi-site unsupervised
representation disentangler (MURD, see Figure 1) to learn content-style
disentangled cycle translation mappings that translate images forward and
backward between any two sites. We employ two encoders to respectively embed
each image in a site-invariant content space, which captures anatomical
information, and a site-specific style space, which captures appearance
information, and a generator to produce a harmonized image using the encoded
content features and site-specific style features.

###  0.8 Content-Style Disentangled Cycle Translation (CS-DCT)

Given image sets $\\{\mathcal{X}_{i}\\}_{i=1}^{N}$ from $N$ distinct sites,
MURD utilizes content-style disentangled cycle translation (CS-DCT) to
disentangle each image set $\mathcal{X}_{i}$ in a site-invariant content space
$\mathcal{C}$ and a site-specific style space $\mathcal{S}_{i}$ (Figure 1a).
CS-DCT, realized with sequential forward and backward translation, can be
site-specific (Figure 1b) or reference-specific (Figure 1c). MURD jointly
learns site-invariant content features $c_{i}\in\mathcal{C}$ and site-specific
style features $s_{i}\in\mathcal{S}_{i}$ from image $x_{i}\in\mathcal{X}_{i}$,
and utilize generator $G$ to construct the harmonized image
$\tilde{x}_{j}\in\mathcal{X}_{j}$ ($j\neq i$) using content features $c_{i}$
and style features $s_{j}\in\mathcal{S}_{j}$ in forward translation. Style
features $s_{j}$ are generated by a style generator $G^{\text{S}}_{j}$ or
extracted from a reference image $x_{j}\in\mathcal{X}_{j}$. In backward
translation, MURD extracts the content features $\tilde{c}_{j}$ from the
harmonized image $\tilde{x}_{j}$, which are in turn fed with style features
$s_{i}$ to generator $G$ to reconstruct image $\hat{x}_{i}$, which is required
to be consistent with the input image $x_{i}$.

###  0.9 Multi-site Unsupervised Representation Disentangler (MURD)

MURD implements CS-DCT in a end-to-end manner via five modules (Figure 1d):

  1. (i)

A content encoder $E^{\text{C}}$, shared for all sites, to extract content
features from an image in a site-invariant space $\mathcal{C}$;

  2. (ii)

A style encoder $E^{\text{S}}_{i}$ for each site $i$ to extract style features
from an image in a site-specific style space $\mathcal{S}_{i}$;

  3. (iii)

A generator $G$, shared for all sites, to synthesize images using content and
style features;

  4. (iv)

A style generator $G^{\text{S}}_{i}$ for each site $i$ to yield style features
$s_{i}$ that reflect the appearance style of images from the site; and

  5. (v)

A discriminator $D_{i}$ for each site $i$ to distinguish between real and
generated images.

Specifically, given an input image $x_{i}$ from image set $\mathcal{X}_{i}$,
content encoder $E^{\text{C}}$ and style encoder $E^{\text{S}}_{i}$,
respectively, extract content features $c_{i}$ and style features $s_{i}$. For
a random site $j\neq i$, style generator $G^{\text{S}}_{j}$ takes a latent
code $z$ randomly sampled from a standard normal distribution
$\mathcal{N}(0,1)$ as input to create a $j$-site style features $s_{j}$. With
content features $c_{i}$ and style features $s_{j}$, generator $G$ constructs
harmonized image $\tilde{x}_{j}$, which the discriminator $D_{j}$ then
classifies as being either real or fake using an adversarial loss
$\mathcal{L}_{\textrm{adv}}$:

$\mathcal{L}_{{}_{\textrm{adv}}}=\mathbb{E}[\log
D_{i}(x_{i})]+\mathbb{E}[\log(1-D_{j}(\tilde{x}_{j}))].$ (1)

Content encoder $E^{\text{C}}$ and style encoder $E^{\text{S}}_{j}$ are used
to extract content features $\tilde{c}_{j}$ and style features $\tilde{s}_{j}$
from $\tilde{x}_{j}$. The consistency between $c_{i}$ and $\tilde{c}_{j}$ is
enforced by a pixel-wise content consistency loss $\mathcal{L}_{\text{cont}}$:

$\mathcal{L}_{\text{cont}}=\|c_{i}-\tilde{c}_{j}\|_{1}.$ (2)

The consistency between $s_{j}$ and $\tilde{s}_{j}$ is enforced by a pixel-
wise style consistency loss $\mathcal{L}_{\text{sty}}$:

$\mathcal{L}_{\text{sty}}=\|s_{i}-\tilde{s}_{j}\|_{1}.$ (3)

Content site-invariance is enforced by a content alignment loss
$\mathcal{L}_{\text{ca}}$:

$\mathcal{L}_{\text{ca}}=\text{KL}(\mathcal{N}(c_{i},I)\|\mathcal{N}(0,I)),$
(4)

where $\text{KL}(\cdot\|\cdot)$ is the Kullback–Leibler divergence and $I$ is
an identity matrix. Content features $c_{i}$ are randomly perturbed during the
feed-forward step:

$c_{i}=E^{\text{C}}(x_{i})+\eta,\quad\eta\sim\mathcal{N}(0,I).$ (5)

The reconstructed image $\hat{x}_{i}$ of $x_{i}$ is produced by generator $G$
using content features $\tilde{c}_{j}$ and style features $s_{i}$. The
consistency between $x_{i}$ and $\hat{x}_{i}$ is ensured by a pixel-wise and
gradient-wise cycle consistency loss $\mathcal{L}_{\text{cyc}}$:

$\mathcal{L}_{\text{cyc}}=\|x_{i}-\hat{x}_{i}\|_{1}+\lambda_{\text{g}}\|g(x_{i})-g(\hat{x}_{i})\|_{1},$
(6)

where $g(\cdot)$ is the image gradient function and $\lambda_{\text{g}}$ is
the loss weight for the gradient loss term. Furthermore, an identity image
$\bar{x}_{i}$ can also be constructed by generator $G$ using content features
$c_{i}$ and style features $s_{i}$, which are identical to $x_{i}$ when
$c_{i}$ and $s_{i}$ are completely disentangled. We devise an identity loss to
measure the pixel-wise difference between $x_{i}$ and $\bar{x}_{i}$ as

$\mathcal{L}_{\text{id}}=\|x_{i}-\bar{x}_{i}\|_{1}.$ (7)

All modules are optimized with total loss function $\mathcal{L}$:

$\min_{E^{\text{C}},E^{\text{S}}_{i},G,G^{\text{S}}_{i}}\max_{D_{i}}\mathcal{L}_{\text{adv}}+\lambda_{\text{cont}}\mathcal{L}_{\text{cont}}+\lambda_{ca}\mathcal{L}_{\text{ca}}+\lambda_{\text{sty}}\mathcal{L}_{\text{sty}}+\lambda_{\text{cyc}}\mathcal{L}_{\text{cyc}}+\lambda_{\text{id}}\mathcal{L}_{\text{id}},\quad
i\in[1,N],$ (8)

where $\lambda_{\text{cont}}$, $\lambda_{\text{ca}}$, $\lambda_{\text{sty}}$,
$\lambda_{\text{cyc}}$, and $\lambda_{\text{id}}$ are loss weights used for
controlling the contributions of the respectively terms in the loss function.
Training ends when all modules are optimized, such that the optimized
discriminator classifies the harmonized images into one of two categories with
equal probability. See Supplementary Figures 4–7 for the effects of the
individual loss functions.

During inference, content features $c_{i}$ of input image $x_{i}$ are
extracted using content encoder $E^{\text{C}}$, and style features $s_{j}$ are
either generated using style generator $G^{\text{S}}_{j}$ or extracted from a
reference image $x_{j}$. Harmonized image $\tilde{x}_{j}$ is created by
generator $G$ using $c_{i}$ and $s_{j}$.

###  0.10 Network Architecture

The architectures of the components of MURD are described next.

Content Encoder Content encoder $E^{\text{C}}$ is shared among all sites and
extracts the content features $c_{i}$ of an input image $x_{i}$ through three
convolutional blocks and 4 residual blocks. Each convolutional block is
composed of three sequential layers, i.e., convolution, instance normalization
(IN)[39], and leaky ReLU (LReLU) activation. Each residual block consists of a
nonlinear mapping branch and a shortcut branch. The nonlinear mapping branch
is constructed by a series of layers, i.e., convolution, IN, LReLU,
convolution, and IN. The shortcut branch is an identity mapping of the block
input. We use an IN layer instead of a batch normalization layer[39] to
accelerate model convergence and maintain independence between features. All
normalized feature maps are activated by LReLU with a negative slope of 0.2.

Style Encoder Style encoder $E_{i}^{\text{S}}$ is composed of site-shared and
site-specific subnetworks. The site-shared subnetwork is constructed by a
convolution layer, four pre-activation residual blocks, and a global average
pooling layer. The site-specific subnetwork is composed of $N$ fully connected
layers corresponding to the $N$ individual sites. The pre-activation residual
block is constructed by integrating LReLU activation followed by a convolution
layer with unit stride into a residual block, where an average pooling layer
is adopted to downsample the intermediate features and the shortcut branch is
implemented by an average pooling layer and a convolution layer with unit
kernel size and stride. Note that we extract style features without IN layers
since IN removes feature means and variances, which contain important style
information. The output dimension is set to 64. Style features $s_{i}$ have
the same dimension.

Generator Site-shared generator $G$ merges content features $c_{i}$ and style
features $s_{j}$ to create a harmonized image $\tilde{x}_{j}$ using four
residual blocks identical to the content encoder, two upsampling blocks and a
convolution layer with hyperbolic tangent (tanh) activation. The upsampling
block consists of deconvolution, IN, and LReLU activation.

Style Generator Style generator $G^{\text{S}}_{j}$ consists of a multilayer
perception (MLP) with $N$ output branches. Six fully connected layers are
shared among all sites, followed by a fully connected layer for each site. We
set the dimensions of the latent code, the hidden layer, and the style
features to 16, 256, and 64, respectively. We randomly sample the latent code
$z$ from the standard Gaussian distribution.

Discriminator Discriminator $D_{j}$ consists of site-shared and site-specific
subnetworks, similar to the style encoder. Specifically, three convolutional
blocks and a global average pooling are shared among all sites, followed by a
specific fully connected layer for real/fake classification for each site.

###  0.11 Implementation Details

MURD was implemented using Tensorflow. Evaluation was based on a machine with
a CPU (i.e., Intel i7-8700K) and a GPU (i.e., NVIDIA GeForce GTX 1080Ti 11GB).
The ADAM optimizer with $1\times 10^{-4}$ learning rate was utilized for
minimization based on the loss function. For all datasets, we used
$\lambda_{\text{cont}}=10$, $\lambda_{\text{ca}}=0.01$,
$\lambda_{\text{sty}}=10$, $\lambda_{\text{cyc}}=10$,
$\lambda_{\text{g}}=0.1$, and $\lambda_{\text{id}}=10$ for the corresponding
losses. For all datasets, every three adjacent slices in each volume were
inserted into three channels. Each channel was then normalized to have a range
between -1 and 1 and zero-padded to 256$\times$256.

This work was supported in part by the United States National Institutes of
Health (NIH) grants EB006733 and MH125479.

The authors declare no competing interests.

Correspondence and requests for materials should be addressed to Pew-Thian Yap
(email: ptyap@med.unc.edu).

The Adolescent Brain Cognitive Development (ABCD) 2.0 data used in this study
can be obtained from the National Institutes of Mental Health Data Archive
(NDA) (http://nda.nih.gov/).

S.L. designed the framework and network architecture, carried out the
implementation, performed the experiments, and analyzed the data. S.L. and
P.-T.Y. wrote and revised the manuscript. P.-T.Y. conceived the study and were
in charge of overall direction and planning.

###  0.12 References

## References

  * [1] Jernigan, T. L., Brown, S. A. & Dowling, G. J.  The adolescent brain cognitive development study.  _Journal of Research on Adolescence_ 28, 154–156 (2018).
  * [2] Mueller, S. G. _et al._ The Alzheimer’s disease neuroimaging initiative.  _Neuroimaging Clinics of North America_ 15, 869–877 (2005).
  * [3] Ellis, K. A. _et al._ The Australian Imaging, Biomarkers and Lifestyle (AIBL) study of aging: Methodology and baseline characteristics of 1112 individuals recruited for a longitudinal study of Alzheimer’s disease.  _International Psychogeriatrics_ 21, 672–687 (2009).
  * [4] Shinohara, R. _et al._ Volumetric analysis from a harmonized multisite brain MRI study of a single subject with multiple sclerosis.  _American Journal of Neuroradiology_ 38, 1501–1509 (2017).
  * [5] Pomponio, R. _et al._ Harmonization of large MRI datasets for the analysis of brain imaging patterns throughout the lifespan.  _NeuroImage_ 208, 116450 (2020).
  * [6] Yu, M. _et al._ Statistical harmonization corrects site effects in functional connectivity measurements from multi-site fMRI data.  _Human Brain Mapping_ 39, 4213–4227 (2018).
  * [7] Shah, M. _et al._ Evaluating intensity normalization on MRIs of human brain with multiple sclerosis.  _Medical Image Analysis_ 15, 267–282 (2011).
  * [8] Udupa, L. G. N. J. K.  On standardizing the mr image intensity scale.  _Magenetic Resonance in Medicine_ 42, 1072–1081 (1999).
  * [9] He, Q., Shiee, N., Reich, D. S., Calabresi, P. A. & Pham, D. L.  Intensity standardization of longitudinal images using 4d clustering.  In _Proceedings of International Symposium on Biomedical Imaging_ (2013).
  * [10] Shinohara, R. T. _et al._ Statistical normalization techniques for magnetic resonance imaging.  _NeuroImage: Clinical_ 6, 9–19 (2014).
  * [11] Wrobel, J. _et al._ Intensity warping for multisite MRI harmonization.  _NeuroImage_ 223, 117242 (2020).
  * [12] Fortin, J.-P., Sweeney, E. M., Muschelli, J., Crainiceanu, C. M. & Shinohara, R. T.  Removing inter-subject technical variability in magnetic resonance imaging studies.  _NeuroImage_ 132, 198–212 (2016).
  * [13] Fortin, J.-P. _et al._ Harmonization of multi-site diffusion tensor imaging data.  _NeuroImage_ 161, 149–170 (2017).
  * [14] Fortin, J.-P. _et al._ Harmonization of cortical thickness measurements across scanners and sites.  _NeuroImage_ 167, 104–120 (2018).
  * [15] Jog, A., Carass, A., Roy, S., Pham, D. L. & Prince, J. L.  Random forest regression for magnetic resonance image synthesis.  _Medical Image Analysis_ 35, 475–488 (2017).
  * [16] Garcia-Dias, R. _et al._ Neuroharmony: A new tool for harmonizing volumetric MRI data from unseen scanners.  _NeuroImage_ 220, 117127 (2020).
  * [17] Dewey, B. E. _et al._ DeepHarmony: A deep learning approach to contrast harmonization across scanner changes.  _Magnetic Resonance Imaging_ 64, 160–170 (2019).
  * [18] Dewey, B. E. _et al._ A disentangled latent space for cross-site MRI harmonization.  In _Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)_ , 720–729 (Springer International Publishing, 2020).
  * [19] Dinsdale, N. K., Jenkinson, M. & Namburete, A. I.  Deep learning-based unlearning of dataset bias for MRI harmonisation and confound removal.  _NeuroImage_ 228, 117689 (2021).
  * [20] Liu, M.-Y., Breuel, T. & Kautz, J.  Unsupervised image-to-image translation networks.  In _Proceedings of Neural Information Processing Systems (NeurIPS)_ , 700–708 (2017).
  * [21] Zhu, J.-Y. _et al._ Toward multimodal image-to-image translation.  In _Proceedings of Neural Information Processing Systems (NeurIPS)_ , 465–476 (2017).
  * [22] Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 1125–1134 (2017).
  * [23] Anoosheh, A., Agustsson, E., Timofte, R. & Van Gool, L.  Combogan: Unrestrained scalability for image domain translation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ (2018).
  * [24] Choi, Y. _et al._ StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2018).
  * [25] Choi, Y., Uh, Y., Yoo, J. & Ha, J.-W.  StarGAN v2: Diverse image synthesis for multiple domains.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ (2020).
  * [26] Lee, H.-Y. _et al._ DRIT++: Diverse image-to-image translation via disentangled representations.  _International Journal of Computer Vision_ 128, 2402–2417 (2020).
  * [27] Casey, B. _et al._ The adolescent brain cognitive development (ABCD) study: Imaging acquisition across 21 sites.  _Developmental Cognitive Neuroscience_ 32, 43–54 (2018).
  * [28] Avants B., S. G., Tustison N.  Advanced normalization tools.  _Insight Journal_ (2009).
  * [29] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B. & Hochreiter., S.  GANs trained by a two time-scale update rule converge to a local nash equilibrium.  In _Proceedings of Neural Information Processing Systems (NeurIPS)_ (2017).
  * [30] Bińkowski, M., Sutherland, D., Arbel, M. & Gretton, A.  Demystifying MMD GANs.  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 1–36 (2018).
  * [31] Salimans, T. _et al._ Improved techniques for training GANs.  In _Advances in Neural Information Processing Systems_ , vol. 29, 2234–2242 (2016).
  * [32] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z.  Rethinking the Inception architecture for computer vision.  In _Proceedings of Computer Vision and Pattern Recognition (CVPR)_ (IEEE, 2016).
  * [33] Deng, J. _et al._ ImageNet: A large-scale hierarchical image database.  In _Computer Vision and Pattern Recogntion (CVPR)_ (2009).
  * [34] Smith, S. M.  Fast robust automated brain extraction.  _Human Brain Mapping_ 17, 143–155 (2002).
  * [35] Zhang, Y., Brady, M. & Smith, S.  Segmentation of brain MR images through a hidden markov random field model and the expectation-maximization algorithm.  _IEEE Transactions on Medical Imaging_ 20, 45–57 (2001).
  * [36] Liu, M. _et al._ Style transfer using generative adversarial networks for multi-site MRI harmonization.  _bioRxiv_ (2021).
  * [37] Zuo, L. _et al._ Information-based disentangled representation learning for unsupervised MR harmonization.  In _International Conference on Information Processing in Medical Imaging (IPMI)_ (2021).
  * [38] Varsavsky, T. _et al._ Test-time unsupervised domain adaptation.  In _Proceedings of Medical Image Computing and Computer Assisted Intervention (MICCAI)_ , 428–436 (2020).
  * [39] Ulyanov, D., Vedaldi, A. & Lempitsky, V.  Improved texture networks: maximizing quality and diversity in feed-forward stylization and texture synthesis.  In _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 6924–6932 (2017).


# Higher dimensional charged $f(R)$ black holes

Ahmad Sheykhi<EMAIL_ADDRESS>Center for Excellence in Astronomy and
Astrophysics (CEAA-RIAAM) Maragha, P. O. Box 55134-441, Iran
Physics Department and Biruni Observatory, Shiraz University, Shiraz 71454,
Iran

###### Abstract

We construct a new class of higher dimensional black hole solutions of $f(R)$
theory coupled to a nonlinear Maxwell field. In deriving these solutions the
traceless property of the energy-momentum tensor of the matter filed plays a
crucial role. In $n$-dimensional spacetime the energy-momentum tensor of
conformally invariant Maxwell field is traceless provided we take $n=4p$,
where $p$ is the power of conformally invariant Maxwell lagrangian. These
black hole solutions are similar to higher dimensional Reissner-Nordstrom AdS
black holes but only exist for dimensions which are multiples of four. We
calculate the conserved and thermodynamic quantities of these black holes and
check the validity of the first law of black hole thermodynamics by computing
a Smarr-type formula for the total mass of the solutions. Finally, we study
the local stability of the solutions and find that there is indeed a phase
transition for higher dimensional $f(R)$ black holes with conformally
invariant Maxwell source.

PACS numbers: 04.70. Bw, 04.70. Dy.

##  I Introduction

A large volume of observational evidences indicate that our universe is now
experiencing a phase of accelerated expansion. There are two main approaches
for explanation of this cosmic acceleration. The first approach which try to
explain the problem in the framework of general relativity, requires the
existence of a strange type of energy called “dark energy” whose gravity is
repulsive and consist an un-clustered component through the universe. The
second approach is to modify general relativity by adding higher powers of the
scalar curvature $R$, the Riemann and Ricci tensors, or their derivatives in
the lagrangian formulation. Among the latter attempts are Lovelock gravity,
braneworld cosmology, scalar-tensor theories like Brans-Dicke one and also
$f(R)$ theories. A fairly comprehensive review on dark energy models can be
seen in Cop . $f(R)$ theories were proved to be able to mimic the whole
cosmological history, from inflation to the actual accelerated expansion era
Odin ; Capo  (see also Feli  for a recent review on $f(R)$ theories). Diverse
applications of $f(R)$ theories on gravitation and cosmology have been also
widely studied Feli , as well as multiple ways to observationally and
experimentally distinguish them from general relativity.

When one considers $f(R)$ theory as a modification of general relativity, it
is quite natural to ask about black hole existence and its features in this
theory. It is expected that some signatures of black holes in $f(R)$ theory
may differ from the expected physical results in Einstein’s gravity.
Therefore, the investigation on $f(R)$ black holes is of particular interest.
Some attempts have been done to construct black hole solutions in $f(R)$
theories (see Cong ; Seb ; Ade ; Hendi1  and references therein). In general,
in the presence of a matter field, the field equations of $f(R)$ gravity are
complicated and it is not easy to find exact analytical solutions. However, if
one considers the traceless energy-momentum tensor for the matter field, one
can extract exact analytical solutions from $R+f(R)$ theory coupled to a
matter field Tae . Since the energy-momentum tensor of Maxwell and Yang-Mills
fields are traceless only in four dimensions, therefore black hole solutions
from $R+f(R)$ theory coupled to the matter field were derived only in four
dimensions Tae . The studies were also generalized to the four dimensional
charged rotating black holes Alex , charged rotating black string shey1  and
magnetic string solutions shey2  in $R+f(R)$-Maxwell theory. However, since
the standard Maxwell energy-momentum tensor is not traceless in higher
dimensions, they failed to derive higher dimensional black hole/string
solutions from $R+f(R)$ gravity coupled to standard Maxwell field. A natural
question then arises: Is there an extension of Maxwell action in arbitrary
dimensions that is traceless and hence possesses the conformal invariance? The
answer is positive and the conformally invariant Maxwell action was presented
as Hass1 ,

$\displaystyle S_{m}=-\int{d^{n}x\sqrt{-g}(F_{\mu\nu}F^{\mu\nu})^{p}},$ (1)

where $p$ is a positive integer, i.e., $p\in\mathbb{N}$. The associated
energy-momentum tensor of the above conformally invariant Maxwell action is
given by

$T_{\mu\nu}=2\left(pF_{\mu\eta}F_{\nu}^{\text{
}\eta}F^{p-1}-\frac{1}{4}g_{\mu\nu}F^{p}\right),$ (2)

where $F=F_{\alpha\beta}F^{\alpha\beta}$ is the Maxwell invariant. One can
easily check that the above energy momentum tensor is traceless for $n=4p$.
The theory of conformally invariant Maxwell field is considerably richer than
that of the linear standard Maxwell field and in the special case $(p=1)$ it
recovers the Maxwell action. It is worthwhile to investigate the effects of
exponent $p$ on the behavior of the solutions and the laws of black hole
mechanics. The motivation is to take advantage of the conformal symmetry to
construct the analogues of the four dimensional Reissner-Nordstrom (RN)
solutions, in higher dimensions. Recently, the studies on the black object
solutions with a nonlinear Maxwell source in Einstein Hass1 ; Hass2 ; Hendi2
and Gauss-Bonnet Mis  gravity have got a lot of attentions.

In this work we would like to extend the investigation on the conformally
invariant Maxwell field to $f(R)$ gravity. We will consider the action (1) as
the matter source of the field equations in $R+f(R)$ theory with constant
curvature scalar. Our purpose is to find the analogues of the four-dimensional
charged black hole solutions of $R+f(R)$-Maxwell theory Tae  in higher
dimensional spacetime. In contrast to the higher dimensional black holes of
Einstein gravity with a conformally invariant Maxwell source presented in
Hass1  which has vanishing scalar curvature $R=0$, the spacetime we construct
here in $R+f(R)$ gravity coupled to a nonlinear Maxwell field has a constant
scalar curvature $R=R_{0}$. Our solutions also differ from higher dimensional
RN solutions in that the electric charge term in the metric coefficient goes
as $r^{-(n-2)}$ while in the standard RN case is $r^{-2(n-3)}$. Also, the
electric field in higher dimensions does not depend on $n$ and goes as
electric field in four dimensions.

This paper is outlined as follows. In Sec. II, we construct exact spherically
symmetric black hole solutions of $R+f(R)$ theory coupled to a nonlinear
Maxwell field in $n=4p$ dimensions and investigate their properties. In Sec.
III, we obtain the conserved and thermodynamic quantities of the solutions and
verify the validity of the first law of black hole thermodynamics. We also
study local stability of the solutions in this section. We summarize our
results in Sec. IV.

##  II Field Equations and solutions

We consider the action of $R+f(R)$ gravity in $n$-dimensional spacetime
coupled to a conformally invariant Maxwell field

$\displaystyle
S=\int_{\mathcal{M}}d^{n}x\sqrt{-g}\left[R+f(R)-(F_{\mu\nu}F^{\mu\nu})^{p}\
\right],$ (3)

where ${R}$ is the scalar curvature, $f(R)$ is an arbitrary function of scalar
curvature, $F_{\mu\nu}=\partial_{\mu}A_{\nu}-\partial_{\nu}A_{\mu}$ is the
electromagnetic field tensor and $A_{\mu}$ is the electromagnetic potential.
The equations of motion can be obtained by varying action (3) with respect to
the gravitational field $g_{\mu\nu}$ and the gauge field $A_{\mu}$,

${R}_{\mu\nu}\left(1+f^{\prime}(R)\right)-\frac{1}{2}g_{\mu\nu}(R+f(R))+\left(g_{\mu\nu}\nabla^{2}-\nabla_{\mu}\nabla_{\nu}\right)f^{\prime}(R)=T_{\mu\nu},$
(4) $\partial_{\mu}(\sqrt{-g}F^{\mu\nu}F^{p-1})=0,$ (5)

where $F=F_{\alpha\beta}F^{\alpha\beta}$ is the Maxwell invariant and the
“prime” denotes differentiation with respect to $R$. In order to obtain the
constant curvature black hole solution in $f(R)$ gravity theory coupled to a
matter field, the trace of stress-energy tensor $T_{\mu\nu}$ should be zero
Tae . Hence, two candidates for the matter field in four dimensions are
Maxwell and Yang-Mills fields. Since the assumption of traceless energy-
momentum tensor is essential for deriving exact black hole solutions in $f(R)$
gravity coupled to the matter field, therefore the solutions exist only for
$n=4p$ dimensions. Assuming the constant scalar curvature $R=R_{0}=const.$,
then the trace of Eq. (4) yields

${R}_{0}\left(1+f^{\prime}(R_{0})\right)-\frac{n}{2}(R_{0}+f(R_{0}))=0.$ (6)

Solving the above equation for $R_{0}$, gives

${R}_{0}=\frac{nf(R_{0})}{2f^{\prime}(R_{0})+2-n}\equiv\frac{2n}{n-2}{\Lambda_{\rm
f}}<0.$ (7)

Substituting the above relation into Eq. (4), we obtain the following equation
for Ricci tensor

${R}_{\mu\nu}(1+f^{\prime}(R_{0}))-\frac{g_{\mu\nu}}{n}R_{0}\left(1+f^{\prime}(R_{0})\right)=T_{\mu\nu}.$
(8)

We are looking for the $n$-dimensional static spherically symmetric solutions.
Motivated by the metric of higher dimensional charged black holes in Einstein
gravity, we assume the metric has the following form

$d{s}^{2}=-N(r)dt^{2}+\frac{dr^{2}}{N(r)}+{r^{2}}d\Omega_{n-2}^{2},$ (9)

where $d\Omega_{n-2}^{2}$ denotes the metric of an unit $(n-2)$-sphere and
$N(r)$ is a functions of $r$ which should be determined. We are seeking for a
purely radial electric solution which means that the only non-vanishing
component of the Maxwell tensor is $F_{tr}$. In this case, the Maxwell
equations (5) can be integrated immediately, where, for the spherically
symmetric spacetime (9), all the components of ${F}_{\mu\nu}$ are zero except
${F}_{tr}$:

${F}_{tr}=\frac{q}{r^{\frac{n-2}{2p-1}}},$ (10)

where $q$ is an integration constant. Substituting $n=4p$ in the above
relation, the Maxwell field becomes

${F}_{tr}=\frac{q}{r^{2}}.$ (11)

It is important to note that the electric field in higher dimensions does not
depend on $n$ and its value coincides with the RN solution in four dimensions.
Using metric (9) and the Maxwell field (11), one can show that Eq. (8) has a
solution of the form

$N(r)=1-\frac{2m}{r^{n-3}}+\frac{q^{2}}{r^{n-2}}\times\frac{(-2q^{2})^{(n-4)/4}}{\left(1+f^{\prime}(R_{0})\right)}-\frac{R_{0}}{n(n-1)}{r}^{2},$
(12)

where $m$ is an integration constant which is related to the mass of the
solution. In four dimension ($n=4$) the solution recovers the result of Tae .
In order to have a real solution we should restrict ourself to the dimensions
which are multiples of four, i.e., $n=4,8,12,....$, which means that $p$
should be only positive integer, as we mentioned already.

Next we study the physical properties of the solutions. To do this, we first
look for the curvature singularities. A simple calculation shows that the
Kretschmann scalar $R_{\mu\nu\lambda\kappa}R^{\mu\nu\lambda\kappa}$ diverges
at $r=0$, it is finite for $r\neq 0$ and is proportional to $R_{0}^{2}$ as
$r\rightarrow\infty$. Therefore, there is a curvature singularity located at
$r=0$. As one can see from Eq. (12), the solution is ill-defined for
$f^{\prime}(R_{0})=-1$. Let us consider the cases with $f^{\prime}(R_{0})>-1$
and $f^{\prime}(R_{0})<-1$ separately. In the first case where
$f^{\prime}(R_{0})>-1$, we have a black hole solution (see Fig. 1). Indeed,
for $1+f^{\prime}(R_{0})>0$ and $R_{0}<0$ black hole can have two inner and
outer horizons, an extreme black hole or a naked singularity provided the
parameters of the solutions are chosen suitably (see Fig. 2). In the latter
case where $f^{\prime}(R_{0})<-1$, we encounter a cosmological horizon for
$R_{0}<0$. Indeed, in this case the signature of the spacetime changes and the
conserved quantities such as mass become negative, as we will see in the next
section, thus this is not a physical case and we rule it out from our
consideration.

It is apparent that the spacetime described by solution (12) is asymptotically
AdS provided we define $R_{0}=-n(n-1)/l^{2}$. However, the solution presented
here differ from the standard higher dimensional Reissner-Nordstrom AdS
(RNAdS) solutions since the electric charge term in the metric coefficient
goes as $r^{-(n-2)}$ while in the standard RNAdS case is $r^{-2(n-3)}$. In
four dimensions, with the following replacement

$\displaystyle\frac{\alpha q^{2}}{\left(1+f^{\prime}(R_{0})\right)}\rightarrow
Q^{2}$ (13) $\displaystyle{R_{0}}\rightarrow 4\Lambda$ (14)

the solution reduces to standard RNAdS black holes for $\Lambda=-3/l^{2}$.

Figure 1: The function $N(r)$ versus $r$ for $m=2$, $n=4$, $q=1$ and
$R_{0}=-12$. $f^{\prime}(R_{0})=-0.5$ (bold line) and $f^{\prime}(R_{0})=-2$
(dashed line).

Figure 2: The function $N(r)$ versus $r$ for $m=2$, $q=1$, $n=4$ and
$R_{0}=-12$. $f^{\prime}(R_{0})=-0.2$ (bold line), $f^{\prime}(R_{0})=-0.54$
(continuous line) and $f^{\prime}(R_{0})=-0.8$ (dashed line).

We use the subtraction method of Brown and York (BY) BY  to calculate the
quasilocal mass of the charged $f(R)$ black hole. Such a procedure causes the
resulting physical quantities to depend on the choice of reference background.
In order to use the BY method one should write the metric in the following
form

$ds^{2}=-W(r)dt^{2}+\frac{d{r}^{2}}{V({r})}+{r}^{2}d\Omega_{n-2}^{2}.$ (15)

Since metric (9) has the above form, it is sufficient to choose the background
metric to be the metric (15) with

$W_{0}({r})=V_{0}({r})=N_{0}(r)=1+\frac{r^{2}}{l^{2}}$ (16)

Where we have defined $R_{0}=-n(n-1)/l^{2}$ which show that the solutions are
asymptotically AdS as we mentioned. It is well-known that the Ricci scalar for
AdS spacetime should have this value (see e.g. Weinberg ). To compute the
conserved mass of the spacetime, we choose a timelike Killing vector field
$\xi$ on the boundary surface ${\cal B}$ of the spacetime (15). Then the
quasilocal conserved mass can be written as

${\cal M}=\frac{1}{8\pi}\int_{{\cal
B}}d^{n-2}x\sqrt{\sigma}\left\\{\left(K_{ab}-Kh_{ab}\right)-\left(K_{ab}^{0}-K^{0}h_{ab}^{0}\right)\right\\}n^{a}\xi^{b},$
(17)

where $\sigma$ is the determinant of the metric of the boundary ${\cal B}$,
$K_{ab}^{0}$ is the extrinsic curvature of the background metric and $n^{a}$
is the timelike unit normal vector to the boundary ${\cal B}$. In the context
of counterterm method, the limit in which the boundary ${\cal B}$ becomes
infinite (${\cal B}_{\infty}$) is taken, and the counterterm prescription
ensures that the action and conserved charges are finite. Thus, we obtain the
mass through the use of the above subtraction method of BY as

${M}=\frac{(n-2)\Omega_{n-2}}{8\pi}m\left[1+f^{\prime}(R_{0})\right],$ (18)

where $\Omega_{n-2}$ is the volume of the unit $(n-2)$-sphere. In the limiting
case ($f^{\prime}(R_{0})=0$), this expression for the mass reduces to the mass
of the $n$-dimensional AdS black hole.

##  III Thermodynamics of Charged $f(R)$ Black holes

In this section we are going to explore thermodynamics of higher dimensional
charged $f(R)$ black holes. The Hawking temperature of the black holes can be
easily obtained by requiring the absence of conical singularity at the horizon
in the Euclidean sector of the black hole solutions. One obtains the
associated temperature with the outer event horizon $r=r_{+}$ as

$\displaystyle T$ $\displaystyle=$
$\displaystyle\frac{1}{4\pi}\left(\frac{dN(r)}{dr}\right)_{r=r_{+}}=\frac{[1+f^{\prime}(R_{0})]\left(2r^{2}(n-1)+2l^{2}(n-3)\right)+(-2q^{2})^{n/4}r^{2-n}l^{2}}{\pi
l^{2}r[1+f^{\prime}(R_{0})]}.$ (19)

where we have used equation $N(r_{+})=0$ for omitting the mass parameter $m$
from temperature expression. Next, we calculate the entropy of the black hole.
Let us first give a brief discussion regarding the entropy of the black hole
in $f(R)$ gravity. To this aim, we follow the arguments presented in Brevik .
If one use the Noether charge method for evaluating the entropy associated
with black hole solutions in $f(R)$ theory with constant curvature, one finds
Cong

${S}=\frac{A}{4G}f^{\prime}(R_{0}),$ (20)

where $A=4\pi r_{+}^{2}$ is the horizon area. As a result, in $f(R)$ gravity,
the entropy does not obey the area law and one obtains a modification of the
area law. Inspired by the above argument, for the $n$-dimensional charged
black hole solutions in $R+f(R)$ gravity, we find the entropy as

${S}=\frac{r_{+}^{n-2}\Omega_{n-2}}{4}\left[1+f^{\prime}(R_{0})\right].$ (21)

The charge of conformally invariant $f(R)$ black holes can be found by
calculating the flux of the electric field at infinity, yielding

${Q}=\frac{n(-2)^{(n-4)/4}q^{(n-2)/2}\Omega_{n-2}}{16\pi\sqrt{1+f^{\prime}(R_{0})}}.$
(22)

The electric potential $\Phi$, measured at infinity with respect to the
horizon, is defined by

$\Phi=A_{\mu}\chi^{\mu}\left|{}_{r\rightarrow\infty}-A_{\mu}\chi^{\mu}\right|_{r=r_{+}},$
(23)

where $\chi=\partial_{t}$ is the null generator of the horizon. We find

$\Phi=\frac{q}{r_{+}}\sqrt{1+f^{\prime}(R_{0})}.$ (24)

Then, we investigate the validity of the first law of thermodynamics for
higher dimensional charged $f(R)$ black hole. For this purpose, we obtain a
Smarr-type formula, namely the mass $M$ as a function of extensive quantities
$S$, and $Q$. Using the expression for the mass, the entropy and the charge
given in Eqs. (18), (21) and (22) and the fact that $N(r_{+})=0$, we find

$\displaystyle M(S,Q)$ $\displaystyle=$ $\displaystyle\frac{n-2}{16\pi
l^{2}}\left(\frac{4S}{1+f^{\prime}(R_{0})}\right)^{-1/(n-2)}\left[[1+f^{\prime}(R_{0})]\left(\frac{4S}{1+f^{\prime}(R_{0})}\right)\times\right.$
(25)
$\displaystyle\left.\left(l^{2}+\left(\frac{4S}{1+f^{\prime}(R_{0})}\right)^{2/(n-2)}\right)+l^{2}q^{2}(-2q^{2})^{(n-4)/4}\right].$

where $q$ is a function of $Q$ according to Eq. (22). One may then regard the
parameters $S$, and $Q$ as a complete set of extensive parameters for the mass
$M(S,Q)$ and define the intensive parameters conjugate to $S$ and $Q$. These
quantities are the temperature and the electric potential

$T=\left(\frac{\partial M}{\partial S}\right)_{Q},\ \
\Phi=\left(\frac{\partial M}{\partial Q}\right)_{S}.$ (26)

Numerical calculations show that the intensive quantities calculated by Eq.
(26) coincide with Eqs. (19) and (24). Thus, the thermodynamics quantities we
obtained in this section satisfy the first law of black hole thermodynamics

$dM=TdS+\Phi d{Q}.$ (27)

In this way we constructed all conserved and thermodynamic quantities of
charged $f(R)$ black holes and verified the validity of the first law of
thermodynamics on the event horizon.

Figure 3: The function $(\partial^{2}M/\partial S^{2})_{Q}$ versus $q$ for
$l=1$, $n=4$, $r_{+}=0.8$. $f^{\prime}(R_{0})=-0.5$ (bold line),
$f^{\prime}(R_{0})=0$ (continuous line) and $f^{\prime}(R_{0})=0.5$ (dashed
line).

Figure 4: The function $(\partial^{2}M/\partial S^{2})_{Q}$ versus $r_{+}$ for
$l=1$, $f^{\prime}(R_{0})=1$ and $n=4$. $q=0.4$ (bold line), $q=1$ (continuous
line) and $q=2$ (dashed line).

Figure 5: The function $(\partial^{2}M/\partial S^{2})_{Q}$ versus $q$ for
$l=1$, $r_{+}=0.8$ and $f^{\prime}(R_{0})=1$. $n=4$ (bold line), $n=8$
(continuous line) and $n=12$ (dashed line).

Figure 6: The function $(\partial^{2}M/\partial S^{2})_{Q}$ versus $q$ for
$l=1$, $n=8$ and $r_{+}=0.8$. $f^{\prime}(R_{0})=-0.5$ (bold line),
$f^{\prime}(R_{0})=0$ (continuous line) and $f^{\prime}(R_{0})=0.5$ (dashed
line).

Figure 7: The function $(\partial^{2}M/\partial S^{2})_{Q}$ versus $q$ for
$l=1$, $n=12$ and $r_{+}=0.8$. $f^{\prime}(R_{0})=-0.5$ (bold line),
$f^{\prime}(R_{0})=0$ (continuous line) and $f^{\prime}(R_{0})=0.5$ (dashed
line).

Finally, we study the local stability of the charged $R+f(R)$ black holes in
the presence of conformally invariant Maxwell source. In the canonical
ensemble, the positivity of the heat capacity $C_{Q}=T/(\partial^{2}M/\partial
S^{2})_{Q}$ and therefore the positivity of $(\partial^{2}M/\partial
S^{2})_{Q}$ is sufficient to ensure the local stability. We have shown the
behavior of $(\partial^{2}M/\partial S^{2})_{Q}$ as a function $q$ and $r_{+}$
for different value of $n$, $q$ and $f^{\prime}(R_{0})$ in figures 3-7 where
we have taken $R_{0}=-n(n-1)/l^{2}$. From figures 3 and 4 we see that the
system is always thermally stable in four dimensions for different value of
the parameters. However, in higher dimensions the system has an unstable phase
as one can see from Fig. 5. As an example, we find that in $8$-dimensions
black holes are always unstable and there is no phase transition (see Fig. 6),
while in $12$-dimensions the system has a transition from unstable phase to
stable phase (see Fig. 7).

##  IV Summary and Discussion

In general the field equations of $f(R)$ gravity coupled to a matter field are
complicated and it is not easy to construct exact analytical solutions.
Recently, it was shown Tae  that by assuming a traceless energy-momentum
tensor for the matter field as well as the constant curvature scalar, one can
extract some analytical black hole solutions in $R+f(R)$ theory coupled to a
matter field. Two examples for the traceless $T_{\mu\nu}$ in four dimensions
are Maxwell and Yang-Mills fields Tae . However, the energy-momentum tensor of
Maxwell field is not traceless in higher dimensions. Seeking for a traceless
energy-momentum tensor in arbitrary dimensions, the authors of Hass1  found a
conformally invariant nonlinear Maxwell action which its energy-momentum
tensor is traceless in $n=4p$ dimensions where $p$ is a positive integer. They
also studied the black hole solutions in Einstein gravity with nonlinear
Maxwell field Hass1 .

In this paper we obtained a class of higher dimensional black holes from
$R+f(R)$ gravity with conformally invariant Maxwell source. The two key
assumptions in finding these solutions are: (i) the constant scalar curvature
$R=R_{0}$ and (ii) the traceless energy momentum tensor. These solutions are
similar to higher dimensional RNAdS black holes with appropriate replacement
of the parameters, but only exist in dimensions which are multiples of four.
Besides, the solutions presented here differ from higher dimensional RNAdS
black holes in two features. First, the electric charge term in the metric
coefficient goes as $r^{-(n-2)}$ while in the standard RNAdS case is
$r^{-2(n-3)}$. Second, the electric field in higher dimensions does not depend
on $n$ and goes as electric field in four dimensional RNAdS black holes. Our
solutions also differ from the higher dimensional black holes of Einstein
gravity with a conformally invariant Maxwell source Hass1  in that they have
vanishing scalar curvature $R=0$, while the obtained solutions here in
$R+f(R)$ gravity coupled to a nonlinear Maxwell field have a constant
curvature scalar $R=R_{0}$. In addition, the conserved and thermodynamic
quantities computed here depend on function $f^{\prime}(R_{0})$ and differ
completely from those of Einstein theory in AdS spaces. Clearly the presence
of the general function $f^{\prime}(R_{0})$ changes the physical values of
conserved and thermodynamic quantities. Furthermore, unlike Einstein gravity,
for the black hole solutions obtained here in $f(R)$ gravity, the entropy does
not obey the area law.

After studying the physical properties of the solutions, we computed the mass,
charge, electric potential and temperature of the black holes. We also found
the entropy expression which does not obey the area law for the $f(R)$ black
holes. We obtained a Smarr-type formula for the mass, and verified that the
conserved and thermodynamics quantities satisfy the first law of black hole
thermodynamics. We also studied the phase behavior of the higher dimensional
charged $f(R)$ black holes. We found that the system is always thermally
stable in four dimensions, while in higher dimensions, there is a phase
transition in the presence of the conformally invariant Maxwell field in
$R+f(R)$ theory with constant curvature.

Finally, we would like to mention that the $n$-dimensional charged $f(R)$
black hole solutions obtained here are static. Thus, it would be interesting
if one can construct charged rotating black brane/hole solutions from $R+f(R)$
theory in the presence of conformally invariant Maxwell source. These issues
are now under investigation and will be appeared elsewhere.

###### Acknowledgements.

This work has been supported financially by Center for Excellence in Astronomy
and Astrophysics (CEAA-RIAAM), Maragha, under research project number 1/2718.

## References

  * (1) E. J. Copeland, M. Sami and S. Tsujikawa, Int. J. Mod. Phys. D 15, 1753 (2006);
M. Li, X. D. Li, S. Wang, Y. Wang, Commun. Theor. Phys. 56, (2011) 525.

  * (2) S. Nojiri and S. D. Odintsov, Phys. Rev. D 68, 123512 (2003);
S. Nojiri and S. D. Odintsov, Phys. Lett. B 576, 5 (2003);
S. Nojiri and S. D. Odintsov, Phys. Rev. D 74, 086005 (2006);
S. Nojiri and S. D. Odintsov, Int. J. Geom. Meth. Mod. Phys. 4, 115 (2007);
S. Nojiri and S. D. Odintsov, J. Phys. Conf. Ser. 66, 012005 (2007);
S. Nojiri and S. D. Odintsov, Phys. Rept. 505, 59 (2011);
S. Capozziello, S. Nojiri, S. D. Odintsov and A. Troisi, Phys. Lett. B 639,
135 (2006);
S. Nojiri and S. D. Odintsov, Phys. Rev. D 78, 046006 (2008).

  * (3) S. Capozziello, Int. J. Mod. Phys. D 11, 483, (2002);
S. Capozziello, V. F. Cardone, and A. Troisi, J. Cosmol. Astropart. Phys. 08,
001 (2006);
S. Capozziello, V. F. Cardone, and A. Troisi, Mon. Not. R. Astron. Soc. 375,
1423 (2007);
K. Atazadeh, M. Farhoudi and H. R. Sepangi, Phys. Lett. B 660, 275 (2008);
C. Corda, Astropart. Phys. 34, 587 (2011).

  * (4) A. De Felice and S. Tsujikawa, Living Rev. Rel. 13, 3 (2010);
S. Capozziello and M. De Laurentis, arXiv:1108.6266, to appear in Physics
Reports.

  * (5) G. Cognola, E. Elizalde, S. Nojiri, S. D. Odintsov and S. Zerbini, JCAP 0502, 010 (2005).
  * (6) L. Sebastiani and S. Zerbini, arXiv:1012.5230.
  * (7) A. de la Cruz-Dombriz, A. Dobado and A. L. Maroto, Phys. Rev. D 80, 124011 (2009);
A. de la Cruz-Dombriz, A. Dobado and A. L. Maroto. Phys Rev. D 83, 029903 (E)
(2011).

  * (8) S. H. Hendi, Phys. Lett. B 690, 220 (2010);
S. H. Hendi, et. al., Gen Relativ Gravit 44 (2012) 835;
S. H. Hendi, and D. Momeni, Eur. Phys. J. C 71 (2011) 1823.

  * (9) T. Moon, Y. S. Myung and E. J. Son, arXiv:1101.1153, to appear in Gen Relativ Grav.
  * (10) A. Larranaga, arXiv:1108.6325;
J. A. R. Cembranos, et al., arXiv:1109.4519

  * (11) A. Sheykhi and S. Salarpour, Submitted to Phys. Scrip.
  * (12) A. Sheykhi, Submitted to Gen. Relativ Grav.
  * (13) M. Hassaine and C. Martinez, Phys. Rev. D 75 (2007) 027502\.
  * (14) M. Hassaine and C. Martinez, Class. Quantum Gravit. 25 (2008) 195023;
H. A. Gonzalez, M. Hassaine and C. Martinez, Phys. Rev. D 80, 104008 (2009);
H. Maeda, M. Hassaine and C. Martinez, Phys. Rev. D 79 (2009) 044012.

  * (15) S. H. Hendi and B. Eslam Panah, Phys. Lett. B 684 (2010) 77;
S. H. Hendi, Eur. Phys. J. C 69 (2010) 281;
S. H. Hendi, Phys. Rev. D 82 (2010) 064040;
S. H. Hendi, Phys. Lett. B 677 (2009) 123.

  * (16) O. Miskovic, R. Olea, Phys. Rev. D 83 (2001) 024011;
O. Miskovic, R. Olea, Phys. Rev. D 83 (2011) 064017\.

  * (17) J. Brown and J. York, Phys. Rev. D 47, 1407 (1993);
J.D. Brown, J. Creighton, and R. B. Mann, Phys. Rev. D 50, 6394 (1994).

  * (18) S. Weinberg, Gravitation and cosmology, (John Wiley 1972, Chapter 15).
  * (19) I. Brevik, S. Nojiri, S.D. Odintsov and L. Vanzo, Phys. Rev. D 70, 043520 (2004).