11institutetext:  Key Lab of Intelligent Information Processing of Chinese
Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing
100190, China
11email<EMAIL_ADDRESS>22institutetext: School of Biomedical
Engineering & Suzhou Institute for Advanced Research Center for Medical
Imaging, Robotics, and Analytic Computing & LEarning (MIRACLE) University of
Science and Technology of China, Suzhou 215123, China 33institutetext:
Hospital of Guizhou Medical University 44institutetext: Tsinghua University

# Relative distance matters for one-shot landmark detection

Qingsong Yao 11 Jianji Wang 33 Yihua Sun 44 Quan Quan 11 Heqin Zhu 11
S. Kevin Zhou 11 2 2

###### Abstract

Contrastive learning based methods such as cascade comparing to detect (CC2D)
have shown great potential for one-shot medical landmark detection. However,
the important cue of relative distance between landmarks is ignored in CC2D.
In this paper, we upgrade CC2D to version II by incorporating a simple-yet-
effective relative distance bias in the training stage, which is theoretically
proved to encourage the encoder to project the relatively distant landmarks to
the embeddings with low similarities. As consequence, CC2Dv2 is less possible
to detect a wrong point far from the correct landmark. Furthermore, we present
an open-source, landmark-labeled dataset for the measurement of biomechanical
parameters of the lower extremity to alleviate the burden of orthopedic
surgeons. The effectiveness of CC2Dv2 is evaluated on the public dataset from
the ISBI 2015 Grand-Challenge of cephalometric radiographs and our new
dataset, which greatly outperforms the state-of-the-art one-shot landmark
detection approaches.

###### Keywords:

Medical Landmark Detection One-shot Learning

##  1 Introduction

As a fundamental first step in therapy planning and intervention, precise and
reliable anatomical landmark detection has attracted great interest from
academia and industry [22, 15, 27, 28]. Moreover, it serves as an important
step in medical image analysis [14, 18], e.g., initialization of segmentation
or registration algorithms. Recently, deep learning based methods have great
success in efficiently localizing anatomical landmarks in radiological images
[2, 10, 18, 15]. Chen et al. [4] regress the heatmap and offset maps
simultaneously. Zhong et al. [25] use cascade U-Net to perform a two-stage
heatmap regression. Li et al. [12] adapt multiple deep adaptive graphs to
capture relationships among landmarks.

A common wisdom is that more training data enhance the supervised methods’
better performance and generalization. However, for medical image analysis,
the annotation of datasets needs well-trained, highly-engaged radiologists,
which is especially challenging as experts are costly and busy. To break this
limitation, an array of self-supervised learning attempts have been explored
in classification and segmentation tasks, including image restoration [32,
31], patch ordering [34], superpixel-wise [16] and patch-wise [3] contrastive
learning [26].

For medical landmark detection, two previous approaches have challenged the
hardest scenario: _Only one annotated data is available_. RPR-Net [11] embeds
the patches with similar anatomical contexts into close relative positions.
Cascade comparing to detect (CC2D) [24] learns to project the pixels with the
same anatomical structures into embeddings with high cosine similarities, by
solving a self-supervised “patch matching” proxy task in a coarse-to-fine
fashion.

Delving into CC2D, despite the great performance with one labeled data
available, we find that the important cue of relative distance between
landmarks [30, 29, 5] is ignored. Therefore, we add a novel relative distance
bias (RDB) in the training stage of CC2D. Theoretical analysis proves that RDB
multiplies the gradient of the negative points by a relative distance aware
factor, which encourages low similarities for relatively distant points. This
add-on term brings two benefits: 1) It makes the framework less possible to
detect a wrong point far from the correct landmark; 2) It prevents the
encoders from wasting too much effort on decreasing the similarities of the
relatively close points with similar anatomical contexts. Here, we name our
upgraded version of “relative distance aware cascade comparing to detect” as
CC2Dv2.

The effectiveness of CC2Dv2 is evaluated by extensive experiments on the
public ISBI 2015 Grand-Challenge dataset [8]. Our CC2Dv2 outperforms the
state-of-the-art medical one-shot landmark detection methods with a
significant accuracy improvement. Furthermore, we contribute an open-source
dataset for the measurement of BioMechanical Parameters of the Lower Extremity
(BMPLE) arising from orthopedic surgery, which is suitable to evaluate the
effectiveness of both one-shot and fully-supervised landmark detection
methods.

##  2 Method

Figure 1: The training strategy of the proposed relative distance aware
cascade comparing (CC2Dv2-SSL). For each layer of the cascade features,
CC2Dv2-SSL penalizes the cosine similarities drastically between the selected
ground-truth point and the relatively distant points, by adding a novel
relative distance bias (RDB) before applying softmax function. As consequence,
the feature extractors are encouraged to embed the distant landmarks to the
embeddings with low similarities.

The overall framework of CC2Dv2 is build on top of CC2D [24], consisting of
two stages: self-supervised learning (SSL) and training from pseudo-labels
(TPL). CC2Dv2 differs from CC2D in the SSL stage: It incorporates a relative
distance bias to encourage low similarities between the relatively distant
landmarks. TPL stays the same, that is, using the pseudo-labels generated from
SSL stage, a multi-task U-Net [23] is trained to generate heatmap and offset-
maps simultaneously, which effectively improves the performance and inference
speed [1].

Cascade cosine similarities: The computation flow in CC2Dv2-SSL is the same as
in CC2D-SSL [24]. As shown in Fig. 1, let $X_{r}$ denote an input image
resized to $H\times W$, we arbitrarily select a target point
$P_{r}=(x_{r},y_{r})$. First, we randomly crop a patch $X_{p}$ containing
$P_{r}$ with size $H_{p}\times W_{p}$, and augment the content of $X_{p}$ by
rotation and color jittering, which moves the anchor point $P_{r}$ to
$P_{p}=(x_{p},y_{p})$. Then, the input $X_{r}$ and the patch $X_{p}$ are
projected into cascade of multi-scale feature embeddings $F_{r}=E_{r}(X_{r})$
and $F_{p}=E_{p}(X_{p})$, by two feature extractors $E_{r}$ and $E_{p}$,
respectively. Here, the embedding of $i^{th}$ layer is marked as $F^{i}$with
size $H_{p}^{i}\times W_{p}^{i}$. Next, for each layer, we extract the anchor
feature $f_{a}^{i}$ of the anchor point $P_{p}$ from $F_{p}^{i}$, guided by
its corresponding coordinates $P_{a}^{i}=(x_{p}/2^{i},y_{p}/2^{i})$ which are
down-sampled for $i$ times. Finally, the cosine similarity map $s^{i}$ for
each layer between input feature $F_{r}^{i}$ and anchor feature $f_{a}^{i}$ is
computed:

$\displaystyle S^{i}=\frac{\langle f_{a}^{i}\cdot
F_{r}^{i}\rangle}{||f_{a}^{i}||_{2}\cdot||F_{r}^{i}||_{2}},$ (1)

Then, following CC2D [24], we crop the matrix of interest
$S^{i}_{\Delta}=\\{s_{mn}^{i}\\}_{M\times N}$ from the similarity map
$S^{i}=\\{s_{xy}^{i}\\}_{H^{i}\times W^{i}}$ with a size of $M\times N$ for
each layer. The selected point $P_{r}^{i}=(x_{r}/2^{i},y_{r}/2^{i})$ now has a
new coordinate $P_{t}^{i}=(m_{t}^{i},n_{t}^{i})$ in the matrix
$S^{i}_{\Delta}$.

Relative distance bias (RDB): Next, we compute the relative Euclidean distance
$d_{mn}^{i}$ for each pixel $P_{mn}^{i}$ in $S^{i}_{\Delta}$ with coordinate
$(m,n)$ to the selected point $P_{t}^{i}=(m_{t}^{i},n_{t}^{i})$ as follow:

$\displaystyle d_{mn}^{i}=\sqrt{(m-m_{t}^{i})^{2}+(n-n_{t}^{i})^{2}},$ (2)

Then the relative distance is clipped to $[0,\beta]$ and added to the cropped
cosine similarity matrix $S^{i}_{\Delta}$, resulting in relative distance
aware similarity matrix $W^{i}_{mn}$, whose element $w_{mn}^{i}$ is given as:

$\\{w^{i}_{mn}\\}_{M\times N}=\\{s^{i}_{mn}+b^{i}_{mn}\\}_{M\times
N},b^{i}_{mn}=\left\\{\alpha*\mathop{clip_{[0,\beta]}}(d_{mn}^{i})\right\\}_{M\times
N},$ (3)

where $\alpha$ controls the slope of the relative distance.

Cascade comparing to detect: Here, we mark $P_{t}^{i}$ as the ground-truth
point and denote other non-ground-truth points in matrix $W^{i}$ (or matrix
$S^{i}_{\Delta}$) as $P_{\hat{m}\hat{n}}^{i}$. The ground truth matrix
$GT^{i}$ is computed as $GT^{i}_{m_{t},n_{t}}=1,GT^{i}_{\hat{m}\hat{n}}=0$.
After applying softmax function to normalize $W^{i}$ to probability matrix
$Q^{i}$ with a temperature $\tau$: $q_{mn}^{i}=softmax(w_{mn}^{i}*\tau)$. we
use cross-entropy loss $L_{CE}^{i}$ to increase the probability
$q_{m_{t},n_{t}}^{i}$ of the selected ground-truth point
$P_{t}^{i}=(m_{t}^{i},n_{t}^{i})$ while decreasing the probabilities
$q_{\hat{m}\hat{n}}^{i}$ of other non-ground-truth points
$P_{\hat{m}\hat{n}}^{i}$ for multi-scale layers:

$\displaystyle
L_{CE}^{i}(Q^{i},GT^{i})=-\sum_{m}\sum_{n}GT_{mn}^{i}*\mathop{log}(q_{mn}^{i})=-\mathop{log}(q_{m_{t},n_{t}}^{i}),$
(4)

Then, we compute $L_{CE}^{i}(q^{i},GT^{i})$ for all layers as final loss
$L_{SSL}$:

$\displaystyle L_{SSL}=\sum_{i}L_{CE}^{i}(Q^{i},GT^{i}),$ (5)

Theoretical analysis: To better understand the effectiveness of the relative
distance bias, we compute the partial derivative of $L_{CE}^{i}$ with respect
to the cosine similarities $s_{\hat{m}\hat{n}}^{i}$ of the negative points
$P_{\hat{m}\hat{n}}^{i}$ in $W_{\Delta}^{i}$ on the layer $i$:

$\displaystyle\nabla_{s_{\hat{m}\hat{n}}^{i}}L_{CE}^{i}$
$\displaystyle=\sum_{m,n}\left[\nabla_{q_{mn}^{i}}L_{CE}^{i}(q_{mn}^{i},GT^{i})\times\nabla_{s_{\hat{m}\hat{n}}^{i}}q_{mn}^{i}\right]$
(6)
$\displaystyle=-\nabla_{q_{m_{t},n_{t}}^{i}}\mathop{log}(q_{m_{t},n_{t}}^{i})\times\sum_{m,n}\left(\nabla_{w_{mn}^{i}}q_{m_{t},n_{t}}^{i}\times\nabla_{s_{\hat{m}\hat{n}}^{i}}w_{mn}^{i}\right)$
$\displaystyle=-\nabla_{q_{m_{t},n_{t}}^{i}}\mathop{log}(q_{m_{t},n_{t}}^{i})\times\nabla_{w_{\hat{m}\hat{n}}^{i}}q_{m_{t},n_{t}}^{i}$
$\displaystyle=\frac{\tau*\mathop{exp}(w_{\hat{m}\hat{n}}^{i}*\tau)}{\sum_{m,n}\mathop{exp}(w_{mn}^{i}*\tau)}\propto\mathop{exp}(b_{\hat{m}\hat{n}}^{i}*\tau)*\mathop{exp}(s_{\hat{m}\hat{n}}^{i}*\tau),$

The derivative results show that the relative distance bias $d^{i}$ directly
multiplies the gradient of the cosine similarities $s_{\hat{m}\hat{n}}^{i}$ of
the negative points by $\mathop{exp}(b_{\hat{m}\hat{n}}^{i}*\tau)$.
Accordingly, the negative points farther from the selected GT (positive) point
(with greater relative distance $b_{\hat{m}\hat{n}}^{i}$) are penalized more
harshly to have low cosine similarities in the embedding space, which is
exactly what we need.

Inference steps of SSL stage: Consistent with CC2D [24], the template anchor
features $f_{a}$ are embedded by $E_{p}$ from the content of the labeled
landmark in the template image. Next, we compute cascade cosine similarities
$s^{i}$ between $f_{a}$ and $F_{q}$, where $F_{q}$ is the query features
extracted from the query image by $E_{r}$. At last, we multiply the cascade
similarities $s^{i}$, which are clipped to range $[0,1]$. The final prediction
is returned by _argmax_ operator in the coarse-to-fine fashion [24]. We infer
the landmark locations for the unlabelled images in the training set as
pseudo-labels needed for the TPL stage.

##  3 Experiments

###  3.1 Settings

Dataset: A widely-used public dataset released for cephalometric landmark
detection in IEEE ISBI 2015 grand Challenge [21, 8] is used, which contains
400 radiographs.19 landmarks of clinical anatomical significance are labeled
by two expert doctors for each radiograph. We take the average annotations as
the ground truth. The image size is $1935\times 2400$ and the pixel spacing is
0.1mm. The dataset is split into training and test subsets with 150 and 250
radiographs according to the official website, respectively.

Metrics: Following the challenge [21], we use mean radial error (MRE) and
successful detection rate (SDR) as metrics. We set four radii (2mm, 2.5mm,
3mm, and 4mm) for the cephalometric dataset.

Implementation details: The models are implemented in PyTorch [17],
accelerated by an NVIDIA TITAN RTX GPU. We resize the images in cephalometric
and BMPLE dataset to $384\times 384$ and $384\times 768$ , respectively. For
SSL stage, the two feature extractors are optimized by Adam [9] optimizer for
5000 epochs with a learning rate of 0.001 decayed by 0.5 every 500 epochs,
with a batch size of 8. The size of cropped patch $X_{p}$ is set to $192\times
192$, while the shape of matrix $T$ is $19\times 19$. The $\beta$ and $\alpha$
in Eq. 3 are 0.7 and 0.1. For the template image, we choose the 125# training
image in cephalometric dataset, and 120# training data in BMPLE dataset. For
TPL stage, the multi-task U-Net is optimized by Adam optimizer for 900 epochs
with a learning rate of 0.0003 decayed by 0.1 every 300 epochs, with a batch
size of 8. We set $\tau=10$.

Table 1: Comparison of the state-of-the-art landmark detection methods under fully-supervised and one-shot setting on the cephalometric testset. * represents the performances copied from their original manuscripts, while # represents the performances we re-implement according to the their official code. Model |  Labeled images |  MRE ($\downarrow$) (mm) |  SDR ($\uparrow$) (%)
---|---|---|---
2mm | 2.5mm | 3mm | 4mm | 6mm | 8mm
Human experts | - | 1.07 | 85.60 | 90.48 | 93.64 | 96.92 | 99.51 | 99.83
Ibragimov et al. [7]* | 150 | - | 68.13 | 74.63 | 79.77 | 86.87 | - | -
Lindner et al. [13]* | 150 | 1.77 | 70.65 | 76.93 | 82.17 | 89.85 | - | -
Payer et al. [19]* | 150 | - | 73.33 | 78.76 | 83.24 | 89.75 | - | -
GU2-Net[33]# | 150 | 1.69 | 76.95 | 83.98 | 88.82 | 94.21 | 98.23 | 99.14
GU2-Net[33]# | 25 | 2.41 | 66.88 | 76.29 | 82.40 | 90.02 | 95.71 | 97.45
GU2-Net[33]# | 10 | 9.79 | 48.77 | 58.74 | 65.07 | 74.23 | 80.65 | 82.61
GU2-Net[33]# | 5 | 18.88 | 38.06 | 46.86 | 52.91 | 62.40 | 70.15 | 73.52
RPR_Net [11]# | 1 | 4.45 | 19.45 | 26.4 | 36.06 | 52.74 | 76.48 | 88.97
CC2D-SSL [24]* | 1 | 4.67 | 40.42 | 47.68 | 55.54 | 68.38 | - | -
CC2D [24]* | 1 | 2.72 | 49.81 | 58.73 | 68.18 | 81.01 | - | -
CC2Dv2-SSL | 1 | 2.70 | 44.15 | 55.16 | 67.05 | 81.64 | 93.87 | 97.60
CC2Dv2 | 1 | 2.23 | 53.75 | 64.93 | 76.48 | 89.14 | 97.43 | 99.16
Figure 2: (a) The comparison between the cosine similarities generated by CC2D
and CC2Dv2. The correct landmark is marked in red circle. (b) Visualizations
of predictions in cephalometric test dataset. i; ii; iii; iv represent the
results inferenced from RPR-Net, CC2D-SSL, CC2Dv2-SSL, and CC2D, respectively.
The landmarks in green and red are the predictions and ground truths, while
the errors are marked in yellow line. (c) Visualizations of testing images in
BMPLE dataset predicted by CC2D and CC2Dv2.

###  3.2 The effectiveness of RDB

We visualize the cosine similarities in Fig. 2(a). Compared with CC2D, the
candidate pixels (pixels with similarity $s^{i}_{xy}>0)$) are more close to
the correct landmark, and the cosine similarities are more contrastive, which
validate that relative distance bias successfully helps the encoders in CC2Dv2
project the relatively distant landmarks to the embeddings with low
similarities. The quantitative performances in Table 1 and the visualizations
in Fig. 2(b,c) demonstrate that CC2Dv2 greatly outperforms other one-shot
stat-of-the-art methods in the two medical landmark detection datasets, and is
even competitive to the fully-supervised methods (the first [13] and second
[7] place in the ISBI Challenge [21]) in terms of SDR-4mm.

Limitation: Despite that CC2Dv2 effectively decreases MRE by reducing the huge
errors (_e.g.,_ 4mm SDR for cephalometric dataset is improved from 81.01% to
89.14%), there is still a considerable performance gap compared with fully-
supervised methods to precisely detect landmarks with error less than 2mm.

###  3.3 Hyper-parameter analysis

We study the influence of different settings of the slope $\alpha$ and the
maximum value of the RDB $\beta$ in Eq. 3. According to Table 2, the
performances are stable when $\alpha$ lies in $0.7\sim 1.3$. For $\beta$, a
small $\beta$ decreases the effectiveness of RDB. As the maximum value of
similarity is 1.0, a too large $\beta$ makes trouble in the convergence of
$L_{SSL}$. Setting $\beta$ to 0.7 leads to the best performance.

Table 2: The performances of CC2Dv2-SSL under different settings of the slope $\alpha$ and the maximum value of the RDB $\beta$ in Eq. 3. Para. | Value |  MRE ($\downarrow$) (mm) |  SDR ($\uparrow$) (%)
---|---|---|---
2mm | 2.5mm | 3mm | 4mm | 6mm | 8mm
$\beta$ | 0.9 | 2.87 | 39.12 | 50.31 | 62.53 | 79.39 | 93.20 | 96.95
0.8 | 2.82 | 40.09 | 51.75 | 63.6 | 79.66 | 93.70 | 97.35
0.7 | 2.70 | 44.15 | 55.16 | 67.05 | 81.64 | 93.87 | 97.60
0.6 | 2.98 | 40.55 | 51.24 | 62.72 | 78.38 | 91.54 | 95.66
0.5 | 3.14 | 41.85 | 51.79 | 61.98 | 76.63 | 89.47 | 94.42
$\alpha$ | 0.04 | 3.32 | 43.32 | 53.52 | 63.37 | 76.57 | 88.48 | 93.22
0.07 | 2.73 | 44.95 | 55.47 | 66.67 | 81.65 | 93.62 | 97.01
0.10 | 2.70 | 44.15 | 55.16 | 67.05 | 81.64 | 93.87 | 97.60
0.13 | 2.71 | 44.86 | 55.77 | 67.22 | 83.01 | 93.81 | 97.22
0.16 | 3.07 | 39.38 | 50.4 | 61.47 | 77.75 | 91.07 | 95.34
Table 3: Comparison of the state-of-the-art landmark detection methods under fully-supervised and one-shot setting on the BMPLE testset. Model |  Labeled images | Test Dataset 1 | Test Dataset 2
---|---|---|---
MRE ($\downarrow$) (mm) |  SDR ($\uparrow$) (%) |  MRE ($\downarrow$) (mm) |  SDR ($\uparrow$) (%)
4mm | 8mm | 12mm | 4mm | 8mm | 12mm
Human experts | - | 1.66 | 90.50 | 97.50 | 98.25 | - | - | - | -
Payer et al. [18] | 120 | 5.61 | 34.00 | 81.50 | 95.50 | 5.92 | 35.00 | 78.30 | 92.60
Chen et al. [4] | 120 | 3.08 | 85.25 | 95.25 | 97.50 | 3.23 | 81.80 | 93.40 | 96.10
RPR-Net [11] | 1 | 129.2 | 0.50 | 1.00 | 2.00 | 132.3 | 0.10 | 0.10 | 0.90
CC2D-SSL [24] | 1 | 27.86 | 17.50 | 35.00 | 43.00 | 30.69 | 14.60 | 34.90 | 45.10
CC2D [24] | 1 | 21.55 | 14.75 | 37.25 | 53.75 | 20.24 | 16.50 | 39.90 | 56.90
CC2Dv2-SSL | 1 | 12.28 | 18.25 | 53.25 | 77.50 | 13.60 | 19.80 | 46.30 | 68.90
CC2Dv2 | 1 | 7.17 | 27.25 | 69.00 | 88.50 | 8.18 | 24.40 | 62.30 | 84.40

##  4 BMPLE Dataset

Clinical significance Biomechanical parameters are essential for orthopedic
refined procedures, _e.g.,_ measuring the degree of varus/valgus of the knee,
planing osteotomy angle, evaluating the improvement of lower limb force lines,
and predicting the risk of abnormal wear and loosening of artificial joints
[6]. However, BMPLE measuration is time-consuming and laborious for orthopedic
surgeons. It takes up to 10 minutes to carefully annotate and connect the bone
markers [20].

Figure 3: (a) The illustration of the 20 annotated landmarks in the lower
extremity radiography in BMPLE dataset, 1-10 landmarks are on the right (R)
lower limb, while the left 11-20 landmarks are on the left (L) lower limb. (b)
The biomechanical angle (“Right FTA” in figure b) is calculated between two
axes, each axis is connected by two landmarks. (c) The detailed definitions of
axis and biomechanical angles.

Data acquisition: The dataset consists of 190 radiographs collected from two
collaborated hospitals. The study has been approved by the Hospitals Committee
and carried out in accordance with the Declaration of Helsinki. All of the
radiographs have been desensitized. The radiographs sizes are distributed from
$2396\times 4950$ to $3200\times 8500$, while the pixel spacing lies in
$0.13mm\sim 0.16mm$.

Annotation: The dataset is split into three subsets (Training, Test1, and
Test2) with 120, 20, and 50 radiographs, respectively. The radiographs in the
training and Test2 subsets are annotated by one senior orthopedic surgeon,
while the Test1 subset is labeled by three experts, we compute the average
annotations as the ground-truth and report the error of human experts in Table
3.

Measurement of biomechanical parameters of lower extremity relies on accurate
anatomical landmark localization. As illustrated in Fig. 3, for each lower
limb, 10 landmarks are defined and labeled, 6 axes are generated by connecting
the corresponding landmarks, and 7 biomechanical angles are computed according
to the corresponding two axes [20], the illustrations of other biomechanical
angles can be found in the supplemental materials.

Fully supervised performances We establish a preliminary automatic BMPLE
measurement framework, which localizes landmarks using the well-trained SOTA
method (Chen et al. [4]) and computes BMPLE according the definitions in Fig.
3(b,c). The errors are 3.90°$\pm$5.17° and 4.25°$\pm$7.09° on Test1 and Test2
subset, respectively, while the error of human experts on Test1 is
0.69°$\pm$2.38°. The inference speed is 1.55s per image. We hope that this
dataset can help the community develop an accurate and fast BMPLE measurement
framework, which is valuable to orthopedic surgeons.

One-shot performances Table 3 shows the landmark detection performances of
varying competing methods. It is evident that CC2Dv2 clearly outperforms other
one-shot approaches by a large margin. However, there is a large performance
gap between CC2Dv2 and fully supervised methods and Human experts, which
motivates us to improve the accuracy in the future work.

##  5 Conclusion

In this paper, we upgrade cascade comparing to detect (CC2D) to CC2Dv2, by
adding a simple-yet-effective relative distance bias (RDB) in the training
stage. The theoretical analysis proves that RDB encourages the encoder to
project the distant landmarks into the embeddings with low cosine
similarities. Extensive experiments on two medical landmark detection datasets
validate that CC2Dv2 greatly surpasses other state-of-the-art one-shot
landmark detection methods. Future work includes developing faster and more
accurate methods for the BMPLE dataset to benefit the orthopedic surgery
community.

## References

  * [1] Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M.S., Maharaj, T., Fischer, A., Courville, A., Bengio, Y., et al.: A closer look at memorization in deep networks. In: ICML. pp. 233–242. PMLR (2017)
  * [2] Bhalodia, R., Kavan, L., Whitaker, R.T.: Self-supervised discovery of anatomical shape landmarks. In: MICCAI. pp. 627–638. Springer (2020)
  * [3] Chaitanya, K., Erdil, E., Karani, N., Konukoglu, E.: Contrastive learning of global and local features for medical image segmentation with limited annotations. arXiv preprint arXiv:2006.10511 (2020)
  * [4] Chen, R., Ma, Y., Chen, N., Lee, D., Wang, W.: Cephalometric landmark detection by attentive feature pyramid fusion and regression-voting. In: MICCAI. pp. 873–881. Springer (2019)
  * [5] Criminisi, A., Robertson, D., Konukoglu, E., Shotton, J., Pathak, S., White, S., Siddiqui, K.: Regression forests for efficient anatomy detection and localization in computed tomography scans. Medical image analysis 17(8), 1293–1303 (2013)
  * [6] van Hamersveld, K.T., Marang-van de Mheen, P.J., Nelissen, R.G.: The effect of coronal alignment on tibial component migration following total knee arthroplasty: a cohort study with long-term radiostereometric analysis results. JBJS 101(13), 1203–1212 (2019)
  * [7] Ibragimov, B., Likar, B., Pernus, F., Vrtovec, T.: Computerized cephalometry by game theory with shape-and appearance-based landmark refinement. In: Proceedings of International Symposium on Biomedical imaging (ISBI) (2015)
  * [8] Kaggle: Cephalometric X-Ray Landmarks Detection Challenge (2015), https://www.kaggle.com/jiahongqian/cephalometric-landmarks/discussion/133268
  * [9] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. ICLR (2015)
  * [10] Lang, Y., Lian, C., Xiao, D., Deng, H., Yuan, P., Gateno, J., Shen, S.G., Alfi, D.M., Yap, P.T., Xia, J.J.: Automatic localization of landmarks in craniomaxillofacial cbct images using a local attention-based graph convolution network. In: MICCAI. pp. 817–826. Springer (2020)
  * [11] Lei, W., Xu, W., Gu, R., Fu, H., Zhang, S., Zhang, S., Wang, G.: Contrastive learning of relative position regression for one-shot object localization in 3d medical images. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 155–165. Springer (2021)
  * [12] Li, W., Lu, Y., Zheng, K., Liao, H., Lin, C., Luo, J., Cheng, C.T., Xiao, J., Lu, L., Kuo, C.F.: Structured landmark detection viatopology-adapting deep graph learning. In: ECCV (2020)
  * [13] Lindner, C., Cootes, T.F.: Fully automatic cephalometric evaluation using random forest regression-voting. Scientific Reports 6, 33581 (2016)
  * [14] Liu, D., Zhou, S.K., Bernhardt, D., Comaniciu, D.: Search strategies for multiple landmark detection by submodular maximization. In: Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. pp. 2831–2838. IEEE (2010)
  * [15] Liu, W., Wang, Y., Jiang, T., Chi, Y., Zhang, L., Hua, X.S.: Landmarks detection with anatomical constraints for total hip arthroplasty preoperative measurements. In: MICCAI. pp. 670–679. Springer (2020)
  * [16] Ouyang, C., Biffi, C., Chen, C., Kart, T., Qiu, H., Rueckert, D.: Self-supervision with superpixels: Training few-shot medical image segmentation without annotation. In: ECCV. pp. 762–780. Springer (2020)
  * [17] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019)
  * [18] Payer, C., Štern, D., Bischof, H., Urschler, M.: Regressing heatmaps for multiple landmark localization using cnns. In: MICCAI. pp. 230–238. Springer (2016)
  * [19] Payer, C., Štern, D., Bischof, H., Urschler, M.: Integrating spatial configuration into heatmap regression based cnns for landmark localization. Medical image analysis 54, 207–219 (2019)
  * [20] Teeter, M.G., Naudie, D.D., McCalden, R.W., Yuan, X., Holdsworth, D.W., MacDonald, S.J., Lanting, B.A.: Varus tibial alignment is associated with greater tibial baseplate migration at 10 years following total knee arthroplasty. Knee Surgery, Sports Traumatology, Arthroscopy 26(6), 1610–1617 (2018)
  * [21] Wang, C.W., Huang, C.T., Lee, J.H., Li, C.H., Chang, S.W., Siao, M.J., Lai, T.M., Ibragimov, B., Vrtovec, T., Ronneberger, O., et al.: A benchmark for comparison of dental radiography analysis algorithms. Medical Image Analysis 31, 63–76 (2016)
  * [22] Yang, D., Xiong, T., Xu, D., Huang, Q., Liu, D., Zhou, S.K., Xu, Z., Park, J., Chen, M., Tran, T.D., et al.: Automatic vertebra labeling in large-scale 3d ct using deep image-to-image network with message passing and sparsity regularization. In: IPMI. pp. 633–644 (2017)
  * [23] Yao, Q., He, Z., Han, H., Zhou, S.K.: Miss the point: Targeted adversarial attack on multiple landmark detection. In: MICCAI. pp. 692–702. Springer (2020)
  * [24] Yao, Q., Quan, Q., Xiao, L., Kevin Zhou, S.: One-shot medical landmark detection. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 177–188. Springer (2021)
  * [25] Zhong, Z., Li, J., Zhang, Z., Jiao, Z., Gao, X.: An attention-guided deep regression model for landmark detection in cephalograms. In: MICCAI. pp. 540–548. Springer (2019)
  * [26] Zhou, H.Y., Yu, S., Bian, C., Hu, Y., Ma, K., Zheng, Y.: Comparing to learn: Surpassing imagenet pretraining on radiographs by comparing image representations. In: MICCAI. pp. 398–407. Springer (2020)
  * [27] Zhou, S.K., Greenspan, H., Davatzikos, C., Duncan, J.S., van Ginneken, B., Madabhushi, A., Prince, J.L., Rueckert, D., Summers, R.M.: A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises. Proceedings of the IEEE (2021)
  * [28] Zhou, S.K., Rueckert, D., Fichtinger, G.: Handbook of Medical Image Computing and Computer Assisted Intervention. Academic Press (2019)
  * [29] Zhou, S.K.: Shape regression machine and efficient segmentation of left ventricle endocardium from 2d b-mode echocardiogram. Medical image analysis 14(4), 563–581 (2010)
  * [30] Zhou, S.K., Zhou, J., Comaniciu, D.: A boosting regression approach to medical anatomy detection. In: Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE Conference on. pp. 1–8. IEEE (2007)
  * [31] Zhou, Z., Sodha, V., Pang, J., Gotway, M.B., Liang, J.: Models genesis. Medical image analysis 67, 101840 (2021)
  * [32] Zhou, Z., Sodha, V., Siddiquee, M.M.R., Feng, R., Tajbakhsh, N., Gotway, M.B., Liang, J.: Models genesis: Generic autodidactic models for 3d medical image analysis. In: MICCAI. pp. 384–393. Springer (2019)
  * [33] Zhu, H., Yao, Q., Xiao, L., Zhou, S.K.: You only learn once: Universal anatomical landmark detection. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 85–95. Springer (2021)
  * [34] Zhu, J., Li, Y., Hu, Y., Ma, K., Zhou, S.K., Zheng, Y.: Rubik’s cube+: A self-supervised feature learning framework for 3d medical image analysis. Medical Image Analysis 64, 101746 (2020)


# Can varying the gravitational constant alleviate the tensions ?

Z. Sakr  and D. Sapone

###### Abstract

Constraints on the cosmological concordance model parameters from observables
at different redshifts are usually obtained using the locally measured value
of the gravitational constant $G_{N}$. Here we relax this assumption, by
considering $G$ as a free parameter, either constant over the redshift range
or dynamical but limited to differ from fiducial value only above a certain
redshift. Using CMB data and distance measurements from galaxy clustering BAO
feature, we constrain the cosmological parameters, along with $G$, through a
MCMC bayesian inference method. Furthermore, we investigate whether the
tensions on the matter fluctuation $\sigma_{8}$ and Hubble $H_{0}$ parameter
could be alleviated by this new variable. We used different parameterisations
spanning from a constant $G$ to a dynamical $G$. In all the cases investigated
in this work we found no mechanism that alleviates the tensions when both CMB
and BAO data are used with $\xi_{\mathrm{g}}=G/G_{N}$ constrained to 1.0$\pm
0.04$ (resp. $\pm 0.01$) in the constant (resp. dynamical) case. Finally, we
studied the cosmological consequences of allowing a running of the spectral
index, since the later is sensitive to a change in $G$. For the two
parameterisations adopted, we found no significant changes to the previous
conclusions.

##  1 Introduction

The $\Lambda$CDM concordance model has been successful in accommodating with
most of nowadays cosmological probes like the cosmic microwave background
(CMB) temperature and polarisation power spectrum, weak lensing correlations,
galaxy clustering and supernova’s Hubble diagram. Recent measurements of the
aforementioned observables, Planck [1], KIDS [2], eBOSS [3], DES [4] and
PANTHEON survey [5] have succeeded in putting strong constraints on the
parameters of the $\Lambda$CDM model with near percent precision. However,
these constraints have been obtained with some prior assumptions, among them,
a fixed value for the Newtonian Gravitational coupling ($G_{N}$) treated as a
constant both in the Newton’s or the general relativity theory which the
Committee on Data for Science and Technology (CODATA) [6] recommends value as
$G_{N}=6.67430\times 10^{-11}\,\rm{m}^{3}\,\rm{kg}^{-1}\,\rm{s}^{-2}$. This is
strongly justified from the stringent constraints on $G$ possible variation in
time or space, obtained from experiments in lab and solar system, see [7, 8,
9, 10]. Further astrophysical studies corroborated this assumption after
attempts to test deviation from fiducial of a time-varying $G$ using Type Ia
supernovae [11, 12] or Tully-Fisher relation data [13], found consistency with
the gravitational constancy hypothesis.

However, unlike the lab or solar system experiments, those on larger
astronomical scales, do not operate on $G$ itself and the analysis rather
include proxies to mass of interacting bodies, leaving room for possible
degeneracies through which a variation from fiducial $G_{N}$ remains possible.
This is even more the case when we explore cosmological scales at higher
redshifts, far from our local universe. That is why some have attempted to
revisit the different cosmological studies while relaxing the $G$ value in
order to test the impact of such change on the cosmological parameters
constraints and to infer $G$ independently from local measurements. The latter
goal has some theoretical justification, since many modified gravity theories
predict a time-dependent $G$ as an attempt to naturally model the accelerated
expansion of the universe, [14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25,
26, 27, 28, 29, 30, 31, 32, 33, 34, 35].

Another motivation, related to the aforementioned impact on the cosmological
parameters constraints, is justified by the growing evidence of two
discrepancies found lately on the matter fluctuation parameter $\sigma_{8}$
and the Hubble parameter $H_{0}$, between their values constrained from deep
universe probes like the CMB correlations in comparison with those from local
ones, like cluster counts or weak lensing correlations for the $\sigma_{8}$
inference [36, 37] or the Cepheids luminosity distance for $H_{0}$ [38]. There
has been attempts to alleviate the tension for, either $\sigma_{8}$ [39, 40],
or the Hubble parameter, for each alone or for both at the same time (see [41]
and references therein), with relative successes. Especially when combining
with the baryonic acoustic oscillation (BAO) angular distance probe, where it
was shown that almost all the gain in reducing the discrepancy between deep
and local probes vanishes when the BAO constraints were added see e.g. [42].
This was also found by similar attempts to fix the discrepancy with some of
the models we mentioned earlier [19, 20, 43, 44, 45]. Other approaches that
were explored meant to have a transition of $G$ at a particular redshift, see
[46, 47, 48] or one in $H_{0}$ in general [49].

In this work we use cosmological observables like CMB temperature and
polarisation to constrain deviations of $G$ from fiducial but we do not
consider modification to $G$ as a result of a particular modified gravity. We
rather choose to test deviation and extension to the fiducial fixed G N value
with general phenomenological parameterisations and try to account for the
consequences of such parameterisation on the observables even on the
microphysics involved.

Similar studies on the deviation from fiducial $G$ in cosmology have been
considered, for instance: in [50], it was provided a method to constrain the
gravitational constant, showing that a degeneracy between its effect on the
expansion rate and the primordial spectrum can be broken by measuring CMB
polarisation along with its temperature correlations; in [51] were presented
cosmological constraints on deviations of Newton’s constant at large scales,
analysing the cosmic microwave background (CMB) anisotropies and primordial
abundances of light elements synthesised by big bang nucleosynthesis; and in
[52], where CMB data were used together with BBN priors to constrain $G$ in
models of free constant or varying G; whereas in [11], SN data were used to
set a limit on $G$ variation with time. These analysis were repeated later,
with updated measurement of $G$ after the first Planck release [53],
confirming that $G/G_{N}$ is in agreement with being equal to one. Recently,
there have been works using BNN data [54] or adding to the CMB, local data
such as BAO or SNIa measurements, [55].

Here we follow the same idea by considering $G$, within general relativity
theory, as a free parameter allowed to vary from the fiducial $G_{N}$, paying
attention to its impact on all the main physical processes involved in the
cosmological observables. This has a double objectives of, first exploring the
consequences on the G constraints, and second its impact on the aforementioned
discrepancies and the other cosmological parameters. Since it has been shown
that models with late modification do not succeed in fixing the discrepancies,
especially when combining with BAO constrains, we shall, as describe later,
consider parameterisations where $G$ deviates from $G_{N}$ only at early times
before reverting back to its fiducial value after a certain redshift, with the
latter threshold chosen at different times outside the BAO domain presently
measured at redshfits below $z\sim 2.5$.

The outline of the paper is the following. In Sect. 2 we discuss models for
modified $G$ and implications on the cosmological probes. In Sect. 3 we
describe the datasets used and the way modifications to observables were
implemented. In Sect. 4 we present and discuss the results of a Bayesian
analysis constraints and their implications on the cosmological observables,
and we conclude in Sect. 5.

##  2 Variable G : Models and cosmological implications

We introduce a dimensionless parameter $\xi_{\mathrm{g}}$ to quantify the
potential deviation of Newton’s gravitational $G$ from the laboratory-based
measurement $G_{N}$. We consider three phenomenological parameterisation:

  * •

constant case scaling with $\xi_{\mathrm{g}}$

$G=\xi_{\mathrm{g}}\times G_{N}\,;$ (2.1)

  * •

a step like function where the parameter $\xi_{\mathrm{g}}$ is different from
fiducial $G_{N}$, above a certain redshift $z_{tr}$,

$\displaystyle G=\begin{cases}\xi_{\mathrm{g}}\times
G_{N},&~{}\text{for}~{}z>z_{tr};\\\ G_{N},&~{}\text{for}~{}z<z_{tr};\\\
\end{cases}$ (2.2)

  * •

an hyperbolic tangent function, where $\delta_{z}$ is the width of the
transition:

$G=\left[1-\left(1-\xi_{\mathrm{g}}\right)\left(1-0.5\right)\times\tanh\left(\frac{z_{tr}-z}{\delta_{z}}\right)\right]\times
G_{N}\,.$ (2.3)

To show the impact on observables like the CMB spectrum or BAO feature, we
consider, without loose of generality, the first parameterisation. The
Friedmann equation then becomes (working in $c=1$ units):

$H^{2}=\left(\frac{\dot{a}}{a}\right)^{2}=\frac{8\pi}{3}\xi_{\mathrm{g}}^{2}G_{N}\rho_{\rm
tot}$ (2.4)

where $\rho_{\rm tot}$ is the total energy density. This implies that the
expansion rate $H$ satisfies:

$H(a,\xi_{\mathrm{g}})=\xi_{\mathrm{g}}\times H(a)\,,$ (2.5)

so that the shape of $H$ is not changed by $\xi_{\mathrm{g}}$. This serves to
see the consequence on the CMB temperature power spectrum. To show its effect,
we start by writing the temperature perturbation as an integral along the line
of sight over the sources:

$\Delta T_{l}\left(k,\xi_{\mathrm{g}}\right)=\int^{1}_{a}{\rm
d}a\;\tilde{S}\left(k,a,\xi_{\mathrm{g}}\right)e^{ik\cdot\hat{n}r\left(a,\xi_{\mathrm{g}}\right)}\tilde{g}\left(a,\xi_{\mathrm{g}}\right)\,,$
(2.6)

where $S(k,a)$ is the anisotropy source term and $r(a)$ is the distance from
the observer to a point along the line of sight at the scale factor $a$, and
$g(a)$ is the visibility function. Assuming, form the moment,
$\xi_{\mathrm{g}}$ constant, the distance $r(a)$ reads:

$r(a,\xi_{\mathrm{g}})=\int_{a}^{1}\frac{\rm{d}a}{H(a,\xi_{\mathrm{g}})a^{2}}=r(a)/\xi_{\mathrm{g}}$
(2.7)

and the $\xi_{\mathrm{g}}$ parameter is factored out in Eq. (2.6), satisfying
$S(k,a,\xi_{\mathrm{g}})=S(k/\xi_{\mathrm{g}},a)$, as also shown by [50].

Since the CMB angular power spectrum is calculated from $\Delta
T_{l}(k,\xi_{\mathrm{g}})$ using

$C_{l}(\xi_{\mathrm{g}})=\int\frac{{\rm
d}k}{k}P(k_{0})|\Delta_{Tl}(k,\xi_{\mathrm{g}})|^{2}$ (2.8)

where $P(k_{0})$ is the power-law primordial power spectrum, we can choose to
scale the latter by means of a transformation $k^{\prime}=k*\xi_{\mathrm{g}}$,
this will compensate the $\xi_{\mathrm{g}}$ modifications so that the CMB
power spectrum ends up being invariant by our parameterisation of $G$.

This holds if the visibility function is not affected by $\xi_{\mathrm{g}}$.
However, the physics of recombination introduces a preferred timescale. This
happens through the visibility function $g(a)$ written in terms of $\kappa$
(the opacity for Thomson scattering) that is dependent of $\xi_{\mathrm{g}}$
as:

$g(a,\xi_{\mathrm{g}})=-a^{2}H\left(a,\xi_{\mathrm{g}}\right)\frac{{\rm
d}}{{\rm d}a}\exp\left(-\kappa\left(a,\xi_{\mathrm{g}}\right)\right)$ (2.9)

with

$\kappa(a,\xi_{\mathrm{g}})=\sigma_{T}\int_{a}^{1}\frac{n_{e}(a,\xi_{\mathrm{g}})}{H(a,\xi_{\mathrm{g}})a^{2}}~{}{\rm
d}a,$ (2.10)

where $\sigma_{T}$ is the Thomson scattering cross section, $a$ is the scale
factor, with the number of free electrons $n_{e}$ affected by
$\xi_{\mathrm{g}}$ as:

$\frac{dx_{e}}{da}=\frac{C_{r}}{a~{}H(a,\xi_{\mathrm{g}})}\left[\beta(T_{b})(1-x_{e})-n_{\rm
H}\alpha(T_{b})x_{e}^{2}\right]\,.$ (2.11)

Here, $n_{\rm H}$ is the total number density of Hydrogen nuclei,
$x_{e}=n_{e}/n_{\rm H}$ is the ionisation fraction, $T_{b}$ is the baryon
temperature, $\beta(T_{b})$ is the collisional ionisation rate from the ground
state, $\alpha(T_{b})$ the recombination rate to excited states and $C_{r}$ is
the Peebles correction factor to account the presence of non-thermal
Lyman-$\alpha$ resonance photons. We also account for modifications in the
ionisation fraction of Hydrogen $x_{p}$ and Helium $x_{\rm HeII}$. Based on
[56], we implemented the changes of $G$ in the Boltzman code (see Sec. 3) by
solving the following differential equations as a function of
$\xi_{\mathrm{g}}$:

$\displaystyle\frac{{\rm d}x_{p}}{{\rm d}a}$ $\displaystyle=$
$\displaystyle\frac{f_{1}(x_{e},x_{p},n_{\rm H},T_{\rm
M})}{a~{}H(\xi_{\mathrm{g}},a)},$ (2.12) $\displaystyle\frac{{\rm d}x_{\rm
HeII}}{{\rm d}a}$ $\displaystyle=$ $\displaystyle\frac{f_{2}(x_{e},x_{\rm
HeII},n_{\rm H},T_{\rm M})}{a~{}H(\xi_{\mathrm{g}},a)},$ (2.13)
$\displaystyle\frac{{\rm d}T_{\rm M}}{{\rm d}a}$ $\displaystyle=$
$\displaystyle\frac{f_{3}(x_{e},T_{\rm M},T_{\rm
R})}{a~{}H(\xi_{\mathrm{g}},a)}+2aT_{\rm M},$ (2.14)

where $T_{\rm M}$ ($T_{\rm R}$) is the matter (radiation) temperature and
where the specific expressions of $f_{1}$, $f_{2}$ and $f_{3}$, independent of
$\xi_{\mathrm{g}}$ are given in [56].

There is still an effect from the change in the Poisson equation that relates
the gravitational potential $\Phi$ to the density contrast $\delta$

$\nabla^{2}\Phi=-4\pi\xi_{\mathrm{g}}G_{N}\rho\,\delta\,,$ (2.15)

which influences the CMB anisotropy on large scale through the Integrated
Sachs Wolfe effect (ISW). The linear ISW temperature shift along direction
$\hat{n}$ is calculated from the time-dependent gravitational potential $\Phi$

$\frac{\Delta T_{\rm
ISW}}{\overline{T}}({\hat{n}})=-2\int_{a}^{1}a\left(1-f(a)\right)\Phi\left({\hat{n}},a\right)\,{\rm
d}a\,,$ (2.16)

where $f=\mathrm{d}\ln D/\mathrm{d}\ln a$ is the linear growth rate of
structure and $D(a)=\delta(a)/\delta(1)$ is the growth of density contrast
$\delta(a)$. The equation of motion for $\delta(a)$ is:

$\delta^{\prime\prime}+\left(\frac{3}{a}+\frac{E^{\prime}}{E}\right)\delta^{\prime}-\frac{3}{2}\frac{\Omega_{\mathrm{m},0}}{a^{5}E^{2}}\delta=0\
$ (2.17)

where prime denotes derivation with respect to a and
$E^{2}(a)=H^{2}(a,\xi_{\mathrm{g}})/H^{2}_{0}$. Thus, $H_{0}$,
$\Omega_{\mathrm{m},0}$ and, $\xi_{\mathrm{g}}$ will be constrained from CMB
through this effect.

The variation of $G$ will also affect the polarisation of the CMB, allowing us
to use the polarisation spectrum and cross correlations with the temperature
fluctuations spectrum of the CMB to constrain $\xi_{\mathrm{g}}$.

To understand the effect of $\xi_{\mathrm{g}}$ on the polarisation, let us
consider that the degree of linear polarisation, $\Delta P$, is defined in
terms of the Stokes parameters $Q$ and $U$ of the CMB radiation. Choosing two
orthogonal directions basis, into which the intensity is projected, so that
$U=0$, we can limit the study to the amplitude of the $Q$ Stokes parameter
produced by a single Fourier mode $k$ [57] yielding:

$Q\propto c_{s}\,k\,\delta\tau_{D}\sin(kc_{s}\tau_{D})g(a,\xi_{\mathrm{g}})$
(2.18)

where $g(a,\xi_{\mathrm{g}})$ is the visibility function, $c_{s}$ is the
photon-baryon sound speed, $\tau_{D}$ is the conformal time ( $\tau=\int
dt~{}a_{0}/a$) when $G$ peaks and $\delta\tau_{D}$ its width. The extra
$\delta\tau_{D}$ in Eq. (2.18) implies that, if the visibility function is
wider, the photons will travel on average longer between their last
scatterings, enhancing the quadrupole anisotropy, thus increasing the
polarisation. This effect will induce a characteristic scale $k^{*}$, above
which the effects on the polarization is larger.

A change in the value of $G$ could also impact the reionisation process
influencing the CMB observations. During the reionisation epoch, CMB photons
are Thomson scattered by the free electrons produced, which suppress the
amplitude of the observed primordial anisotropies. Additionally, large-scale
polarisation of the CMB radiation could be produced above the earlier ones
during recombination. However the reionisation process is complicated and
includes many astrophysical variables whose study is outside the scope of this
work. Therefore, we consider only model with simple almost instant
parameterisazation so that any influence of $G$ on time and volume will not
manifest, as well as any gravitational effect degenerate with $G$ will not
influence the physics of reionisation from forming stars. In practice the
ionisation fraction $x_{e}$ is parameterised as a $\tanh$ function with a
width $\Delta\,z=0.5$ around the reionisation redshift [58].

Figure 1: CMB temperature angular power spectrum for different
$\xi_{\mathrm{g}}=G/G_{N}$. Top left panel is obtained with a constant
$\xi_{\mathrm{g}}$, whereas the top right panel is obtained with
$\xi_{\mathrm{g}}\neq 1$ for a redshift transition from $G_{N}$ at $z_{tr}>8$.
Bottom left panel is obtained with $\xi_{\mathrm{g}}\neq 1$ for $z_{tr}>30$,
whereas the bottom right panel is obtained with $\xi_{\mathrm{g}}\neq 1$ with
a transition width $\pm 10$ around $z_{tr}\sim 30$.

##  3 Datasets and analyse method

Figure 2: CMB polarisation angular power spectrum for different
$\xi_{\mathrm{g}}=G/G_{N}$ and their residuals with respect to fiducial
$G_{N}$. Top: Obtained for the same change in $G$ at all redshifts. Middle:
Obtained for a redshift transition from $G_{N}$ at $z_{tr}>8$. Bottom:
Obtained for a redshift transition from $G_{N}$ at $z_{tr}>30$.

In order to implement our model, we modify the cosmological solver CLASS [59]
that is able of calculating the temperature-temperature (TT) and polarisation-
polarisation (EE) angular correlations and their cross correlations (TE) of
the CMB, as well as the cosmological distances and the redshift epoch of the
baryonic acoustic oscillation early occurrence, both needed to compare with
galaxy clustering BAO measurements.

We then perform a Monte Carlo Markov Chain (MCMC) analysis to constrain
deviations of $G$ from fiducial CODATA value using MontePython [60], a tool
that embeds CLASS functions in the Planck likelihoods as well those needed for
the BAO likelihood. For that, we use Planck $C^{\rm TT}_{\ell}$, $C^{\rm
TE}_{\ell}$, and $C^{\rm EE}_{\ell}$ at high $\ell$ and TT and EE correlations
at low $\ell$ from the final mission survey release of Planck (2018) [1], as
well as the BAO measurements from Beutler et al. (2011) [61], Ross et al.
(2015) [62] and Alam et al. (2017) [63].

Besides the free constant case, we choose the redshift threshold at which the
model reverts back to constant fixed $G$ case, which allows us to distinguish
between the different effects. We choose two different thresholds: the first
one is $z_{tr}\sim 30$, which occurs later than the recombination epoch
(around the beginning of formation of structures) but well before the
reionisation epoch; the second one is around $z_{tr}\sim 8$, happening at the
latter epoch but well before the BAO local times, since this last epoch would
have been covered by the free constant case that goes close to our present
times around $z\sim 0.0025$. Even with the free constant case we do not
consider redshifts below the latter value, to try evade solar and
astrophysical constraints from non linear structure formation, since also
$z\sim 0.0025$ translates into the size of $\sim 10$ Mpc, close to that of our
local galaxy cluster. In all these models the transition back from modified
$G$ to the fiducial $G_{N}$ happens sharply. This could enhance the ISW
effect. To test the impact of such effect, we consider the same binning but
with a smooth transition with a large width following the model described by
Eq. (2.2) in Sect. 2.

To justify the above choices, we illustrate their effects on our observables.
In Fig. 1, we show the temperature correlations angular power spectrum $C^{\rm
TT}_{\ell}$s for some of the different aferomentioned cases. On top left
panel, we observe that the free constant case ($\xi_{\mathrm{g}}\neq 1$)
affects the angular power spectrum at small scales. For the other cases, the
effects are prominent at larger scales. We, thus, expect that the latter will
put more constraints on deviations of $G$ from the fiducial. Furthermore, we
observe that the dynamical $G$ affects strongly the ISW scales (see Sect. 2),
contrary to the case of constant $G$ where the variation has a monotonic
increase. However, the influence of the dynamical $G$ gets smaller (in
average) when the redshfit threshold gets close to late times (bottom left and
bottom right panels). Thus, in order to take this effect into account, we
compare the MCMC chains with and without the smooth transition.

In Fig. 2, we show the polarisation correlations angular power spectrum
$C_{\ell}^{\rm EE}$s for the three cases: one with the change in $G_{N}$ for
all redshifts, then, in the middle, with a transition threshold below
$z_{tr}\sim 30$ and, at bottom, for $z_{tr}\sim 8$. As described in Sec. 2, we
observe, especially in the last two panels, that the difference in amplitude
of the $C_{\ell}^{\rm EE}$s is significant and it could reach twice its value
even for a $10\%$ modification on value of $G$. This suggests that the probe
could put strong constraints on the value of $G$.

Figure 3: Confidence contours (68 and 95%) for cosmological parameters in the
fixed CODATA $G_{N}$ case in comparison with a free constant
$\xi_{\mathrm{g}}\neq 1$ using CMB data only or combined with BAO data.

##  4 Results and discussion

Figure 4: Confidence contours (68 and 95%) for cosmological parameters in the
fixed CODATA $G_{N}$ case in comparison with a free constant
$\xi_{\mathrm{g}}\neq 1$ for a redshift transition from $G_{N}$ at $z_{tr}>30$
using CMB data only or combined with BAO data.

In Fig. 3, we show the confidence contours of the cosmological parameters
using the CMB temperature and polarisation correlations from Planck 2018
datasets with the free $G$ constant case, together with the case where $G$ is
fixed to the CODATA value. We usually show the confidence values obtained at
the 68% level unless otherwise stated. We quantify the deviation with the
parameter $\xi_{\mathrm{g}}=G/G_{N}$, where $\xi_{\mathrm{g}}=1.0$ corresponds
to $G$ taking the fiducial CODATA value. We first observe that the fiducial
value of $G$ is within the inferred $\xi_{\mathrm{g}}$, with the maximum
likelihood at $1.0\pm$0.04. This is the consequence of $\xi_{\mathrm{g}}\neq
1.0$ effect on the small scales of the CMB spectrum as illustrated in Sec. 3
and in Fig. 1, as a result of the correlation between $n_{s}$ and
$\xi_{\mathrm{g}}$. However the discrepancy with $\sigma_{8}$ or $H_{0}$
remains high as the constraints do not change significantly with respect to
the fixed $G_{N}$ case. We neither observe a change in $\xi_{\mathrm{g}}$
constraints when we combine CMB with BAO probes. This is due to the fact that
a different constant fiducial value of $G$ only changes the cosmic clock
without altering the BAO observable.

Figure 5: Confidence contours (68 and 95%) for cosmological parameters in the
fixed CODATA $G_{N}$ case in comparison with a free constant
$\xi_{\mathrm{g}}\neq 1$ for a redshift transition from $G_{N}$ at $z_{tr}>8$
using CMB data only or combined with BAO data.

In Fig. 4, we show the confidence contours when we allow $\xi_{\mathrm{g}}$ to
vary for $z_{tr}>30$. We observe that, using Planck 2018 only, the constraints
on $G$ are similar to those of the free constant $\xi_{\mathrm{g}}$ case, with
the maximum likelihood at $\sim$ 1.0 and a 1$\sigma$ variation of $\pm$0.03.
The reason is that, at lower $\ell$, the impact of $\xi_{\mathrm{g}}$ on TT
correlation (Fig. 1 bottom left) is lower than the one observed with the free
constant $\xi_{\mathrm{g}}$ on all redshifts case, while the opposite is seen
for the effect on the polarisation correlations as shown in Fig. 2. For the
Hubble parameter, we observe that the discrepancy is alleviated as the
constraints become much broader than the constant $G$ case, with the local
Hubble value of $\sim 73-74$ now falling within less than 1$\sigma$ of CMB
inferred one. The $\sigma_{8}$ discrepancy is reduced in this parameterisation
as well, and the value corresponding to that from local probes is reached at
the 2$\sigma$ level. This is expected from the non monotonic effect of
$\xi_{\mathrm{g}}$ on the ISW level seen in Sec. 3 in Fig. 1. However, the
combination with the BAO datasets restores the discrepancy on both
$\sigma_{8}$ and $H_{0}$. The BAO addition also tightens the constraints on
$\xi_{\mathrm{g}}$ to $\pm 0.03$ at 3$\sigma$, as a consequence of
$\xi_{\mathrm{g}}$ that now needs to accommodate this paramerterization with
two observations that disfavor a transition between the two redshifts.

Figure 6: Confidence contours (68 and 95%) for cosmological parameters for a
free constant $\xi_{\mathrm{g}}\neq 1$ with a sharp transition from $G_{N}$
after $z_{tr}>30$ in comparison with a transition width $\pm 10$ around the
same redshift threshold using CMB data only or combined with BAO data.

Assuming a value of $z_{tr}\sim 8$, we observe in Fig. 5 that the constraints
on $\xi_{\mathrm{g}}$ stay almost the same or get slightly tighter despite
that $\xi_{\mathrm{g}}$ is allowed to deviate from its fiducial value for a
larger range of redshifts. The reason here is that the sensitivity of the CMB
polarisation spectrum to $\xi_{\mathrm{g}}$ is weaker than the $z_{tr}\sim 30$
case, as seen in Sect. 3 in Fig. 2. The same discussion applies to
$\sigma_{8}$ or $H_{0}$ constraints with the only difference that now BAO data
are unable to completely restore the discrepancy as in the previous case
because $G$ is allowed to vary for a much longer period.

Figure 7: Confidence contours (68 and 95%) for cosmological parameters in the
fixed CODATA $G_{N}$ case in comparison with a free constant
$\xi_{\mathrm{g}}\neq 1$ and a free redshift transition threshold from $G_{N}$
using CMB data only or combined with BAO data. Figure 8: Confidence contours
(68 and 95%) using CMB data only or combined with BAO data for cosmological
parameters in the case of a free $\alpha_{s}$ and a fixed CODATA $G_{N}$ in
comparison with a free $\alpha_{s}$ along with $\xi_{\mathrm{g}}\neq 1$.
Figure 9: Confidence contours (68 and 95%) using CMB data only or combined
with BAO data for cosmological parameters for a free $\xi_{\mathrm{g}}$ with a
free redshift threshold and a free $\alpha_{s}$ together with a free
$\alpha_{s}$ but with a fixed CODATA $G_{N}$.

So far we have considered models that deviate sharply at some chosen redshift;
however it is interesting to consider smooth transitions that mimics a linear
variation around the threshold redshift. In Fig. 6 we show the results
combining CMB and BAO data when using a $\tanh$ function which allows for the
transition at $z_{tr}\sim 30$. We observe no substantial impact on all the
parameters when we combine CMB to BAO data, indicating that the ISW effect is
not enough to constrain deviations of $G$. We also verified (without showing
here the results) the case where $z_{tr}\sim 8$ and we found an even smaller
difference between the confidence contours of the sharp transition
parameterisation and the smoothed one.

Finally, since we found that the constraints could change with the redshift
transition threshold, we decided, in a more model independent approach, to
repeat the bayesian analysis with $z_{tr}$ as an additional free parameter.
Using Planck 2018 data alone, we observe in Fig. 7 that the $\xi_{\mathrm{g}}$
maximum likelihood coincides with the fiducial $G_{N}$ one with much tighter
constraints then the other cases to $\pm 0.02$. The $H_{0}$ and $\sigma_{8}$
tensions are alleviated but after adding BAO data, the discrepancy on $H_{0}$
is restored and the ability to fix the $\sigma_{8}$ discrepancy is reduced.
The addition of BAO also tighten the constraints on $\xi_{\mathrm{g}}$ with
68$\%$ confidences values at $\pm 0.01$.

Besides the redshift threshold influence, we noticed that the spectral index
$n_{s}$ shows a correlation with $\xi_{\mathrm{g}}$ in all the cases
considered in this work. Since the CMB probe puts strong constraints on
$n_{s}$, we decided to relax the latter by allowing a scale dependency through
the spectral index running parameter $\alpha_{s}={\rm d}\rm{n}_{s}/{\rm
d}\rm{ln}k$. In Figs. 8 and 9 we show the confidence contours on the
cosmological parameters including a free running spectral index different from
the null fiducial value in the fixed CODATA $G_{N}$ case along with two of the
parmetrisations for $\xi_{\mathrm{g}}$ with opposite behavior in terms of
their redshift transition value: the first considering a free constant
$\xi_{\mathrm{g}}$ different from fiducial at all redhsifts (i.e. with a
threshold fixed to $z_{tr}\sim 0.0$), while the second allows a free $z_{tr}$
on a very large range spanning from nowadays until the recombination epoch.
For the case of a constant free $G$ and a free $\alpha_{s}$ (Fig. 8), we
observe that the running is correlated with the $\xi_{\mathrm{g}}$ with the
later confidence contours tighter than the case with vanishing $\alpha_{s}$
(represented by the dashed vertical lines). This comes from the fact that the
data rather prefers a negative $\alpha_{s}$ instead of larger deviations of
$\xi_{g}$. The latter constraints are however slightly broader than the
$\xi_{g}=1$ case, resulting in a larger constraints on the Hubble parameter
with respect to the previous case with a vanishing $\alpha_{s}$, represented
by the dashed vertical lines in the $H$ single likelihood box. We finally note
that the confidence contours remain almost the same for all the parameters
when combining with BAO. In the case of allowing the redshift transition
threshold also to vary (Fig. 9), we observe that the introduction of the
running parameter does not have much impact on the different constraints,
since already in the previous case relaxing $z_{tr}$ has showed a preference
for a tighter $\xi_{g}$ though the deviation of the $\alpha_{s}$ from the null
fiducial value translates into slightly wider constraints on the $\xi_{g}$,
$\sigma_{8}$ or $H_{0}$ (with the previous represented by the vertical dashed
lines in each box), with the last two parameters again restored to their
fiducial values when we combine with BAO.

##  5 Conclusions

In this paper we have studied the impact of a varying $G$ onto some of the
cosmological observables at both macro or micro physical scales. For that, we
used CMB temperature and polarisation correlations data alone or in
combination with BAO distance measurements from galaxy clustering. We
performed a MCMC analysis allowing a constant free $G$ different from fiducial
or a dynamical $G$ that switches value at a given redshift. The objective of
this analysis was to update the constraints on the deviation of $G$ from
$G_{N}$ and investigate whether the discrepancy on the matter fluctuation
$\sigma_{8}$ and $H_{0}$ parameters could be alleviated by these
parameterisations.

For a free constant $G$ case, we found that this parameterisation has no
effect on fixing the $\sigma_{8}$ or $H_{0}$ discrepancy while the
$\xi_{\mathrm{g}}=G/G_{N}$ is constrained to 1.0$\pm 0.04$. Then, to separate
constraints coming from lower redshifts from the ones at recombination epoch,
we considered a dynamical $G$, with two redshift threshold at which $G$
transits from $G_{N}$, one at high redshift $z_{tr}\sim 30$ and another at
$z_{tr}\sim 8$. With both redshift transitions, we found that we were able of
alleviating the tensions on $H_{0}$ and $\sigma_{8}$ only if we use the CMB
temperature and polarisation correlations alone and at the expense of wider
constraints, while the discrepancies were restored when we combine with BAO in
the higher redshift transition option or only reduced with the lower redshift
transition case. We also found that the constraints remain almost unchanged
whether we consider a sharp or a smooth wide transition of the $G$ from its
fiducial value at the given redshift, as an indication that the ISW effect is
not enough constrained by CMB data due to the large cosmic variance present at
large scales in the CMB correlations measured.

Finally, we considered two additional more general cases: one allowing a free
redshift threshold in a further model independent approach, and the other
introducing another degree of freedom by a non vanishing running of the
spectral index, since the later was found in correlation with
$\xi_{\mathrm{g}}$. In the first case, we found that we are still not able of
alleviating the tension on the Hubble constant when the BAO are combined with
CMB data, while the discrepancy on $\sigma_{8}$ is reduced and
$\xi_{\mathrm{g}}$ is constrained to 1.0$\pm 0.01$. In the case where we
allowed a free running spectral index $\alpha_{s}$, we found that the
constraints on $\xi_{\mathrm{g}}$ are tightened with the data rather
preferring a negative $\alpha_{s}$, with only a small improvement of
alleviating the discrepancies on $H_{0}$ and $\sigma_{8}$ was found. When we
considered a free redshift transition value along with a free running of the
spectral index, we observed that the data is now preferring a stronger
correlation $\xi_{\mathrm{g}}-\alpha$ than $\xi_{\mathrm{g}}-n_{s}$ allowing
further deviation of $\xi_{\mathrm{g}}$ from the fiducial and an alleviation
of the tensions but only if BAO observations are not included. We conclude
that there remains a preference to the fixed CODATA $G_{N}$ model and that the
different parameterisations we considered are not able of alleviating the
Hubble and the matter fluctuation parameter discrepancies.

## Acknowledgements

DS acknowledges financial support from the Fondecyt Regular project number
1200171. ZS acknowledge fruitful discussions with Brahim Lamine.

## References

  * [1] Planck Collaboration, N. Aghanim et. al., Planck 2018 results. VI. Cosmological parameters, Astron. Astrophys. 641 (2020) A6, [arXiv:1807.0620]. [Erratum: Astron.Astrophys. 652, C4 (2021)].
  * [2] KiDS Collaboration, M. Asgari et. al., KiDS-1000 Cosmology: Cosmic shear constraints and comparison between two point statistics, Astron. Astrophys. 645 (2021) A104, [arXiv:2007.1563].
  * [3] A. de Mattia et. al., The Completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: measurement of the BAO and growth rate of structure of the emission line galaxy sample from the anisotropic power spectrum between redshift 0.6 and 1.1, Mon. Not. Roy. Astron. Soc. 501 (2021), no. 4 5616–5645, [arXiv:2007.0900].
  * [4] DES Collaboration, T. M. C. Abbott et. al., Dark Energy Survey year 1 results: Cosmological constraints from galaxy clustering and weak lensing, Phys. Rev. D 98 (2018), no. 4 043526, [arXiv:1708.0153].
  * [5] SNLS Collaboration, A. Conley et. al., Supernova Constraints and Systematic Uncertainties from the First 3 Years of the Supernova Legacy Survey, Astrophys. J. Suppl. 192 (2011) 1, [arXiv:1104.1443].
  * [6] P. J. Mohr, D. B. Newell, and B. N. Taylor, CODATA Recommended Values of the Fundamental Physical Constants: 2014, Rev. Mod. Phys. 88 (2016), no. 3 035009, [arXiv:1507.0795].
  * [7] J. D. Anderson, G. Schubert, V. Trimble, and M. R. Feldman, Measurements of Newton’s gravitational constant and the length of day, EPL (Europhysics Letters) 110 (Apr., 2015) 10002, [arXiv:1504.0660].
  * [8] S. Schlamminger, J. H. Gundlach, and R. D. Newman, Recent measurements of the gravitational constant as a function of time, Phys. Rev. D 91 (2015), no. 12 121101, [arXiv:1505.0177].
  * [9] Q. Li, C. Xue, J.-P. Liu, J. fei Wu, S. Yang, C.-G. Shao, L.-D. Quan, W.-H. Tan, L.-C. Tu, Q. Liu, H. Xu, L. Liu, Q.-L. Wang, Z. Hu, Z. bing Zhou, P. Luo, S. Wu, V. Milyukov, and J. Luo, Measurements of the gravitational constant using two independent methods, Nature 560 (2018) 582–588.
  * [10] D.-C. Dai, Variance of Newtonian constant from local gravitational acceleration measurements, Phys. Rev. D 103 (2021), no. 6 064059, [arXiv:2103.1115].
  * [11] J. Mould and S. Uddin, Constraining a possible variation of G with Type Ia supernovae, Publ. Astron. Soc. Austral. 31 (2014) 15, [arXiv:1402.1534].
  * [12] D. Sapone, S. Nesseris, and C. A. P. Bengaly, Is there any measurable redshift dependence on the SN Ia absolute magnitude?, Phys. Dark Univ. 32 (2021) 100814, [arXiv:2006.0546].
  * [13] G. Alestas, I. Antoniou, and t. Perivolaropoulos, Hints for a gravitational constant transition in Tully-Fisher data, arXiv:2104.1448.
  * [14] E. A. Milne, Relativity, gravitation and world-structure.  1935\.
  * [15] P. A. M. Dirac, The Cosmological constants, Nature 139 (1937) 323.
  * [16] C. Brans and R. H. Dicke, Mach’s Principle and a Relativistic Theory of Gravitation, Physical Review 124 (Nov., 1961) 925–935.
  * [17] C. Wetterich, Variable gravity Universe, Phys. Rev. D 89 (2014), no. 2 024005, [arXiv:1308.1019].
  * [18] C. Burrage and J. Dombrowski, Constraining the cosmological evolution of scalar-tensor theories with local measurements of the time variation of G, JCAP 07 (2020) 060, [arXiv:2004.1426].
  * [19] M. Ballardini, M. Braglia, F. Finelli, D. Paoletti, A. A. Starobinsky, and C. Umiltà, Scalar-tensor theories of gravity, neutrino physics, and the $H_{0}$ tension, JCAP 10 (2020) 044, [arXiv:2004.1434].
  * [20] M. Braglia, M. Ballardini, W. T. Emond, F. Finelli, A. E. Gümrükçüoğlu, K. Koyama, and D. Paoletti, Larger value for h0 by an evolving gravitational constant, Physical Review D (2020).
  * [21] C. M. Will, The Confrontation between General Relativity and Experiment, Living Rev. Rel. 17 (2014) 4, [arXiv:1403.7377].
  * [22] J. C. Fabris, T. Gomes, J. D. Toniato, and H. Velten, Newtonian-like gravity with variable G, European Physical Journal Plus 136 (Feb., 2021) 143, [arXiv:2009.0443].
  * [23] T. Kaluza, Zum Unitätsproblem der Physik, Sitzungsber. Preuss. Akad. Wiss. Berlin (Math. Phys. ) 1921 (1921) 966–972, [arXiv:1803.0861].
  * [24] O. Klein, The Atomicity of Electricity as a Quantum Theory Law, Nature 118 (1926) 516.
  * [25] N. Arkani-Hamed, S. Dimopoulos, and G. R. Dvali, The Hierarchy problem and new dimensions at a millimeter, Phys. Lett. B 429 (1998) 263–272, [hep-ph/9803315].
  * [26] L. Randall and R. Sundrum, A Large mass hierarchy from a small extra dimension, Phys. Rev. Lett. 83 (1999) 3370–3373, [hep-ph/9905221].
  * [27] L. Randall and R. Sundrum, An Alternative to compactification, Phys. Rev. Lett. 83 (1999) 4690–4693, [hep-th/9906064].
  * [28] G. R. Dvali, G. Gabadadze, and M. Porrati, 4-D gravity on a brane in 5-D Minkowski space, Phys. Lett. B 485 (2000) 208–214, [hep-th/0005016].
  * [29] T. Damour, G. W. Gibbons, and C. Gundlach, Dark Matter, Time Varying $G$, and a Dilaton Field, Phys. Rev. Lett. 64 (1990) 123–126.
  * [30] A. a. J. S. Capistrano, L. A. Cabral, J. A. P. F. Marão, and C. H. Coimbra-Araújo, Analysis on the evolution of $\sigma_{8}(z)$ from Linear Nash-Greene fluctuations, arXiv:2009.0216.
  * [31] K. A. Bronnikov, On variations of $G$ in the geometric scalar theory of gravity, Eur. Phys. J. C 80 (2020), no. 5 434, [arXiv:2004.0123].
  * [32] E. T. Hanımeli, B. Lamine, A. Blanchard, and I. Tutusaus, Time-dependent $G$ in Einstein’s equations as an alternative to the cosmological constant, Phys. Rev. D 101 (2020), no. 6 063513, [arXiv:1910.0832].
  * [33] S. Nesseris, G. Pantazis, and L. Perivolaropoulos, Tension and constraints on modified gravity parametrizations of $G_{\textrm{eff}}(z)$ from growth rate and Planck data, Phys. Rev. D 96 (2017), no. 2 023542, [arXiv:1703.1053].
  * [34] M. Ballardini, F. Finelli, and D. Sapone, Cosmological constraints on Newton’s gravitational constant, arXiv:2111.0916.
  * [35] T. Abadi and E. D. Kovetz, Can conformally coupled modified gravity solve the Hubble tension?, Phys. Rev. D 103 (2021), no. 2 023530, [arXiv:2011.1385].
  * [36] H. Hoekstra, R. Herbonnet, A. Muzzin, A. Babul, A. Mahdavi, M. Viola, and M. Cacciato, The Canadian Cluster Comparison Project: detailed study of systematics and updated weak lensing masses, Mon. Not. Roy. Astron. Soc. 449 (2015), no. 1 685–714, [arXiv:1502.0188].
  * [37] Planck Collaboration, P. A. R. Ade et. al., Planck 2013 results. XXIX. The Planck catalogue of Sunyaev-Zeldovich sources, Astron. Astrophys. 571 (2014) A29, [arXiv:1303.5089].
  * [38] A. G. Riess, S. Casertano, W. Yuan, L. M. Macri, and D. Scolnic, Large Magellanic Cloud Cepheid Standards Provide a 1% Foundation for the Determination of the Hubble Constant and Stronger Evidence for Physics beyond $\Lambda$CDM, Astrophys. J. 876 (2019), no. 1 85, [arXiv:1903.0760].
  * [39] Z. Sakr, S. Ilić, A. Blanchard, J. Bittar, and W. Farah, Cluster counts: Calibration issue or new physics?, Astron. Astrophys. 620 (2018) A78, [arXiv:1803.1117].
  * [40] S. Ilić, Z. Sakr, and A. Blanchard, Cluster counts. II. Tensions, massive neutrinos, and modified gravity, Astron. Astrophys. 631 (2019) A96, [arXiv:1908.0016].
  * [41] E. Di Valentino, O. Mena, S. Pan, L. Visinelli, W. Yang, A. Melchiorri, D. F. Mota, A. G. Riess, and J. Silk, In the realm of the Hubble tension—a review of solutions, Class. Quant. Grav. 38 (2021), no. 15 153001, [arXiv:2103.0118].
  * [42] K. Jedamzik, L. Pogosian, and G.-B. Zhao, Why reducing the cosmic sound horizon alone can not fully resolve the Hubble tension, Commun. in Phys. 4 (2021) 123, [arXiv:2010.0415].
  * [43] A. a. J. S. Capistrano, Constraints on $\sigma_{8}$ and degeneracies from linear Nash-Greene perturbations in subhorizon scale, Eur. Phys. J. C 80 (2020), no. 9 898.
  * [44] J. Sola, A. Gomez-Valent, J. de Cruz Perez, and C. Moreno-Pulido, Brans-Dicke cosmology with a $\Lambda$\- term: a possible solution to $\Lambda$CDM tensions, arXiv e-prints (June, 2020) arXiv:2006.04273, [arXiv:2006.0427].
  * [45] R. P. Gupta, Effect of evolutionary physical constants on type-1a supernova luminosity, arXiv:2112.1065.
  * [46] L. Kazantzidis and L. Perivolaropoulos, Hints of a Local Matter Underdensity or Modified Gravity in the Low $z$ Pantheon data, Phys. Rev. D 102 (2020), no. 2 023520, [arXiv:2004.0215].
  * [47] V. Marra and L. Perivolaropoulos, Rapid transition of Geff at zt$\simeq$0.01 as a possible solution of the Hubble and growth tensions, Phys. Rev. D 104 (2021), no. 2 L021303, [arXiv:2102.0601].
  * [48] G. Alestas, D. Camarena, E. Di Valentino, L. Kazantzidis, V. Marra, S. Nesseris, and L. Perivolaropoulos, Late-transition vs smooth $H(z)$ deformation models for the resolution of the Hubble crisis, arXiv:2110.0433.
  * [49] M. G. Dainotti, B. De Simone, T. Schiavone, G. Montani, E. Rinaldi, and G. Lambiase, On the Hubble constant tension in the SNe Ia Pantheon sample, Astrophys. J. 912 (2021), no. 2 150, [arXiv:2103.0211].
  * [50] O. Zahn and M. Zaldarriaga, Probing the Friedmann equation during recombination with future CMB experiments, Phys. Rev. D 67 (2003) 063002, [astro-ph/0212360].
  * [51] K.-i. Umezu, K. Ichiki, and M. Yahiro, Cosmological constraints on Newton’s constant, Phys. Rev. D 72 (2005) 044010, [astro-ph/0503578].
  * [52] S. Galli, A. Melchiorri, G. F. Smoot, and O. Zahn, From Cavendish to PLANCK: Constraining Newton’s Gravitational Constant with CMB Temperature and Polarization Anisotropy, Phys. Rev. D 80 (2009) 023508, [arXiv:0905.1808].
  * [53] Y. Bai, J. Salvado, and B. A. Stefanek, Cosmological Constraints on the Gravitational Interactions of Matter and Dark Matter, JCAP 10 (2015) 029, [arXiv:1505.0478].
  * [54] J. Alvey, N. Sabti, M. Escudero, and M. Fairbairn, Improved BBN Constraints on the Variation of the Gravitational Constant, Eur. Phys. J. C 80 (2020), no. 2 148, [arXiv:1910.1073].
  * [55] K. Wang and L. Chen, Constraints on Newton’s constant from cosmological observations, Eur. Phys. J. C 80 (2020), no. 6 570, [arXiv:2004.1397].
  * [56] S. Seager, D. D. Sasselov, and D. Scott, A new calculation of the recombination epoch, Astrophys. J. Lett. 523 (1999) L1–L5, [astro-ph/9909275].
  * [57] M. Zaldarriaga and D. D. Harari, Analytic approach to the polarization of the cosmic microwave background in flat and open universes, Phys. Rev. D 52 (1995) 3276–3287, [astro-ph/9504085].
  * [58] A. Lewis, Cosmological parameters from WMAP 5-year temperature maps, Phys. Rev. D 78 (2008) 023002, [arXiv:0804.3865].
  * [59] J. Lesgourgues, The Cosmic Linear Anisotropy Solving System (CLASS) I: Overview, ArXiv e-prints (Apr., 2011) [arXiv:1104.2932].
  * [60] B. Audren, J. Lesgourgues, K. Benabed, and S. Prunet, Conservative Constraints on Early Cosmology: an illustration of the Monte Python cosmological parameter inference code, JCAP 1302 (2013) 001.
  * [61] F. Beutler, C. Blake, M. Colless, D. H. Jones, L. Staveley-Smith, L. Campbell, Q. Parker, W. Saunders, and F. Watson, The 6dF Galaxy Survey: Baryon Acoustic Oscillations and the Local Hubble Constant, Mon. Not. Roy. Astron. Soc. 416 (2011) 3017–3032, [arXiv:1106.3366].
  * [62] A. J. Ross, L. Samushia, C. Howlett, W. J. Percival, A. Burden, and M. Manera, The clustering of the SDSS DR7 main Galaxy sample – I. A 4 per cent distance measure at $z=0.15$, Mon. Not. Roy. Astron. Soc. 449 (2015), no. 1 835–847, [arXiv:1409.3242].
  * [63] BOSS Collaboration, S. Alam et. al., The clustering of galaxies in the completed SDSS-III Baryon Oscillation Spectroscopic Survey: cosmological analysis of the DR12 galaxy sample, Mon. Not. Roy. Astron. Soc. 470 (2017), no. 3 2617–2652, [arXiv:1607.0315].


# An Adversarial Active Sampling-based Data Augmentation Framework for
Manufacturable Chip Design

Mingjie Liu1,2 Haoyu Yang1 Zongyi Li1,3 Kumara Sastry1 Saumyadip Mukhopadhyay1
Selim Dogru1 Anima Anandkumar1,3 David Z. Pan2 Brucek Khailany1 Haoxing Ren1
1 Nvidia Corporation 2 The University of Texas at Austin 3 California
Institute of Technology
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Lithography modeling is a crucial problem in chip design to ensure a chip
design mask is manufacturable. It requires rigorous simulations of optical and
chemical models that are computationally expensive. Recent developments in
machine learning have provided alternative solutions in replacing the time-
consuming lithography simulations with deep neural networks. However, the
considerable accuracy drop still impedes its industrial adoption. Most
importantly, the quality and quantity of the training dataset directly affect
the model performance. To tackle this problem, we propose a litho-aware data
augmentation (LADA) framework to resolve the dilemma of limited data and
improve the machine learning model performance. First, we pretrain the neural
networks for lithography modeling and a gradient-friendly StyleGAN2 generator.
We then perform adversarial active sampling to generate informative and
synthetic in-distribution mask designs. These synthetic mask images will
augment the original limited training dataset used to finetune the lithography
model for improved performance. Experimental results demonstrate that LADA can
successfully exploits the neural network capacity by narrowing down the
performance gap between the training and testing data instances.

##  1 Introduction

The advancement of semiconductor industry has enabled rapid development of AI
and deep learning technologies, which in turn offers great opportunities for
novel chip design methodology, enabling faster design turn-around-time, better
PPA (power, performance and area) and higher yield [13, 14, 21, 5, 10].
Particularly, recent researches have demonstrated efficacy using reinforcement
learning to place circuit components (macros and standard cells) on to chip
canvas [13, 5], which is one of the time consuming phases in chip design flow.

In this paper, we focus on the lithography modeling, a critical problem in
chip design and manufacturing flow, which has been a very active area for
machine learning applications since 2010s [6, 12, 17]. Lithography modeling
computes the patterns (resist image) of a chip design (mask image) on the
silicon wafer without going through real manufacturing process. It allows
designers to find potential circuit manufacture failures and conduct necessary
post design optimization prior to manufacturing. Lithography modeling
traditionally consists of two stages [11]. The optical modeling outputs the
intensity (aerial image) of the light beams that are projected on the silicon
wafer, which is given by the weighted convolution between the mask image and a
set of lithography kernels. The resist modeling then applies thresholds on the
intensity map and obtains the final resist image on the wafer.

However, legacy approaches are time consuming to compute complex physical
models, normally taking couple of seconds to minutes. To reduce the design
turn-around-time, recent researchers have investigated the possibility of
using deep neural network models as an alternative of physical models,
reducing the computing runtime by nearly 3 orders of magnitudes. Commonly used
model architectures are UNet-backboned image-to-image translation networks
[16, 3, 24, 23], which can make fast resist image predictions with moderate
accuracy loss. State-of-art dual-band optics-inspired neural networks (DOINN)
[22] further integrates lithography physics into the neural network design and
improves the prediction accuracy. However the quality gap between the machine
predicted resist image and the real physical simulated results still prohibits
learning-based solutions in the chip manufacturing flow. Resolving this
challenge is not trivial because of the limited availability of high quality
data.

To address data limitation, we present a litho-aware data-augmentation
framework (LADA) that takes advantage of adversarial data generation [18, 1],
active learning [15, 26], and modern generative adversarial network (GAN)
design (Figure 1). Specifically we want to generate realistic in-distribution
data with increased data efficiency, such that the synthesized data are most
informative for model improvement. LADA is designed with an end-to-end data
framework where a pretrained GAN generator and a lithography modeling network
are connected. The generator enforces in-distribution data generation, and the
generator latent are further optimized based on sampling criteria from the
model output to generate informative examples most likely to fail predictions.

Figure 1: Overall framework of LADA.

##  2 Method

We present our litho-aware data augmentation (LADA) framework, where we use
StyleGAN2 to generate realistic in-distribution data and formulate adversarial
active sampling as an optimization problem to harness informative samples
based on model output.

Figure 1 illustrates the pipeline architecture of LADA, where the generator
$G$ targets to generate mask images that will fail the lithography modeling
network $F$. For the best of LADA performance, we select StyleGAN-2 [7] as our
generator backbone and DOINN [22] to be the machine learning-based lithography
model. However, their limitations require litho-dedicated design to make the
whole framework feasible.

We leverage StyleGAN2 generator to synthesize realistic in-distribution data.
StyleGAN2 is a state-of-the-art generative model trained in adversarial
setting, where a novel generator architecture leads to disentangled high-level
attributes and stochastic variations in an unsupervised manner. The generated
_styles_ variables from random _latent_ code control granular style variations
of the generated image, such as the background lightning or sex of human
faces. The random _noise_ variables affect low-level details of the images,
allowing small perturbations barely noticeable to the human eye, injecting
stochastic details such as finer details background details or finer curls of
hair. This attribute is consistent with our demands on mask image generation
because the lithography process is highly sensitive and minor perturbations on
the mask image will result in significant change on the output.

Figure 2: Multi-task DOINN with loss prediction module.

As shown in Figure 2, the DOINN backbone consists of three processing paths:
global perception (GP), local perception (LP) and image reconstruction (IR).
GP and LP work together to capture high quality embedding that will be feed
into IR to generate resist images. Inspired by Fourier Optics, the GP path
leverage Fourier Neural Operator [9] to obtain low-frequency global
information feature maps, while the LP path consists of a series of
convolutional layers for high frequency local information. The obtained
bottleneck feature maps are fed to a series of deconvolution layers for image
reconstruction, similar to a U-Net [16] structure. To make the LADA framework
feasible, we leverage the loss prediction module [25] that can be integrated
into the original DOINN following GP and LP.

### Loss Prediction Module:

The loss prediction module (LPM) predict the loss value $\hat{l_{x_{U}}}$
without ground truth labels for any given unlabeled $x_{U}$ in the input
domain. It takes feature maps as the inputs that are extracted between mid-
level blocks of the target DOINN model. Each feature map is reduced to a fixed
embedding size through the LPM module, which consists sequentially of global
average pooling (GAP), FC layers, and ReLU activation layers. The embeddings
from different layers are concatenated and finally passed through another FC
layer to predict the scalar loss value.

The goal of adversarial active sampling lies in two key aspects: (1) Generate
realistic in-distribution synthetic data; and (2) Improve data efficiency and
training overhead with generating most informative data inputs. Prior work on
active learning focus on reducing the labeling cost by selecting from pool-
based settings, where our work synthesize images from a pre-trained GAN
generator in membership query synthesis fashion. The optimization problem that
describe novel and informative sample generation is as follows:

$\underset{\bm{z},\mathrm{\textit{noise}}\in
N(0,1)}{\mathrm{argmax}}C(G(\bm{z},\textit{noise};\Theta_{G}),F(\cdot;\Theta_{F,t})),$
(1)

where $\bm{z},\textit{noise}$ are the latent code and noise variables, $G$ is
the StyleGAN2 generator with pre-trained weights $\Theta_{G}$, and $F$ is the
DOINN model with weights $\Theta_{F,t}$ during some iteration $t$. $C$ is a
criteria function to evaluate how informative the synthesize image would be
towards at improving the model $F$ if the model is to be retrained with the
image added at the next iteration $t+1$.

We explore two different solutions to the relaxed original problem, where the
sampling process is conducted on separated input domains of the StyleGAN2
generator style latent code $\bm{z}$ and noise. Sampling in the style latent
code $\bm{z}$ domain is equivalent to:

$\begin{split}\underset{\bm{z}}{\mathrm{argmax}}C_{\bm{z}}(G(\bm{z};\Theta_{G}),F(\cdot;\Theta_{F,t}))\\\
+\frac{\lambda_{1}}{|\bm{z}|}\mathrm{log}(p(\bm{z};N(0,1)),\end{split}$ (2)

where the constraint that $\bm{z}\in N(0,1)$ is relaxed instead to maximize
the log-likelihood and noise is fixed to 0. Similarly, we can random sample
$\bm{z}$ from $N(0,1)$ which is kept fixed and optimize noise variable:

$\begin{split}\underset{\mathrm{\textit{noise}}}{\mathrm{argmax}}C_{\mathrm{\textit{noise}}}(G(\mathrm{\textit{noise}};\Theta_{G}),F(\cdot;\Theta_{F,t}))\\\
+\frac{\lambda_{2}}{|N|}\mathrm{log}(p(\mathrm{\textit{noise}};N(0,1)).\end{split}$
(3)

We leverage the multi-task DOINN LPM predicted loss as the sampling criteria:

$C=F_{LPM}(G(\bm{z},\mathrm{\textit{noise}})),$ (4)

which motivates synthesized data to have larger loss between DOINN model $F$
prediction and labels without having to access the lithography simulator. This
method is domain agnostic as the network learns a single loss scalar and
should generalize to synthesized image regardless of granular style changes or
small noise perturbations. The output of the generator $G$ is directly chained
as the input of the DOINN LPM model $F_{LPM}$.

We also experiment with alternative sampling criterion from methods in pool-
based active learning and adversarial attack.

### Model Uncertainty:

The Dice Loss [8] between prediction logits $\\{p_{i},q_{i}\\}$ of the DOINN
model $F$ captures the model uncertainty:

$C=L_{Dice}(P,Q)-1=-\frac{2\sum p_{i}q_{i}}{\sum p_{i}^{2}+\sum q_{i}^{2}}.$
(5)

### Model Perturbation:

Inspired by adversarial attack [19], the cross-entropy loss between model $F$
outputs samples maximal model perturbations:

$C_{\mathrm{\textit{noise}}}=L_{CE}(F(G(\bm{z},\mathrm{\textit{noise}})),\tilde{F}(G(\bm{z},0))).$
(6)

##  3 Experiments

Lithography modeling lacks sufficient open-source datasets to train deep
machine learning models. In this section, we briefly explain how we
constructed the initial training dataset, and pretrain models for StyleGAN2
and DOINN.

### Testing Dataset:

We test our models on ICCAD 2013 CAD Contest [2] which consists of 10 M1
(Metal Layer 1) designs on $32nm$ design node at 2k resolution.

### Initial Training Dataset:

We follow prior work [20] in generating a shape-based generator to synthesize
initial training dataset. The shape-based generator synthesized rectlinear
design patterns following design rules similar to test data patterns. The
generated images are optimized for Optical Proximity Correction (OPC) with
industrial level tools, such that the mask images are manufacturable. The
synthesized mask images after OPC are simulated with a lithograph simulator
[4] to obtain the resist image label.111OPC takes hours to complete for a
single design, while lithography simulation takes seconds. We construct an
initial training dataset of 2000 M1 designs for model pretraining.

### Model Pretraining

The StyleGAN2 and DOINN models are pretrained on the initial training dataset
of 2000 M1 designs. In training
StyleGAN2222https://github.com/NVlabs/stylegan2-ada-pytorch we follow the
proposed default parameters.The model is trained with downscaled image
resolution of 256 to decrease training time and generated images are bilinear
upsampled to 2k resolution.

### Evaluation Metrics

We evaluate models using the Jacard index of the foreground class (fIoU):

$fIoU=\frac{P_{1}\cap G_{1}}{P_{1}\cup G_{1}},$ (7)

where $P_{1},G_{1}$ denotes the prediction and ground truth for the foreground
class. We use fIoU since the high proportion of background pixels typically
inflates the overall mIoU score.

Table 1: Testing results for iterative adversarial active sampling. The shape use shape-based pattern generator, random randomly samples for style and noise from StyleGAN2, style_dice use dice loss for uncertainty in where we only sample for style for granular changes in the generation, noise_CE is adversarial sampling on noise domain for maximal model output perturbation, style_pred and noise_pred are adversarial sampling for the loss prediction module’s predicted loss on style and noise respectively. pretrain presents both train and test results of the pretrained model on 2k data. Other results are models finetuned on all generated 32k data after 16 iterations. Item | $fIoU\%$ | $error\%$ | $Gap\%$
---|---|---|---
pretrain (train) | 98.4583 | 1.5417 | -
pretrain (test) | 94.3589 | 5.6411 | 4.0994
shape | 96.3467 | 3.6533 | 2.1116
random | 97.1370 | 2.8630 | 1.3213
style_dice | 97.3223 | 2.6777 | 1.1360
noise_CE | 97.3222 | 2.6778 | 1.1361
style_pred | 98.2216 | 1.7784 | 0.2367
noise_pred | 98.1474 | 1.8526 | 0.3109

We conduct experimental studies to evaluate the proposed active learning
approach on improving lithograph modeling. We extensively compare against
different adversarial sampling criterion functions inspired from prior work on
pool-based active learning.

We follow the proposed method in where we set total iteration to $T=16$ and
the labeling budget for each iteration $B=2k$. Table 1 compares the testing
results for iterative adversarial active learning. Gap measures the
generalization gap between model performance with train results on pretrain.
The style_pred performs the best, with a $fIoU$ testing error of $1.78\%$,
reducing the generalization error by $37.9\%$ compared with random of
$2.86\%$, and narrows the generalization gap to less than $0.24\%$.

Table 2: Mask legalization makes standard adversarial attack meaningless. Legalization restores noised injected mask image to the original image, yielding meaningless adversarial example generation. Adv masks are illegal and do not have PhysicalSim results. Design |  Mask Image |  DOINN |  PhysicalSim
---|---|---|---
Original |  |  |
Adv |  |  |  N/A
Adv-Legal |  |  |
Figure 3: Visualization and comparison for test performance. The $fIoU$ are
listed at the bottom of each predictive output.

##  4 Conclusion and Future Work

In this work, we present LADA, a litho-aware data augmentation framework based
on adversarial active sampling techniques, which targets at reducing the gap
between the empirical and the generalization error of the machine learning-
based lithography simulator. We aim at resolving both the data limitation in
data synthesis while increasing the data efficiency and reducing training
overhead through adversarial active sampling. Our work in resolving data
limitation and improving data quality in a query-based activation learning
setting, could serve to further boost neural network generalization for
learning physical-based systems. Future work include further increasing the
DOINN backbone model capacity and improving its generalization bound.

## References

  * Antoniou et al. [2017] A. Antoniou, A. Storkey, and H. Edwards.  Data augmentation generative adversarial networks.  _arXiv preprint arXiv:1711.04340_ , 2017.
  * Banerjee et al. [2013] S. Banerjee, Z. Li, and S. R. Nassif.  ICCAD-2013 CAD contest in mask optimization and benchmark suite.  In _IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_ , pages 271–274, 2013.
  * Chen et al. [2020] G. Chen, W. Chen, Y. Ma, H. Yang, and B. Yu.  DAMO: Deep agile mask optimization for full chip scale.  In _IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_ , 2020.
  * Chen et al. [2021] G. Chen, Z. Yu, H. Liu, Y. Ma, and B. Yu.  Develset: Deep neural level set for instant mask optimization.  In _IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_ , pages 1–9. IEEE, 2021.
  * Cheng and Yan [2021] R. Cheng and J. Yan.  On joint learning for solving placement and routing in chip design.  _Conference on Neural Information Processing Systems (NIPS)_ , 34:16508–16519, 2021.
  * Ding et al. [2011] D. Ding, A. J. Torres, F. G. Pikus, and D. Z. Pan.  High performance lithographic hotspot detection using hierarchically refined machine learning.  In _IEEE/ACM Asia and South Pacific Design Automation Conference (ASPDAC)_ , pages 775–780, 2011.
  * Karras et al. [2020] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8110–8119, 2020.
  * Li et al. [2019] X. Li, X. Sun, Y. Meng, J. Liang, F. Wu, and J. Li.  Dice loss for data-imbalanced nlp tasks.  _arXiv preprint arXiv:1911.02855_ , 2019.
  * Li et al. [2021] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar.  Fourier neural operator for parametric partial differential equations.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Lin et al. [2019] Y. Lin, S. Dhar, W. Li, H. Ren, B. Khailany, and D. Z. Pan.  DREAMPlace: Deep learning toolkit-enabled GPU acceleration for modern VLSI placement.  In _ACM/IEEE Design Automation Conference (DAC)_ , 2019.
  * Ma and Arce [2011] X. Ma and G. R. Arce.  _Computational lithography_ , volume 77.  John Wiley & Sons, 2011.
  * Matsunawa et al. [2012] T. Matsunawa, S. Maeda, H. Ichikawa, S. Nojima, S. Tanaka, S. Mimotogi, H. Nosato, H. Sakanashi, M. Murakawa, and E. Takahashi.  Generator of predictive verification pattern using vision system based on higher-order local autocorrelation.  In _Proceedings of SPIE_ , volume 8326, 2012.
  * Mirhoseini et al. [2021] A. Mirhoseini, A. Goldie, M. Yazgan, J. W. Jiang, E. Songhori, S. Wang, Y.-J. Lee, E. Johnson, O. Pathak, A. Nazi, et al.  A graph placement methodology for fast chip design.  _Nature_ , 594(7862):207–212, 2021.
  * Ren and Fojtik [2021] H. Ren and M. Fojtik.  Nvcell: Standard cell layout in advanced technology nodes with reinforcement learning.  In _ACM/IEEE Design Automation Conference (DAC)_ , pages 1291–1294. IEEE, 2021.
  * Ren et al. [2021] P. Ren, Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang.  A survey of deep active learning.  _ACM computing surveys (CSUR)_ , 54(9):1–40, 2021\.
  * Ronneberger et al. [2015] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)_ , pages 234–241, 2015.
  * Shim et al. [2017] S. Shim, S. Choi, and Y. Shin.  Machine learning-based 3d resist model.  In _Proceedings of SPIE_ , volume 10147, 2017.
  * Tanaka and Aranha [2019] F. H. K. d. S. Tanaka and C. Aranha.  Data augmentation using gans.  _arXiv preprint arXiv:1904.09135_ , 2019.
  * Xie et al. [2017] C. Xie, J. Wang, Z. Zhang, Y. Zhou, L. Xie, and A. Yuille.  Adversarial examples for semantic segmentation and object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1369–1378, 2017.
  * Yang et al. [2018] H. Yang, S. Li, Y. Ma, B. Yu, and E. F. Young.  GAN-OPC: Mask optimization with lithography-guided generative adversarial nets.  In _ACM/IEEE Design Automation Conference (DAC)_ , pages 131:1–131:6, 2018.
  * Yang et al. [2020] H. Yang, S. Li, Z. Deng, Y. Ma, B. Yu, and E. F. Y. Young.  GAN-OPC: Mask optimization with lithography-guided generative adversarial nets.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD)_ , 2020.
  * Yang et al. [2022] H. Yang, Z. Li, K. Sastry, S. Mukhopadhyay, M. Kilgard, A. Anandkumar, B. Khailany, V. Singh, and H. Ren.  Generic lithography modeling with dual-band optics-inspired neural networks.  In _ACM/IEEE Design Automation Conference (DAC)_ , 2022.
  * Ye et al. [2019] W. Ye, M. B. Alawieh, Y. Lin, and D. Z. Pan.  LithoGAN: End-to-end lithography modeling with generative adversarial networks.  In _ACM/IEEE Design Automation Conference (DAC)_ , pages 107:1–107:6, 2019.
  * Ye et al. [2020] W. Ye, M. B. Alawieh, Y. Watanabe, S. Nojima, Y. Lin, and D. Z. Pan.  Tempo: Fast mask topography effect modeling with deep learning.  In _ACM International Symposium on Physical Design (ISPD)_ , pages 127–134, 2020.
  * Yoo and Kweon [2019] D. Yoo and I. S. Kweon.  Learning loss for active learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 93–102, 2019.
  * Zhu and Bento [2017] J.-J. Zhu and J. Bento.  Generative adversarial active learning.  _arXiv preprint arXiv:1702.07956_ , 2017.