[table]capposition=top

# Spatial-Interference Aware Cooperative Resource Allocation for 5G NR
Sidelink Communications

E-mails: {silvia.mura, francesco.linsalata<EMAIL_ADDRESS>Silvia
Mura1, Francesco Linsalata1, Marouan Mizmizi1, Maurizio Magarini1,
Majid Nasiri Khormuji2, Peng Wang2, Alberto Perotti2, Umberto Spagnolini13
1Dipartimento di Elettronica, Informazione e Bioingegneria,
Politecnico di Milano, Via Ponzio 34/5, 20133, Milano Italy  2Huawei
Technologies Sweden AB, Skalholtsgatan 9-11, SE-164 94 Kista, Stockholm,
Sweden  3Huawei Industry Chair

###### Abstract

Distributed resource allocation (RA) schemes have been introduced in cellular
vehicle-to-everything (C-V2X) standard for vehicle-to-vehicle (V2V) sidelink
(SL) communications to share the limited spectrum (sub-6GHz) efficiently.
However, the recent progress in connected and automated vehicles and mobility
services requires a huge amount of available spectrum resources. Therefore,
millimeter-wave and sub-THz frequencies are being considered as they offer a
large free bandwidth. However, they require beamforming techniques to
compensate for the higher path loss attenuation. The current fifth-generation
(5G) RA standard for SL communication is inherited from the previous C-V2X
standard, which is not suited for beam-based communication since it does not
explore the spatial dimension. In this context, we propose a novel RA scheme
that addresses the directional component by adding this third spatial
dimension to the bandwidth part structure and promotes cooperation between
vehicles in resource selection, namely cooperative three-dimensional RA.
Numerical results show an average of 10% improvement in packet delivery ratio,
an average 50% decrease in collision probability, and 30% better channel busy
ratio compared to the current standard, thus, confirming the validity of the
proposed method.

###### Index Terms:

5G NR, millimeter waves, resource allocation, sidelink

##  I Introduction

In recent years, the advances in vehicle-to-everything (V2X) communications
have boosted the evolution of intelligent transportation systems (ITS) toward
a safer and more efficient road network. This has also led to advanced
services with increasingly stringent communication requirements. Currently
deployed V2X technologies, such as dedicated short-range communication (DSRC)
[1] and long-term evolution (LTE) cellular-V2X (C-V2X) [2], operate at
sub-6GHz frequencies. Compared to DSRC, C-V2X exhibits better performances
thanks to resource allocation (RA) schemes that allow for efficient sharing of
the available bandwidth while reducing mutual interference [3].

In the context of LTE C-V2X, there are two classes of RA schemes: (i)
centralized RA, the base station allocates the resources for all the vehicles
in the network, e.g., mode 3, and (ii) distributed RA, the vehicles
autonomously sense and select the available resources, i.e., mode 4 [4]. The
authors in [5] propose a short-term sensing window for periodic and aperiodic
traffic for C-V2X, where sidelink (SL) beam-based communications are not
considered. The work in [6] proposes a resource selection based on the
bandwidth partitioning according to the CAVs’ positions. However, this
approach can limit the performances by encouraging collisions in frequency
resources selection. The same drawback can affect the RA solution proposed in
[7], where frequency resources are allocated based on vehicles’ speed,
density, heading, and positions. Thus, the static partitioning of the
bandwidth according to the scenario topology may affect the candidate sub-
channels selection. Cluster-based solutions are used to deal with semi-
distributed RA. These involve the election of a cluster head that manages
resource sharing and power allocation. In [8] the bandwidth is partitioned
dynamically according to the cluster definition, but the presence of an
elected vehicle may cause undesired latency.

The literature mainly focuses on sub-6GHz portion of the spectrum with a
limited bandwidth that barely meets the requirements of basic V2X services. To
accommodate the needs of enhanced V2X (e-V2X) services, the third generation
partnership project (3GPP) has proposed the use of millimeter-wave (mmW) bands
for fifth-generation (5G) NR-based V2X systems [9], and sub-THz frequencies
are being investigated for the upcoming sixth-generation (6G) V2X
communication. Wireless propagation at such very high frequencies is subject
to an order of magnitude higher path loss attenuation and needs beamforming
techniques to ensure the required coverage [10]. The centralized mode 1 and
distributed mode 2 schemes for RA in 5G NR are inherited from the C-V2X mode 3
and mode 4, respectively. However, these fail to properly include the spatial
dimension introduced by the beam-type communication. Therefore, the resulting
interference, in particular for vehicle-to-vehicle (V2V) SL communication, can
compromise the 5G NR RA effectiveness and it results in poor reliability [11].

Figure 1: Bandwidth part (BWP) for sidelink RA representing the sensing and
selection window and sub-channel structure.

As main result of this paper, we show that the standard RA procedure is not
properly suited to the new 5G physical layer (PHY) paradigm. Therefore, we
propose a novel three-dimensional (3D) cooperative RA scheme that accounts for
the directional component and, when compared with the current standard RA, it
improves the packet delivery ratio (PDR) by 10% on average, the channel busy
ratio (CBR) of the bandwidth of 30%, and halves the average collision
probability.

The remainder of this paper is organized as follows. The PHY design of SL 5G
NR is discussed in Sec. II. Section III presents the semi-persistent RA and
the cooperative 3D RA schemes. Simulation setup and numerical results are
discussed in Sec. IV. Finally, Sec. V draws the conclusions.

##  II Sidelink 5G NR PHY

This section highlights the main features of the current 5G NR PHY structure
of SL RA, such as the bandwidth part (BWP) structure (Sec. II-A), the logical
channels (Sec. II-B), and the RA modes (Sec. II-C).

###  II-A Bandwidth Part

The BWP, depicted in Fig.1, defines the 2D resource grid in the time and
frequency domains. SL transmissions and receptions are contained within the SL
BWP. The minimum BWP element, called resource block (RB), spans one-time slot,
composed of 14 OFDM symbols, and 12 consecutive sub-carriers with the same
sub-carrier spacing (SCS). The BWP numerology $\mu$ is related the slot
duration as $2^{-\mu}$ ms and to the SCS, evaluated as $2^{\mu}\times$ 15 kHz
[12]. The sub-channel represents the smallest unit for a SL data transmission
or reception and it is composed of a pre-configured number of consecutive RBs.
The ensemble of sub-channels dedicated to SL data transmission defines the
resource pool (RP) and it does not include the resources allocated for the
initial access and synchronization [13, 14].

###  II-B Logical Channels

The 5G NR SL provides the following logical channels [15]

  * •

Physical Sidelink Control Channel (PSCCH): carries the 1st-stage sidelink
control information (SCI), such as the information regarding the PSCCH sub-
channels resources, the information for decoding the 2nd-stage SCI, and the
priority of the transport block (TB) transmission. When a transmitting vehicle
(TxV) sends the TB to a receiving one (RxV), it first sends the PSCCH that is
decoded for the channel sensing purpose.

  * •

Physical Sidelink Shared Channel (PSSCH): carries the 2nd-stage SCI and the TB
data payload. The 2nd-stage SCI contains the remaining information to decode
the PSSCH and the TxV/RxV identifiers (IDs).

  * •

Physical Sidelink Feedback Channel (PSFCH): carries the hybrid automatic
repeat request (HARQ) feedback concerning the TB reception.

###  II-C RA Modes

SL RA in the current standard defines two RA modes.

  * •

Centralized Mode 1: the SL resources are allocated by the base station when
vehicles are in its coverage area. It can use either (i) dynamic grant
scheduling, i.e., for each TB, the base station allocates the dedicated
resources, or (ii) configured grant scheduling, which enables the reservation
of sub-channels for transmitting multiple TBs [16].

  * •

Distributed Mode 2: it is used when vehicles are out of coverage and they
autonomously select SL resources from the RP. It can operate either in the
dynamic scheduling, where new resources are selected for the transmission of
each TB or in the semi-persistent scheduling, where resources are reserved for
the transmission of multiple TBs [17].

The semi-persistent scheduling is more suited to latency-critical applications
than the dynamic one since the resources are allocated for multiple TBs [18].
Due to its generality, in the next section, we present the 5G NR standard
semi-persistent scheme and we discuss its limitation in the mmW V2V
communications.

##  III Sidelink Semi-Persistent Scheme

The semi-persistent RA scheme is pre-configured in the RP, and each vehicle,
after the initial access and synchronization, can select the resources for the
transmission of several TBs. The maximum number of consecutive TBs that can be
allocated is defined by the re-selection counter (RC), and it depends on the
resource reservation interval(RRI), which defines the periodicity for the
allocated TBs. These semi-persistent parameters are mutually related: if $RRI$
$\geq$ $100\,$ms RC is randomly selected in $[5,15]$ , otherwise the counter
is randomly set within the interval $[5C,15C]$ with $C$ = 100
/$\text{max}(20,RRI)$ and it is decremented by 1 after each transmission [19].
When the RC value is equal to zero, new resources are selected with
probability $1-P$, with P $\in[0,0.8]$ [20]. In the following, the standard SL
RA scheme is detailed.

###  III-A Standard-compliant RA

The standard SL RA scheme consists of two phases [21], the sensing of the
available resources, followed by the selection of the transmission time-
frequency resources.

####  III-A1 Sensing

each vehicle, when not transmitting, senses for a time interval defined by the
sensing window for the available resources in the RP by measuring the received
signal reference power (RSRP) for each sub-channel. If RSRP $\leq\gamma_{th}$,
the sub-channel is available, otherwise busy. The threshold $\gamma_{th}$
depends on the priority of the TBs to be transmitted and the percentage of
occupied RBs in the sensing window. Therefore, the $i$th vehicle stores a
bitmap matrix $\mathbf{B}_{i}$ defined as

$\displaystyle\left[\mathbf{B}_{i}\right]_{(t,f)}=\begin{cases}1,&\text{if
}\text{RSRP}\leq\gamma_{th},\\\ 0,&\text{otherwise},\end{cases}$ (1)

with $f$ and $t$ referring to the sub-channel and time-slot indexes,
respectively. The ratio between the number of candidate resources and the
total number of resources in the sensing window is calculated by each vehicle
as

$\displaystyle p=\frac{|\mathbf{B}_{i}|_{0}\cdot 100}{|\mathbf{B}_{i}|},$ (2)

where $|.|_{0}$ refers to the $l_{0}$-norm and $|.|$ to the matrix
cardinality. In highly congested networks, $p$ is low, which implies that
limited resources are available. Thus, when $p\leq\Gamma_{th}$, the threshold
$\gamma_{th}$ is decreased by $3\,$dB, and the above procedure is repeated.
When RSRP $>\gamma_{th}$, the TxV attempts to decode the PSCCH and the related
1st-stage SCI, which contains the RRI information. The RRI is exploited to
estimate the RC, and by combining RRI and RC, the TxV predicts the resources
that have been allocated and excludes them from the available resources.
Figure 1 depicts the SL-BWP perceived by a vehicle; in the sensing window each
sub-channel is categorized as busy or available, according to measured RSRP.

####  III-A2 Selection

when a TB is ready for transmission, a selection trigger is activated. Among
the available resources, the vehicle chooses $M$ candidates for TB
transmission and $M-1$ candidates for TB re-transmission or HARQs [22], with
$1\leq M\leq 32$. Then, the vehicle randomly selects the resources among the
candidate ones. Re-selection and re-assessment are introduced by 3GPP to
manage traffic priority and TB latency requirements. However, these are still
under discussion for further enhancements of the congestion control in Rel. 17
and they are not included in this work.

The pseudocode of the standard semi-persistent RA is reported in Alg. 1.

Algorithm 1 Standard-complaint RA

1:Sensing

2: RSRP measurement for each sub-channel $f$

3: Update bitmap $\mathbf{B}_{i}$ according to (1)

4: Evaluate $p$ as in (2)

5: if $p>\Gamma_{th}$

6: RRI of the sub-channels (PSCCH) decoding

7: Estimate RC

8: $\left[\mathbf{B}_{i}\right]_{(t_{RRI},f)}$ = $0$

9: with $t_{RRI}$ $\,=\,$ $\\{t,t+RRI,...,t+(RC-1)\,RRI\\}$

10: else

11: $\gamma_{th}$ = $\gamma_{th}-3$ [dB]

12: Repeat from step 1

13: end

14:Selection

15:TxV randomly selects sub-channels and RRI based on $\mathbf{B}_{i}$

Figure 2: Simulated Interference power $P_{i}$ versus $\Delta_{\theta}$.

###  III-B Cooperative 3D RA

Figure 3: 3D resource pool for sidelink RA with beam pointing directions.
Algorithm 2 Cooperative 3D RA

1:Sensing

2: for $\theta\in\boldsymbol{\Theta}$

3: RSRP measurement for each sub-channel $f$

4: Update bitmap $\mathbf{B}_{i,\theta}$ according to (1)

5: Evaluate $p_{\theta}$ as in (2)

6: if $p_{\theta}>\Gamma_{th}$

7: RRI of the sub-channels (PSCCH) decoding

8: Estimate RC

9: $\left[\mathbf{B}_{i,\theta}\right]_{(t_{RRI},f)}$ = $0$

10: with $t_{RRI}$ = $\\{t,t+RRI,...,t+(RC-1)RRI\\}$

11: else

12: $\gamma_{th}$ = $\gamma_{th}-3$ [dB]

13: Repeat from step 1

14: end

15: end

16:Selection

17:TxV sends RA request to RxV

18:RxV randomly selects the sub-channels and RRI based on
$\mathbf{B}_{i,\bar{\theta}}$

19:RxV feedbacks the RA configuration to TxV

The standard procedure for RA suffers from several drawbacks as it is not
suited for beam-based mmW communications. The main issues are related to the
sensing procedure, in which the spatial interference is not addressed. This
can be observed in Fig. 2, where the interference power is evaluated varying
the angle of the interfering vehicle. This result shows that the interference,
which may affect the perception of available sub-channel resources, is mainly
related to the beam pointing directions. Moreover, in the standard RA, the TxV
allocates resources based on its sensing, and it may not sense the
interference affecting the RxV.

We propose a RA scheme that explores the spatial dimension given by the beams,
and the cooperation among vehicles, i.e., cooperative 3D RA as shown in Fig.
3. Compared to the standard RA, the proposed approach introduces two main
novelties: (i) during the selection phase, the cooperative 3D RA employs a
bitmap $\mathbf{B}_{i,\theta}$ for each beam $\theta$ defined in the
beamforming codebook $\Theta$ = $[\theta_{1},\dots,\theta_{L}]$, where $L$ is
the codebook depth and, (ii) cooperation between TxV and RxV in the candidate
resource selection. In the selection phase, the TxV sends an RA request to the
RxV, which replies with the set of resources available according to its
$\mathbf{B}_{i,\bar{\theta}}$ bitmap, where $\bar{\theta}$ refers to its
receiving spatial direction. The request for RA and the response from the RxV
are signaling messages sent over a set of resources selected using the
standard RA selection procedure. The cooperative 3D RA procedure is shown in
Alg. 2.

##  IV Simulation Results

TABLE I: PHY design parameters Parameter | Value
---|---
Numerology ($\mu$) | $3$
Sub-Carrier Spacing (SCS) |  $120$ kHz
Frequency Range | FR2
Cyclic Prefix | Normal
OFDM symbols/Slot | $14$
Slots/Subframe | $8$
Slot duration |  $0.125$ ms
Maximum carrier bandwidth |  $400$ MHz
Sub-channels/Bandwidth part | $10$
Transmitted Power |  $23$ dBm
Noise Power |  $-68$ dBm
Carrier Frequency |  $30$ GHz
Antenna Height (wrt rooftop) |  $0.1$ m
Antenna Elements | $4\times 4$
TB Size |  $1$ slot
PSCCH/PSSCH Modulation | QPSK
TABLE II: Semi-persistent scheduling parameters Parameter | Value
---|---
Resource Reservation Interval (RRI) |  $2$ ms
Re-selection Counter (RC) | 4
Sub-channels/Resource pool | $10$
Resource Blocks PSCCH/Sub-channel | $10$
Resource Blocks PSSCH/Sub-channel | $12$
PSCCH symbols/Slot | $2$
DMRS symbols/Slot | $3$
Sensing Window |  $796$ slots
Selection Window (W) |  $20,40$ slots
RSRP threshold ($\gamma_{th}$) |  $-60$ dB
Resource occupancy threshold ($\Gamma_{th}$) | $20\%$
Blind retransmissions | $3$
Maximum number of CAVs for selection ($N_{max}$) | $10,20$
CBR interval |  $50$ slots
Figure 4: Simulated urban scenario in Milan, Italy lat =
$[45.46848,\,45.47222]$ and lon = $[9.21239,\,9.21606]$.

Exhaustive numerical simulations were carried out to evaluate the performance
of the cooperative 3D RA approach compared with the standard-compliant scheme,
considered here as benchmark. The implemented system model is fully standard-
compliant. The latest 3GPP recommendations for the setting of PHY are used
[21]. The main PHY parameters are detailed in Tab. I, while Tab. II reports
those for the 5G NR standard Mode 2 RA. The communication channel is half-
duplex with a bidirectional alternating transmission. Hence, when a vehicle
senses the communication channel cannot transmit and vice versa. Moreover,
vehicles employ an array of planar antennas with $4\times 4$ configuration,
mounted on the rooftop of CAVs as in [23]. The communication between each pair
of TxV and RxV is based on the legacy orthogonal frequency division multiple
access [15], and the received power in an arbitrary sub-channel is computed as

$\displaystyle\text{RSRP}=P_{t}-PL+2G_{b}\,\,\text{[dBm]}\,,$ (3)

where $P_{t}$ is the transmitted power, PL is the path loss, and $G_{b}$ is
the beamforming gain [24]. The RA algorithms detailed in Algs. 1 and 2, are
applied by vehicles after the initial access and synchronization phase.

###  IV-A Simulation Methodology

The numerical simulations examine an urban area of Milan, Italy, represented
in Fig. 4. The road network scenario, buildings, and trees are obtained from
OpenStreeMap [25] while the mobility is simulated through Simulator of Urban
Mobility (SUMO) [26]. The propagation channels of the SL links are generated
based on the latest 3GPP guidelines in [23] using a geometric-based efficient
propagation model for V2V (GEMV2) [27]. This uses the urban topology and the
vehicles’ mobility to compute the large-scale channel components with a
deterministic approach and the small-scale ones with a geometry-based
stochastic approach.

###  IV-B Performance Metrics

Figure 5: PDR versus average interference power for the two RA schemes. Figure
6: ECDF of the CBR of the two RA approaches. Figure 7: ECDF of the collision
probability of the two RA approaches.

The performances of the RA schemes are analyzed in terms of PDR, CBR, and
collision probability. The PDR is defined as

$\displaystyle PDR=\frac{\sum_{m=1}^{N_{tb}}I(i,m)}{N_{tb}},$ (4)

where $N_{tb}$ is the number of transmitted TBs, and

$\displaystyle I(i,m)=\begin{cases}1,&\text{if $i$th RxV receives the $m$th
TB,}\\\ 0,&\text{otherwise}.\end{cases}$ (5)

The CBR is a metric introduced by 3GPP to highlight the RP occupancy and it is
defined as the ratio of busy sub-channels within a predefined interval [17].
Moreover, in order to further investigate the performances of the RA schemes,
the two methods are compared in terms of collision probability, defined as the
probability that at least two vehicles select the same sub-channels. In the
case of cooperative 3D scheme, this means that two different RxVs assign the
same RA configurations to their corresponding TxVs. Assuming $N$ vehicles that
simultaneously perform the selection procedure, the $i$th vehicle has a set of
available sub-channels, which contains the non null elements of
$\mathbf{B}_{i}$ in (1) for the standard procedure and of
$\mathbf{B}_{i,\theta}$ for the cooperative 3D scheme. As the perception of an
individual vehicle may differ due to the dynamic urban scenario, these sets
can be distinct.

The collision probability is defined as

$\displaystyle P=\frac{\sum_{m=1}^{N_{sub}}S(t,f)}{N_{sub}},$ (6)

where $N_{sub}$ is the number of selected sub-channels, and

$\displaystyle S(t,f)=\begin{cases}1,&\text{if $\geq 2$ CAVs select the
$(t,f)$th sub-channel,}\\\ 0,&\text{otherwise}.\end{cases}$ (7)

###  IV-C Results Discussion

Figure 5 depicts the PDR versus the average interference power experienced by
the V2V system, obtained for a selection window of $W$ = $40$ slots and by
limiting the maximum number of vehicles that can simultaneously perform the
selection to $N_{max}$ = $\\{10,20\\}$. The cooperative 3D RA shows a robust
response to interference compared to the standard RA scheme, achieving on
average 10% higher PDR, being able to discriminate the occupation of resources
also along the spatial dimension spanned by the beams.

The CBR of the two RA approaches are reported in Fig. 6 in the case of a
selection window $W$ = $\\{20,40\\}$ slots with $N_{max}$ = $10$. In both
scenarios, the proposed method outperforms the standard one by guaranteeing a
CBR 30% lower (obtained by intercepting the ECDF curve at 0.5).

Figure 7 shows that the cooperative 3D RA halves the average collision
probability in (6) in the case of different $W$ and $N_{max}$ = $10$.

The cooperative 3D RA enhances the perception of the available resources by
adding the third directional dimension in the BWP. This counteracts the
spatial interference and improves the average PDR for the V2V system.
Moreover, the CBR and collision probability results suggest that the
cooperative 3D approach limits the bandwidth occupancy, although it needs a
two-way communication and consequently uses more sub-channels.

##  V Conclusion

In mmW V2V communications, interference is a critical issue that needs to be
addressed to achieve the stringent requirements of the upcoming V2X advanced
services. One of the methods to reduce interference is through resource
allocation schemes. In this paper, we demonstrate that the standard-complaint
resource allocation procedure is not efficient in mitigating interference, as
it is originally designed for the previous release, i.e., C-V2X, where beam-
based communications are not contemplated. Motivated by this, we present a
resource allocation strategy that exploits the spatial dimension spanned by
the beamforming and cooperation among vehicles to mitigate V2V interference.
Extensive 3GPP-compliant numerical simulations show that the proposed approach
improves the packet delivery ratio versus interference power by 10% on average
compared to the current standard method. Moreover, both channel busy ratio and
collision probability are evaluated for the two methods. The performances
highlight that the cooperative 3D resource allocation determines an improved
perception of the available resources.

## Acknowledgment

This research was carried out in the framework of the Huawei-Politecnico di
Milano Joint Research Lab activities. The Authors want to acknowledge the
Huawei Milan Research Centre.

## References

  * [1] K. Abboud, H. A. Omar, and W. Zhuang, “Interworking of dsrc and cellular network technologies for v2x communications: A survey,” _IEEE Transactions on Vehicular Technology_ , vol. 65, no. 12, pp. 9457–9470, 2016.
  * [2] R. Molina-Masegosa, J. Gozalvez, and M. Sepulcre, “Configuration of the c-v2x mode 4 sidelink pc5 interface for vehicular communication,” in _2018 14th International Conference on Mobile Ad-Hoc and Sensor Networks (MSN)_ , 2018, pp. 43–48.
  * [3] M. Gonzalez-Martín, M. Sepulcre, R. Molina-Masegosa, and J. Gozalvez, “Analytical models of the performance of c-v2x mode 4 vehicular communications,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 2, pp. 1155–1166, 2019.
  * [4] F. Tang, Y. Zhou, and N. Kato, “Deep reinforcement learning for dynamic uplink/downlink resource allocation in high mobility 5g hetnet,” _IEEE Journal on Selected Areas in Communications_ , vol. 38, no. 12, pp. 2773–2782, 2020.
  * [5] X. He, J. Lv, J. Zhao, X. Hou, and T. Luo, “Design and analysis of a short-term sensing-based resource selection scheme for c-v2x networks,” _IEEE Internet of Things Journal_ , vol. 7, no. 11, pp. 11 209–11 222, 2020\.
  * [6] R. Molina-Masegosa, M. Sepulcre, and J. Gozalvez, “Geo-based scheduling for c-v2x networks,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 9, pp. 8397–8407, 2019.
  * [7] J. Kim, J. Lee, S. Moon, and I. Hwang, “A position-based resource allocation scheme for v2v communication,” _Wireless Personal Communications_ , vol. 98, no. 1, pp. 1569–1586, 2018.
  * [8] J. Zhao, X. He, H. Wang, X. Zheng, J. Lv, T. Luo, and X. Hou, “Cluster-based resource selection scheme for 5g v2x,” in _2019 IEEE 89th Vehicular Technology Conference (VTC2019-Spring)_. IEEE, 2019, pp. 1–5.
  * [9] K. Sakaguchi, R. Fukatsu, T. Yu, E. Fukuda, K. Mahler, R. Heath, T. Fujii, K. Takahashi, A. Khoryaev, S. Nagata _et al._ , “Towards mmwave v2x in 5g and beyond to support automated driving,” _IEICE Transactions on Communications_ , 2020.
  * [10] T. S. Rappaport, S. Sun, R. Mayzus, H. Zhao, Y. Azar, K. Wang, G. N. Wong, J. K. Schulz, M. Samimi, and F. Gutierrez, “Millimeter Wave Mobile Communications for 5G Cellular: It Will Work!” _IEEE Access_ , vol. 1, pp. 335–349, May 2013.
  * [11] V. Petrov, J. Kokkoniemi, D. Moltchanov, J. Lehtomäki, M. Juntti, and Y. Koucheryavy, “The impact of interference from the side lanes on mmwave/thz band v2v communication systems with directional antennas,” _IEEE Transactions on Vehicular Technology_ , vol. 67, no. 6, pp. 5028–5041, 2018.
  * [12] 3GPP TS 38.214 V16.3.0, “3rd Generation Partnership Project; Technical Specification Group Radio Access Network; NR; Physical layer procedures for data (Release 16),” Sep. 2020.
  * [13] S. Lien, D. Deng, C. Lin, H. Tsai, T. Chen, C. Guo, and S. Cheng, “3gpp nr sidelink transmissions toward 5g v2x,” _IEEE Access_ , vol. 8, pp. 35 368–35 382, 2020.
  * [14] F. Morandi, F. Linsalata, M. Brambilla, M. Mizmizi, M. Magarini, and U. Spagnolini, “A probabilistic codebook technique for fast initial access in 6G Vehicle-to-Vehicle communications,” in _IEEE Int. Conf. Commun. Workshops_ , 2021.
  * [15] 3GPP 38.211 V16.0.0, “3rd generation partnership project; technical specification nr; physical channels and modulation (release 16),” Mar. 2020.
  * [16] LG Electronics, “Summary of ran1 agreements/working assumptions in wi 5g v2x with nr sidelink,” 3GPP TSG RAN WG1 Meeting 99, Tech. Rep. R1-1913601, Nov. 2019, Reno, USA.
  * [17] M. H. C. Garcia, A. Molina-Galan, M. Boban, J. Gozalvez, B. Coll-Perales, T. Şahin, and A. Kousaridas, “A tutorial on 5g nr v2x communications,” _IEEE Communications Surveys Tutorials_ , vol. 23, no. 3, pp. 1972–2026, 2021\.
  * [18] M. H. C. Garcia, A. Molina-Galan, M. Boban, J. Gozalvez, B. Coll-Perales, T. Şahin, and A. Kousaridas, “A tutorial on 5g nr v2x communications,” _arXiv preprint arXiv:2102.04538_ , 2021.
  * [19] N. Bonjorn, F. Foukalas, F. Cañellas, and P. Pop, “Cooperative resource allocation and scheduling for 5g ev2x services,” _IEEE Access_ , vol. 7, pp. 58 212–58 220, 2019.
  * [20] A. Bazzi, G. Cecchini, A. Zanella, and B. M. Masini, “Study of the impact of phy and mac parameters in 3gpp c-v2v mode 4,” _IEEE Access_ , vol. 6, pp. 71 685–71 698, 2018.
  * [21] 3GPP TR 37.985, “Technical specification group radio access network; overall description of radio access network (ran) aspects for vehicle-to-everything (v2x) based on lte and nr,” 2020.
  * [22] 3GPP TS 38.213 V16.3.0, “3rd Generation Partnership Project; Technical Specification Group Radio Access Network; NR; Physical layer procedures for control (Release 16),” Sep. 2020.
  * [23] 3GPP TR 37.885 v15.3.0, “Study on evaluation methodology of new Vehicle-to-Everything (V2X) use cases for LTE and NR (Release 15),” Jun. 2019\.
  * [24] C. Perfecto, J. Del Ser, and M. Bennis, “Millimeter-Wave V2V Communications: Distributed Association and Beam Alignment,” _IEEE J. Sel. Areas Commun._ , vol. 35, no. 9, pp. 2148–2162, Jun. 2017.
  * [25] OpenStreetMap contributors, “Planet dump retrieved from https://planet.osm.org ,” https://www.openstreetmap.org, 2017.
  * [26] P. A. Lopez et al., “Microscopic Traffic Simulation using SUMO,” in _IEEE Int. Conf. Intell. Transp. Syst._ IEEE, Dec. 2018. [Online]. Available: https://elib.dlr.de/124092/
  * [27] M. Boban, J. Barros, and O. K. Tonguz, “Geometry-Based Vehicle-to-Vehicle Channel Modeling for Large-Scale Simulation,” _IEEE Transaction Vehicular Technology_ , vol. 63, no. 9, pp. 4146–4164, Apr. 2014.


# Towards the Human Global Context:
Does the Vision-Language Model Really Judge Like a Human Being?

Sangmyeong Woh$\dagger$, Jaemin Lee$\dagger$, Ho joong Kim  Jinsuk Lee*
Testworks Inc., Seoul, Republic of Korea
{wohsang, jaemin, hjk1221<EMAIL_ADDRESS>

###### Abstract

As computer vision and NLP make progress, Vision-Language (VL) is becoming an
important area of research. Despite the importance, evaluation metrics of the
research domain is still at a preliminary stage of development. In this paper,
we propose a quantitative metric "Equivariance Score" and evaluation dataset
"Human Puzzle" to assess whether a VL model is understanding an image like a
human. We observed that the VL model does not interpret the overall context of
an input image but instead shows biases toward a specific object or shape that
forms the local context. We aim to quantitatively measure a model’s
performance in understanding context. To verify the current existing VL
model’s capability, we sliced the original input image into pieces and
randomly placed them, distorting the global context of the image. Our paper
discusses each VL model’s level of interpretation on global context and
addresses how the structural characteristics influenced the results.

$\dagger$ $\dagger$ footnotetext: These authors contributed equally.* *
footnotetext: Corresponding author.

##  1 Introduction

Figure 1: Captioning result of each model in “Human Puzzle” dataset: Can you
recognize which model is better at handling global context? In general, OSCAR+
performs better in most VL tasks, but this does not guarantee that it will
excel in treating context information as well.

Vision-Language task interprets the context of the given image and attempts to
represent it in a human language. Image captioning Wang et al. (2022); Zeng et
al. (2021); Wang et al. (2021); Pan et al. (2020) for example, analyzes
context (relations between objects, situational background, etc.) to construct
a sentence that adequately explains the given image. In a similar domain, VQA
Antol et al. (2015) uses multi-modal to generate adequate answers to the text
(question) associated with the image. As image-based text generation models
infer results from fusing computer vision domain and NLP domain, majority of
VL models are classified into “vision task” He et al. (2016); Ren et al.
(2015), which interprets the given image, and “text task” which generates the
associated texts. Most common methodologies used is the 2-stage approach,
where the vision model first extracts the features from the image that are
easier for the language model to interpret and train from. In the case of
OSCAR Li et al. (2020), the maximum number of objects are detected from the
object detection model and are passed on to a language model with the object’s
class name. In the case of X-VLM Zeng et al. (2021), inter-object relation was
used in training for better information sharing between image-text models. In
the case of OFA Wang et al. (2022), conveying information that is much more
comprehensible for language models was the focus to enhancing performance. As
stated above, there has been a growing body of research that explores how
recent VL models can better interpret images and deliver critical information
to the language model.

Then what evaluation method would be appropriate to compare which structure is
better at Vision-Language process? Perplexity Jelinek et al. (1977) at
language generation? BLEU Papineni et al. (2002) score? CIDEr Vedantam et al.
(2015)? These are all valid metrics to measure VL model’s performance, but
they all do not properly evaluate all aspects of VL model. Figure 1 depicts
results of captioning from several VL models. Each model came up with its own
answer for the image of a distorted human body. We human beings can easily
recognize differences between normal and distorted images, so VL models are
expected to be equivariant (exlpained in section 2.4) as well but that is not
the case. In view of the captioning results, all VL models seem to concentrate
more on local-context without considering distorted human body(a.k.a global
context of human body). Despite VL models being invariant(non-equivariant)
with captioning, each model has different understandings in global context of
human body. This is obvious because they all have their own feature extraction
stage on image. To perform evaluation on such VL models, we propose
Equivariance Score and dataset named as “Human Puzzle”. As a result, our
contributions are the following:

1\. We illustrate how Vision-Language models are biased towards local context
instead of global context.

2\. We present an evaluation dataset and metrics to measure global context
interpretation.

Figure 2: A pipeline from construction of Human Puzzle dataset to train the
dataset.

##  2 Related Works

###  2.1 Vision-Language Networks

As vision and language studies make progress, importance in the Vision-
Language model is also steadily rising. In recent years, OSCAR+, which
replaces image interpreting stage of OSCAR with VinVL Zhang et al. (2021), has
demonstrated notable performance. Through the method of placing an object’s
label with its extracted feature from the image  and sending it to the
transformer-encoder structure, OSCAR+ has achieved state-of-the-art
performances in numerous VL tasks and quickly became a dominant model in the
VL field. In a similar fashion, X-VLM also extracts features from images and
trains them to language model, but through many layers of contrastive learning
on inter-object relations, the model sought to understand not only the single
object context but also the relationship between objects in general. OFA also
tried to extract information from the image and made the language model
interpret the results. In addition, unique tokens were allocated to help the
language model have a better perception of the extracted information, and this
kind of process played a pivotal role in narrowing the gap between the
language and image models. In Vision Language Processing, researchers are
still studying various models to find a better way of extracting information
from images.

###  2.2 Image Networks

Human Puzzle dataset was first constructed by producing large amounts of data
with distorted context of human body. We then filtered by choosing ones where
the vision model could not identify correctly from the undistorted. Vision
models utilized in this process are ResNet He et al. (2016), Vision
Transformer Dosovitskiy et al. (2020), and Swin Transformer Liu et al. (2021),
which were selected based on prevalence. In ResNet, skip-connection was
applied to increase the model’s depth without vanishing gradient problem. As a
result, stable training while maintaining deep networks became possible,
propelling ResNet to be a dominant backbone in fields of vision. Vision
Transformer is a model that integrated the structure of a transformer, a model
that made compelling breakthroughs in the field of NLP. By partitioning images
into patches and treating them like text tokens, Vision Transformer shows
state-of-the-art performance comparable to conventional CNN-based models
regarding classification. Swin Transformer is a model derived from CNN’s
approach to overcome shortcomings of Transformer and likewise exhibits state-
of-the-art performance in classification.

###  2.3 Interpretation of Global Context

There have been many preceding attempts Huang et al. (2019) to analyze global
context in the field of vision. In prior studies, DeepLabV3 Chen et al. (2017)
and CCNet Huang et al. (2019) applied ASPP (Atrous Spatial Pyramid Pooling)
and CC-attention (Criss-Cross attention) to widen feature’s receptive field to
utilize global context, reporting high performances.

In Vision Language Processing, there also has been implicit attempts to
interpret global context in images. VL models are generally directed towards
extracting additional information from the image such as specifying object
relation with natural languages, giving positional information of detected
object to Language Model, translating image features into natural language,
and so on. These can all be seen as an attempt to make models have a better
grasp of understanding global context. In the case of OSCAR+, the most
dominant model in the field of VL that tried to convey positional information
with features and labels of detected object however, has been observed that
intentionally corrupting an object’s positional information has no impact on
its performance. Based on these findings, we guess that OSCAR+ has less
consideration in interpretation of global context .

###  2.4 Equivariance and Invariance

Figure 3: Face images with same local context, but different global context.
In ideal model equivariance, all models should give different results for each
image. Figure 4: Samples of Human Puzzle dataset: Normal images(True label)
above and distorted images(False label) below.

From previous works, equivariance is defined as change in output according to
corresponding change in input’s context. Invariance is the opposite of
equivariance where output does not change to change in input. Figure 3 simply
shows concept of equivariance. In vision task, many researchers try to boost
invariance to get better efficiency by using sub-sampling such as max-pooling.
This way of boosting invariance makes model focus more on shape or texture,
but gives limitation in understanding context such as objects’ relations. To
overcome this limitation, Romero et al. (2020) proposes “attentive group
equivariant convolutions” Romero et al. (2020) and shows that it has effect on
equivariant training. In addition, Sabour et al. (2017) leads model to be more
equivariant by applying "dynamic routing" Sabour et al. (2017). From insights
above, we believe that goal of VL tasks should lean towards equivariant model
because goal of VL is to mimic human expressions, and we humans are
equivariant. In this paper, We construct “Human Puzzle” dataset which contains
distorted context of human body and propose "Equivariance Score" that can
quantify the VL models’ equivariance.

##  3 Framework for Global Context and Equivariance

In Vision-Language Processing, understanding the global context of an image is
an evident matter of emphasis for human-like understanding. We suggest a
simple approach to quantify how much a model is comprehending global context.
Figure 2 is an overview of our approach. Each image is forwarded to the VL
model, where normal image is set to True, while image with distorted global
context in human body is set to False.

The output of the model is then sent to a classifier instead of word embedding
to distinguish its boolean value. Over this course, weights of the VL model
are frozen, and only the classifier proceeds to train from the process. If a
model is well capable of understanding global context, it would excel in
classifying normal images from images with distorted human body, and this
suggests the model is equivariant. If it can’t classify well, then the model
would be seen as invariant. In section 6, we suggest the quantifying metric of
equivariance as Equivariance Score, and this metric with Human Puzzle dataset
could reflect how model understands global context of human body. The above
process is applicable to both VL models and vision models that take image as
an input.

##  4 Dataset Construction

Our suggestion to measure if a model is equivariant at global context is
comparing the output of the normal image with that of image with distorted
context. For this process, a complementary dataset is essential. To prepare
normal images and images with distorted global contexts, we had to construct
synthesized data. Basic concept originates from the fact that VL models do not
fully understand human shape as shown in Figure 1. To acquire enough synthetic
data, raw data with humans separated from the background was necessary.
VideoMatte240K Lin et al. (2021) dataset was selected as the source of raw
data for human figures as it met such criteria. Total of 958 human figure
images were collected by sampling the frame from VideoMatte240K dataset, which
filmed humans in diverse actions without any background. In addition, 193
background images from PhotoMatte85 dataset were used as source of raw data.

model | pre-trained dataset | entropy | acc. | equivariance score
---|---|---|---|---
Virtex Desai and Johnson (2021) | COCO Chen et al. (2015) | 0.039 | 91.57% | 0.736
OSCAR+ Li et al. (2020) | COCO Chen et al. (2015) \+ $extra^{*}$ | 0.299 | 93.0% | 0.740
X-VLM Zeng et al. (2021) | COCO Chen et al. (2015) | 0.216 | 96.42% | 0.830
OFA Wang et al. (2022) | COCO Chen et al. (2015) | 0.0928 | 96.92% | 0.914
ViT Dosovitskiy et al. (2020) | ImageNet-1K Deng et al. (2009) | 0.806 | 63.7% | 0.473
DViT Zhou et al. (2021) | ImageNet-1K Deng et al. (2009) | 0.678 | 73.72% | 0.500
SwinT Liu et al. (2021) | ImageNet-1K Deng et al. (2009) | 0.205 | 95.3% | 0.821
ResNet He et al. (2016) | ImageNet-1K Deng et al. (2009) | 0.096 | 95.5% | 0.910
SE-ResNet Hu et al. (2018) | ImageNet-1K Deng et al. (2009) | 0.079 | 96.1% | 0.920

∗ OSCAR+ follow the datasets settings of VinVLZhang et al. (2021)

Table 1: Comparison of quantitative performance and Equivariance Score for
each model on Human Puzzle dataset.

model | pre-trained dataset | entropy | acc. | equivariance score
---|---|---|---|---
FCN Long et al. (2015) | Cityscapes Cordts et al. (2016) | 0.538 | 89.1% | 0.544
DeepLabV3 Chen et al. (2017) | Cityscapes Cordts et al. (2016) | 0.447 | 84.2% | 0.633
CCNet Huang et al. (2019) | Cityscapes Cordts et al. (2016) | 0.382 | 84.3% | 0.677

Table 2: Equivariance Score performance comparison for CNN-based networks.

##  5 Human Puzzle Dataset

Generating tens of millions of images by mix-and-matching 958 human images
with 193 background images is not difficult. Yet, we cannot argue that the
entirety of arbitrarily modified and synthesized image dataset is suitable for
qualifying degree of interpretation on global context as we suggested. Hence,
highly recognized vision models were used to select relevant and meaningful
images from the pool of millions. The methods used to impair the global
contexts of human images are simple. Methods of slicing, reflecting,
relocating, etc. depending on arbitrary offsets were applied while maintaining
the axis of the original image. Raw synthetic data reached 2.6M by combining
background images with all possible numbers of cases of modification on
original human images. To filter out the 2.6M images, ResNet, ViT Dosovitskiy
et al. (2020), and SwinT Liu et al. (2021) were used. After training each
model’s classifier while the individual weights were frozen, images that
models could not accurately classify from a separate test set were chosen.
Finalists for the dataset were selected from images where 2 out of 3 models
gave incorrect classification. In the end, we manually filtered out the
remaining images to create the Human Puzzle dataset.

##  6 Equivariance Score

The aim of our research is not to classify distorted data, but to carry out
regression analysis on the extent of global context interpretation. Hence,
simply measuring accuracy would be inappropriate, while applying entropy would
serve as an adequate metric to quantify a model’s rate of confusion. We
suggest an entropy-based metric like Equation 2 to assess the model’s global
context comprehension level.

$H(p)=-plog_{2}p-(1-p)log_{2}(1-p),$ (1)

$p\in P$ refers to the probability value after applying softmax on Human
Puzzle-trained model’s logits. Applying Equation 1, we calculate the entropy
of this probability $p$ to finally calculate Equivariance Score written in
Equation 2.

$\textit{Equivariance Score}=\\\
\frac{\sum_{t}(1-H(p_{t}))+\sum_{f}H(p_{f})}{N_{t}+N_{f}},$ (2)

In Equation 2, $p_{t}$ and $p_{f}$ ($p_{t},p_{f}\in P$) represent the
probabilities of correct and incorrect prediction. Here, $N_{t}$ and $N_{f}$
refer to the number of $p_{t}$ and $p_{f}$ respectively. Through Equation 2,
the value of rightfully predicted results becomes substantial, while the value
of wrongfully predicted results goes minimal. Theoretically, if all the
predicted answers are correct, and entropy of the result is also zero, then a
perfect score would be given.

##  7 Experiment

###  7.1 Experiment Details

For training of the Human Puzzle dataset in this study, we used encoders of
the pre-trained VL and vision model. In the case of VL, since most of the
models use transformers, the last hidden layer output was applied. In the case
of CNN-based vision model, since ResNet backbone forms the majority, features
of ResNet’s fourth stage were applied. In the case of DeepLabV3, and CCNet, we
extracted from each model’s ASPP, and CC-attention module. We designed the
extracted features from each of the models to be forwarded to independent
individual fully-connected layers to carry out binary prediction. This was
done to differentiate the features with distorted global context.

###  7.2 Quantitative Result

Table 1 shows the result of the measured Equivariance Score trained and
evaluated with Human Puzzle dataset. We proceeded with training and testing on
the latest VL models and vision models. In the case of OSCAR+, it was trained
with tremendous amounts of data compared to other VL models and performed very
well in most of VL tasks. Nonetheless, OSCAR+ only extracts as many features
as it can from images without processing relational information, leading us to
predict relatively low Equivariance Score, and this was found to be true based
on our experiment results. In the case of X-VLM, it was aimed to apply global
context to its training. By constructing inter-object relations as a separate
dataset, X-VLM carried out step-by-step contrastive learning. Despite X-VLM
being trained on a much smaller dataset compared to OSCAR+, it demonstrated a
much better Equivariance Score. Unified vocab suggested in OFA consists of not
only the vocab’s text but also locational information or even image itself.
This was implemented to convert transferred information to the model for
better interpretation. We expected that unified vocab will enhance global
context interpretation, and OFA in our experiment result indeed conveyed the
highest Equivariance Score.

We also proceeded on experimenting on vision models. In general, CNN-based
models showed higher Equivariance Score over transformer-based models. This
can be seen as a result of structural characteristics of CNN, where features
of the input layer are forwarded to the end layer.

Drastic difference between ViT and SwinT’s Equivariance Score is implying this
as well. ViT showed that it can apply images to transformer by slicing images
into patches, and to overcome ViT’s shortcoming of inductive bias, SwinT
applied shifted window structure ideated from CNN. Drawing from this, we
expected SwinT’s Equivariance Score to be higher than ViT’s, and this was
indeed true, as SwinT’s Equivariance Score was closer to CNN-based model, far
exceeding that of ViT.

Furthermore, we explored studies exploiting the global context in the field of
vision to extensively validate the usefulness of our Human Puzzle dataset.
Table 2 shows results of evaluating context score on semantic segmentation
models. Amongst them, FCN did not put global context in consideration, and
each model all used the ResNet backbone. FCN displayed the highest performance
in classification on our Human Puzzle dataset, but in contrast, scored the
lowest on Equivariance Score. Meanwhile, other models that utilized global
context measured higher in Equivariance Score compared to FCN. This result
establishes that our Human Puzzle dataset and our suggested Equivariance Score
are, indeed, valid.

##  8 Conclusion

In this paper, we propose the Equivariance Score to measure Vision-Language
model’s degree of equivariance and provide the Human Puzzle dataset. These two
can help future works with measuring how VL models are equivariant in global
context. Overall, models that attempted to extract information on global
context measured relatively higher Equivariance Score than models that did
not, and our Equivariance Score is a valid metric to indicate how much such
attempts are meaningful. The proposed method in our paper is applicable not
only to the fields of VL, but also in the fields of vision and can serve as a
basis for research in many different domains.

##  9 Limitation and Discussion

Human Puzzle dataset and Equivariance Score metric suggested in this paper are
not just restricted to VL, but in all fields that work with images. However,
the dataset has its own limitations in that it only considers the global
context of human body. In consequence, there exists a need to construct
appropriate dataset with distorted global context in both humans and diverse
objects with backgrounds as well. Through this additional construction,
metrics to quantify VL and vision models’ equivariance in global context can
become more robust. Furthermore, Equivariance Score suggested in this paper is
a new measuring metric that is previously nonexistent, implying focus on
global context for future research in VL and vision.

## References

  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015.  Vqa: Visual question answering.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2425–2433.
  * Chen et al. (2017) Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. 2017.  Rethinking atrous convolution for semantic image segmentation.  _arXiv preprint arXiv:1706.05587_.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_.
  * Cordts et al. (2016) Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. 2016.  The cityscapes dataset for semantic urban scene understanding.  In _Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Deng et al. (2009) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009.  ImageNet: A Large-Scale Hierarchical Image Database.  In _CVPR09_.
  * Desai and Johnson (2021) Karan Desai and Justin Johnson. 2021.  Virtex: Learning visual representations from textual annotations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11162–11173.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778.
  * Hu et al. (2018) Jie Hu, Li Shen, and Gang Sun. 2018.  Squeeze-and-excitation networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 7132–7141.
  * Huang et al. (2019) Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. 2019.  Ccnet: Criss-cross attention for semantic segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 603–612.
  * Jelinek et al. (1977) Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. 1977.  Perplexity—a measure of the difficulty of speech recognition tasks.  _The Journal of the Acoustical Society of America_ , 62(S1):S63–S63.
  * Li et al. (2020) Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _European Conference on Computer Vision_ , pages 121–137. Springer.
  * Lin et al. (2021) Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian L Curless, Steven M Seitz, and Ira Kemelmacher-Shlizerman. 2021.  Real-time high-resolution background matting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8762–8771.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10012–10022.
  * Long et al. (2015) Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015.  Fully convolutional networks for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3431–3440.
  * Pan et al. (2020) Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. 2020.  X-linear attention networks for image captioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10971–10980.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28.
  * Romero et al. (2020) David Romero, Erik Bekkers, Jakub Tomczak, and Mark Hoogendoorn. 2020.  Attentive group equivariant convolutional networks.  In _International Conference on Machine Learning_ , pages 8188–8199. PMLR.
  * Sabour et al. (2017) Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017.  Dynamic routing between capsules.  _Advances in neural information processing systems_ , 30.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4566–4575.
  * Wang et al. (2022) Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022.  Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  _arXiv preprint arXiv:2202.03052_.
  * Wang et al. (2021) Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021\.  Simvlm: Simple visual language model pretraining with weak supervision.  _arXiv preprint arXiv:2108.10904_.
  * Zeng et al. (2021) Yan Zeng, Xinsong Zhang, and Hang Li. 2021.  Multi-grained vision language pre-training: Aligning texts with visual concepts.  _arXiv preprint arXiv:2111.08276_.
  * Zhang et al. (2021) Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021.  Vinvl: Revisiting visual representations in vision-language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5579–5588.
  * Zhou et al. (2021) Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. 2021.  Deepvit: Towards deeper vision transformer.  _arXiv preprint arXiv:2103.11886_.


11institutetext:  Department of Biomedical Engineering, School of Biomedical
Engineering $\&$ Imaging Sciences, King’s College London, London, UK
22institutetext: NVIDIA 33institutetext: DeepMind 44institutetext: Oxford
Nanopore Technologies, Gosling Building, Oxford Science Park, Edmund Halley
Rd, Littlemore, Oxford OX4 4DQ, UK 55institutetext: Queen Square Institute of
Neurology, University College London, London, UK
@ Corresponding Author
# Work done while at NVIDIA

# Morphology-preserving Autoregressive 3D Generative Modelling of the Brain

Petru-Daniel Tudosiu@∗ 11 0000-0001-6435-5079 Walter Hugo Lopez Pinaya 11
0000-0002-2795-9209 Mark S. Graham 11 0000-0002-4170-1095 Pedro Borges 11
0000-0001-5357-1673 Virginia Fernandez 11 0000-0001-5984-197X Dai Yang 22
Jeremy Appleyard 22 Guido Novati# 33 Disha Mehra 22 Mike Vella# 44 Parashkev
Nachev 55 0000-0002-2718-4423 Sebastien Ourselin 11 0000-0002-5694-5340 Jorge
Cardoso 11 0000-0003-1284-2558

###### Abstract

Human anatomy, morphology, and associated diseases can be studied using
medical imaging data. However, access to medical imaging data is restricted by
governance and privacy concerns, data ownership, and the cost of acquisition,
thus limiting our ability to understand the human body. A possible solution to
this issue is the creation of a model able to learn and then generate
synthetic images of the human body conditioned on specific characteristics of
relevance (e.g., age, sex, and disease status). Deep generative models, in the
form of neural networks, have been recently used to create synthetic 2D images
of natural scenes. Still, the ability to produce high-resolution 3D volumetric
imaging data with correct anatomical morphology has been hampered by data
scarcity and algorithmic and computational limitations. This work proposes a
generative model that can be scaled to produce anatomically correct, high-
resolution, and realistic images of the human brain, with the necessary
quality to allow further downstream analyses. The ability to generate a
potentially unlimited amount of data not only enables large-scale studies of
human anatomy and pathology without jeopardizing patient privacy, but also
significantly advances research in the field of anomaly detection, modality
synthesis, learning under limited data, and fair and ethical AI. Code and
trained models are available at: https://github.com/AmigoLab/SynthAnatomy.

###### Keywords:

Transformers VQ-VAE Generative Modelling Neuroimaging Neuromorphology.

##  1 Introduction

Current advances in the application of deep learning (DL) in medical imaging
were driven by substantial initiatives and challenges such as UK Biobank
(UKB)[1], Alzheimer’s Disease Neuroimaging Initiative (ADNI)[2], and the
Medical Segmentation Decathlon[3]. However, these are relatively small
compared to computer vision datasets. Owing to the lack of access to
sufficient data due to privacy concerns, medical imaging data is not fully
leveraging DL’s full potential and this hinders its translation from research
to the clinical environment. State-of-the-art (SOTA) algorithms rely on a
handful of highly curated datasets which could lead to biases due to
imbalanced demographics or acquisition parameters, that may negatively affect
their performance for certain populations. A solution to this problem could
come from the generative modelling of the underlying available data to balance
the prevalence of confounding variables in the training dataset.

While semi-supervised 3D generative modelling of the brain has been steadily
explored and improved [4, 5, 6], progress in unsupervised generative modelling
has been more limited. Generative Adversarial Network (GAN) based approaches,
which suffer from memory constraints and stability issues, have mostly been
trained on low-resolution 3D images [7, 8, 9], having only recently been able
to synthesise full resolution images via learning partial sub-volumes [10].
Whereas previous methods quantify sample diversity using classic metrics such
as Multi-Scale Structural Similarity Index (MS-SSIM) [11], distribution
alignment via Fréchet Inception Distance (FID) [12] and Maximum Mean
Discrepancy (MMD) [13], none have quantified if the generated data preserves
the morphological characteristics of the data – crucial if we are to use such
methods.

Recently, autoregressive models have achieved SOTA results synthesising high
resolution natural images [14, 15, 16]. This was accomplished by employing a
compression model, namely a Vector Quantised-Variational Autoencoder (VQ-VAE)
[17, 14], to project the images into a discrete latent representation where
the images’ likelihood becomes tractable. An attention-based Transformer
network [18, 19] is then used to model the product of conditional
distributions by maximising the expected log-likelihood of the training data.

Following [20] as part of the Synthetic Data Desiderata, a good synthetic
dataset should share many if not all statistical properties of the real
dataset. One such property, if not the most important, of synthetic structural
medical images is their morphological correctness. Covariates of interest such
as demographic and pathological ones determine the phenotype of each subject
which in turn contributes to the population-level morphological statistics.
Without it, any development done on the synthetic data as part of the Train on
Synthetic, Test on Real [21] paradigm could suffer from higher domain
distribution shifts slowing down the development. Furthermore, without
morphological assessment, any hypothesis tested on the synthetic data would be
rendered highly uncertain.

In this study, we scale and optimise VQ-VAE and Transformer models for high-
resolution volumetric data, aiming to learn the data distribution of both
radiologically healthy and pathological brains. A thorough morphological
evaluation is employed by using Voxel-Based Morphometry (VBM) [22] and
volumetric analysis using Geodesic Information Flows (GIF) [23], demonstrating
that synthetic data generated by the proposed model preserves the
morphological characteristics and phenotype of the data.

##  2 Background

Our model is based on the two-stage architecture introduced by [17, 14] and
extended by [15], where a VQ-VAE model is used to project a high-resolution
image into a compressed latent representation and a transformer is trained to
maximize the likelihood of the flattened representations.

###  2.1 VQ-VAE

The VQ-VAE [17, 14] is comprised of an encoder $E$ that projects the input
image $\mathbf{x}\in{\mathbb{R}}^{H\times W\times D}$ to a latent
representation space $\widehat{\mathbf{z}}\in\mathbb{R}^{h\times w\times
d\times{n}_{z}}$ where ${n}_{z}$ is the latent embedding vector’s
dimensionality. Afterwards, an element-wise quantization is done for each
spatial code $\widehat{\mathbf{z}}_{ijk}\in\mathbb{R}^{n_{z}}$ onto its
nearest vector $e_{k}\in\mathbb{R}^{n_{z}},k\in 1,...,K$ from a codebook,
where $K$ denotes the vocabulary size of the codebook, obtaining
$\widehat{\mathbf{z}}_{q}$. The codebook’s elements are learned in an online
manner, together with the other model’s parameters. Based on the quantized
latent space, a decoder $G$ tries to reconstruct the observations
$\widehat{\mathbf{x}}\in\mathbb{R}^{H\times W\times D}$. By replacing each of
the codebook elements vector $\widehat{\mathbf{z}}_{q}\in\mathbb{R}^{h\times
w\times d\times n_{z}}$ with their associated index $k$, the latent discrete
representation is obtained.

###  2.2 Transformer

Transformers models and their associated self-attention mechanisms can capture
the interactions between inputs regardless of their relative positioning. Due
to this, the attention mechanism scales quadratically with the size of the
input sequence. Since the VQ-VAE’s latent discrete representation when applied
to volumetric medical data is 3D and thus large in scale, standard
transformers do not scale to the necessary sequence length. Recently, multiple
advances have made Transformers more efficient [24]; models such as the
Performer, with its FAVOR+ linear scaling attention approximation [19] offers
a good compromise between accurately modelling long-sequences while preserving
a reasonable computational complexity [24]. Thus the Performer is used to
model the latent sequences; by minimizing the conditional distribution of
codebook indices $p(s_{i})=p(s_{i}|s_{<i})$ on the flattened 1D sequences of
the 3D latent discrete representations, the data log-likelihood is maximized
in an autoregressive fashion.

##  3 Methods

###  3.1 Descriptive Quantization for Transformer Usage

To create a Transformer-based generative model of the brain, the image volume
needs to be transformed into a 1D sequence of tokens. To achieve this, a VQ-
VAE model that reduces the overall spatial size by a factor of 4096, allowing
an input image of size X to be represented by a sequence of 1400 tokens. This
1400-long token sequence is learnt in an online fashion together with the VQ-
VAE model by using the Exponential Moving Average (EMA) algorithm [17, 14] as
per Eq. 1.

${\mathcal{L}}_{VQ-
VAE}(\mathbf{x},G(\mathbf{\widehat{\mathbf{z}}_{q}}))={\mathcal{L}}_{Rec}+{\mathcal{L}}_{Adv}+\|sg[E(\mathbf{x})]-\mathbf{\widehat{\mathbf{z}}_{q}}\|_{2}^{2}+\beta\|sg[\mathbf{\widehat{\mathbf{z}}_{q}}]-E(\mathbf{x})\|_{2}^{2}$

(1)

$N_{i}^{(t)}:=N_{i}^{(t-1)}*\gamma+n_{i}^{(t)}(1-\gamma),\quad
m_{i}^{(t)}:=m_{i}^{(t-1)}*\gamma+\sum_{j}^{n_{i}^{(t)}}E(x)_{i,j}^{(t)}(1-\gamma),\quad\widehat{\mathbf{z}}_{qi}^{(t)}:=\frac{m_{i}^{(t)}}{N_{i}^{(t)}}$

(2)

where $sg$ stands for the stop-gradient operation. As per [17, 14], the third
loss component in Eq. 1 is replaced by Eq. 2, where $n_{i}^{(t)}$ stands for
the number of vectors in $E(\mathbf{x})$ that will be quantized to codebook
element $\widehat{\mathbf{z}}_{qi}$. The hyper-parameters $\gamma$ and $\beta$
control the decay of the EMA and the commitment of the encoder output to a
certain quantized element respectively.

For the codebook to be perceptually rich, a loss similar to [15, 25] which is
formed by the first and second elements of Eq. 1 as defined bellow:

${\mathcal{L}}_{Rec}=\|\mathbf{x}-\widehat{\mathbf{x}}\|_{1}+\||FFT(\mathbf{x})|-|FFT(\widehat{\mathbf{x}})|\|_{2}+{\mathcal{LPIPS}}_{0.5}(\mathbf{x},\widehat{\mathbf{x}})$

(3)

Where the first term is a pixel-space L1 norm, the second term is the L2 norm
of the image’s Fourier representations based on [26] which aims at
facilitating high-frequency feature preservation, the third term is the LPIPS
[27] loss using AlexNet applied on 50% of slices on each axis. Lastly, the
${\mathcal{L}}_{Adv}$ is based on a Patch-GAN discriminator-based adversarial
loss [28, 15], replacing the original loss by the LS-GAN [29] one (see Eq 4):

$\begin{array}[]{l}\min_{D}{\mathcal{L}}_{LSGAN}(D)=\frac{1}{2}\mathbb{E}_{\mathbf{x}\sim
p_{\mathrm{data}}(\mathbf{x})}\left[(D(\mathbf{x})-1)^{2}\right]+\frac{1}{2}\mathbb{E}_{\mathbf{x}\sim
p_{\mathbf{data}}(\mathbf{x})}\left[(D(G(\widehat{\mathbf{x}})))^{2}\right]\\\
\\\ \min_{G}{\mathcal{L}}_{LSGAN}(G)=\frac{1}{2}\mathbb{E}_{\mathbf{x}\sim
p_{\mathbf{x}}(\mathbf{z})}\left[(D(G(\widehat{\mathbf{x}}))-1)^{2}\right]\end{array}$

(4)

Each of these losses independently contributes to model training stability and
reconstruction quality.

###  3.2 Autoregressive Modelling of the Brain

The VQ-VAE model was first trained on T1w MRI images of neurologically healthy
subjects from UKB [1] until convergence, and then their $z_{q}$
representations were extracted. Afterwards, further fine-tuning on the
pathological dataset formed from the baseline T1w MRI scans of ADNI [2]
subjects was done until over-fitting was noticed, at which point the ADNI
subjects’ $z_{q}$ representation was also extracted. This paradigm was chosen
since in [30] it was shown to either be on par or better compared to training
a VQ-VAE model only on the pathological dataset. Furthermore, we aim also to
highlight that the pre-trained model can be fine-tuned and learn new
morphology, in this case, a pathological one, thus increasing the usefulness
of the UKB trained VQ-VAE as a pretrained model for the community.

In order to ensure a higher quality of the Transformer’s samples, the top 1%
generated samples, based on the score obtained by averaging the Patch-GAN
discriminator output, were used in this work.

As the VQ-VAE representations cover all phenotypes, a separate Transformer
model has been trained on the latent representations of different sub-
populations to model individual morphological subgroups. More specifically, to
demonstrate the morphological phenotype preservation, the UKB [1] dataset was
partitioned into young vs. old sub-populations, and small vs. big ventricles
sub-populations. We defined all of these groups based on the first and last of
five quantiles based on "age when attended assessment centre" (21003-2.0) and
"volume of the ventricular cerebrospinal fluid" (25004-2.0) UKB variables,
respectively. To test the preservation of disease morphology, we split the
ADNI dataset into cognitively normal (CN) and Alzheimer’s disease (AD)
subgroups based on the "diagnosis/scan category assignment field".

##  4 Experiments and Results

The performance of the proposed model is assessed in two ways: first, the
quality of generated samples is measured according to image fidelity metrics
commonly used in generative models; second, we verify if the morphological
characteristics of a population and the differences between sub-populations
are preserved when comparing real and synthetic data. We compared our model to
a baseline volumetric VAE model. The models by [8, 9, 31, 29] underwent
extensive hyper-parameter exploration at the original resolutions but failed
to converge on our data. Only the VAE results are thus presented as a
baseline.

###  4.1 Quantitative Image Fidelity Evaluation

Figure 1: Synthetic samples. On the left UKB small ventricles and on the right
UKB big ventricles.

Similarly to [8, 9], we use the FID [12] to assess the visual quality of the
generated images. Since originally the metric is based on a pretrained
Inception V3 network on 2D natural images, it cannot be applied on 3D volumes
directly, so here it is applied on the middle slice of each axis and reported
individually. To measure the quality of the 3D samples, batch-wise MMD with a
dot product as the kernel is being used as suggested in [8, 9]. Briefly, MMD
quantifies the distance between the distributions with finite sample estimates
in kernel functions in the reproducing kernel Hilbert space [13]. Lastly, to
estimate the diversity of the generated images, MS-SSIM is being used in a
pair-wise fashion between the generated synthetic samples as in [8, 9]. For
easy comparison, all metrics have also been calculated between each sub-
population’s real images such that a ground truth baseline is also offered.

Model | Dataset | Population | FID (Ax|Cor|Sag) | ${bMMD}^{2}$ | MS-SSIM
---|---|---|---|---|---
Real | UKB | Young | 0.35 | 0.85 | 0.42 | $0.00208_{\pm 0.00026}$ | $0.65_{\pm 0.08}$
VQ-VAE (ours) | UKB | Young | 31.04 | 57.19 | 57.19 | $0.00903_{\pm 0.00090}$ | $0.68_{\pm 0.03}$
VAE | UKB | Young | 193.56 | 302.74 | 251.34 | $0.02757_{\pm 0.00091}$ | $0.15_{\pm 0.001}$
Real | UKB | Old | 1.16 | 1.42 | 0.37 | $0.00217_{\pm 0.00045}$ | $0.65_{\pm 0.07}$
VQ-VAE (ours) | UKB | Old | 33.68 | 60.60 | 78.82 | $0.00887_{\pm 0.00104}$ | $\mathbf{0.67_{\pm 0.03}}$
VAE | UKB | Old | 234.86 | 289.21 | 242.18 | $0.02622_{\pm 0.00044}$ | $0.15_{\pm 0.001}$
Real | UKB | Small Ventricles | 1.74 | 1.99 | 0.87 | $0.00220_{\pm 0.00044}$ | $0.67_{\pm 0.07}$
VQ-VAE (ours) | UKB | Small Ventricles | 28.33 | 58.23 | 76.68 | $0.00892_{\pm 0.00106}$ | $\mathbf{0.70_{\pm 0.04}}$
VAE | UKB | Small Ventricles | 206.92 | 318.37 | 258.17 | $0.02836_{\pm 0.00078}$ | $0.14_{\pm 0.001}$
Real | UKB | Big Ventricles | 1.15 | 1.44 | 0.53 | $0.00231_{\pm 0.00041}$ | $0.64_{\pm 0.05}$
VQ-VAE (ours) | UKB | Big Ventricles | 36.02 | 57.76 | 76.51 | $0.00937_{\pm 0.00069}$ | $0.68_{\pm 0.04}$
VAE | UKB | Big Ventricles | 215.37 | 293.97 | 244.84 | $0.02738_{\pm 0.00058}$ | $0.16_{\pm 0.001}$
Real | ADNI | Cognitively Normal | 21.49 | 17.31 | 9.34 | $0.00123_{\pm 0.00021}$ | $0.56_{\pm 0.05}$
VQ-VAE (ours) | ADNI | Cognitively Normal | 53.88 | 93.62 | 112.32 | $0.01558_{\pm 0.00348}$ | $0.71_{\pm 0.08}$
VAE | ADNI | Cognitively Normal | 233.59 | 397.52 | 421.04 | $0.02562_{\pm 0.00119}$ | $0.14_{\pm 0.05}$
Real | ADNI | Alzheimer’s Diseased | 9.08 | 16.85 | 13.49 | $0.00167_{\pm 0.00034}$ | $0.55_{\pm 0.13}$
VQ-VAE (ours) | ADNI | Alzheimer’s Diseased | 87.75 | 51.74 | 90.95 | $0.01562_{\pm 0.00304}$ | $0.61_{\pm 0.11}$
VAE | ADNI | Alzheimer’s Diseased | 235.33 | 332.70 | 340.78 | $0.02804_{\pm 0.00177}$ | $0.12_{\pm 0.06}$

Table 1: The ${bMMD}^{2}$ and MS-SSIM were calculated on 3D generated images
while FID was done middle-slices-wise of generated volumes.

Across the board, as is described in Table 1, both in regards to the sub-
populations and axial, coronal and sagittal slices, the FID of the VQ-VAE
model outperforms the VAE baseline by a high margin, showcasing the realistic
appearance of the sampled synthetic brains as seen in Fig. 3. The same can be
said for the $bMMD^{2}$, where the VQ-VAE is one order of magnitude smaller
for UKB sub-populations and substantially better for the ADNI sub-populations.
The difference in $bMMD^{2}$ performance between VQ-VAE’s UKB and ADNI sub-
populations might be because the ADNI dataset is considerably smaller than the
UKB one, and to circumvent that, the VQ-VAE compression model was firstly
trained on the UKB dataset and then fine-tuned on the ADNI one. Thus, the
$z_{q}$ representation fed into the Transformer, which is the generative model
per se, is not specialised for ADNI, but instead, it tries to encompass it.
Finally, MS-SSIM shows that the VQ-VAE achieves a life-like high diversity of
samples across all sub-populations, significantly surpassing the VAE. The
peculiar case of the ADNI AD sub-population might be attributed to the same
cause as the $bMMD^{2}$.

###  4.2 Morphological Evaluation

Figure 2: Thresholded uncorrected VBM t-statistics maps processed as per [32]
showcasing the morphological differences between two populations based on real
samples, VQ-VAE synthetic samples, and VAE synthetic samples. For UKB small
vs. big ventricles modulated CSF tissue segments were used, while for ADNI,
cognitively normal vs. AD modulated GM tissue segments were used.

To evaluate the morphological correctness of the synthetic samples, Voxel-
Based Morphometry (VBM) [22] was used to investigate the focal differences in
the brain anatomy of the sub-populations. At the core of VBM stands the
application of a generalised linear model and associated statistical tests
across all voxels of a group-aligned population, to identify morphological
differences in modulated tissue compartment between the selected groups.

The VBM analysis did not factor out any covariates available in the real
datasets since the generative process was unconditioned. All t-statistics maps
have been corrected to minimise the effects of low variance areas following
[32]. As shown in Fig. 2 the t-statistics maps between synthetic images
generated by the VQ-VAE strongly agree with the VBM maps of real data,
primarily when compared with the VAE baseline. In the UKB small ventricles vs.
big ventricles experiment, VQ-VAE again successfully models the ventricular
differences correctly compared to the real-data VBM maps, while the VAE model
strongly emphasizes them or exacerbates the subarachnoid CSF. Lastly, the
morphological differences between cognitively normal vs. AD subjects on the
ADNI dataset on the VQVAE generated data strongly preserve the known temporal
lobe and hippocampal atrophy patterns associated with AD, producing a VBM
t-map that strongly resembles the one from real data. Conversely, the VAE
fails to show coherent structural differences in the GM.

Furthermore, we compared the volumes of key brain regions between populations
of real and synthetic data. All images were segmented using GIF [23], a robust
multi-atlas based probabilistic segmentation model of the human brain which
segments the brain into non-overlapping hierarchical 155 regions. Based on
probabilistic segmentations, the total volume of each tissue was calculated,
and then we ran a two-sided t-test to assess if there was a statistically
significant difference between the tissue volumes of the real vs. synthetic
populations. The Bonferroni-corrected target p-value was 2.083e-05.

Model | Dataset | Population | Gray Matter | White Matter | CSF | Deep Gray Matter
---|---|---|---|---|---|---
Real | UKB | Young | $595_{\pm 32}$ | $460_{\pm 29}$ | $280_{\pm 21}$ | $40_{\pm 3}$
VQ-VAE (ours) | UKB | Young | $\mathbf{587_{\pm 24}}$ | $\mathbf{472_{\pm 20}}$ | $\mathbf{283_{\pm 11}}$ | $\mathbf{40_{\pm 2}}$
VAE | UKB | Young | $576_{\pm 1}$ | $444_{\pm 1}$ | $234_{\pm 1}$ | $34_{\pm 0}$
Real | UKB | Old | $587_{\pm 31}$ | $457_{\pm 29}$ | $283_{\pm 22}$ | $40_{\pm 3}$
VQ-VAE (ours) | UKB | Old | $\mathbf{576_{\pm 22}}$ | $\mathbf{465_{\pm 20}}$ | $310_{\pm 14}$ | $\mathbf{39_{\pm 2}}$
VAE | UKB | Old | $560_{\pm 1}$ | $434_{\pm 1}$ | $250_{\pm 1}$ | $33_{\pm 0}$
Real | UKB | Small Ventricles | $596_{\pm 30}$ | $462_{\pm 27}$ | $270_{\pm 17}$ | $41_{\pm 3}$
VQ-VAE (ours) | UKB | Small Ventricles | $\mathbf{594_{\pm 19}}$ | $477_{\pm 18}$ | $280_{\pm 12}$ | $\mathbf{41_{\pm 2}}$
VAE | UKB | Small Ventricles | $572_{\pm 1}$ | $444_{\pm 1}$ | $235_{\pm 1}$ | $35_{\pm 0}$
Real | UKB | Big Ventricles | $589_{\pm 34}$ | $459_{\pm 30}$ | $283_{\pm 19}$ | $41_{\pm 3}$
VQ-VAE (ours) | UKB | Big Ventricles | $574_{\pm 20}$ | $\mathbf{467_{\pm 17}}$ | $307_{\pm 15}$ | $39_{\pm 2}$
VAE | UKB | Big Ventricles | $570_{\pm 1}$ | $442_{\pm 1}$ | $246_{\pm 1}$ | $34_{\pm 0}$
Real | ADNI | Cognitively Normal | $530_{\pm 51}$ | $430_{\pm 40}$ | $309_{\pm 32}$ | $40_{\pm 5}$
VQ-VAE (ours) | ADNI | Cognitively Normal | $\mathbf{554_{\pm 19}}$ | $\mathbf{458_{\pm 18}}$ | $\mathbf{299_{\pm 12}}$ | $\mathbf{39_{\pm 2}}$
VAE | ADNI | Cognitively Normal | $\mathbf{518_{\pm 5}}$ | $\mathbf{440_{\pm 4}}$ | $258_{\pm 3}$ | $34_{\pm 1}$
Real | ADNI | Alzheimer’s Diseased | $526_{\pm 47}$ | $443_{\pm 36}$ | $330_{\pm 28}$ | $38_{\pm 3}$
VQ-VAE (ours) | ADNI | Alzheimer’s Diseased | $\mathbf{532_{\pm 38}}$ | $\mathbf{446_{\pm 20}}$ | $298_{\pm 27}$ | $\mathbf{37_{\pm 3}}$
VAE | ADNI | Alzheimer’s Diseased | $\mathbf{510_{\pm 5}}$ | $\mathbf{443_{\pm 4}}$ | $269_{\pm 4}$ | $34_{\pm 1}$

Table 2: Tissue volumes based on GIF’s probabilistic tissue segmentations.
Mean and standard deviations were rounded to the nearest $10^{3}$. The bold
values indicate the two-sided t-tests did not pass the statistical
significance threshold compared to the real data.

Overall, no significant volume differences were found between real and VQ-VAE
samples for most subgroups and tissue types, while significant differences
were found for most VAE statistics, demonstrating that the proposed method
strongly preserves tissue volumes. The CSF volumes of the VQ-VAE UKB small and
big ventricle populations were found to be statistically significantly
different from their real counterparts as shown in Table 2, following the VBM
results from Fig. 2, and which could explain the increase in the t-statistic
observed in the ventricular regions of the synthetic samples. On the other
hand, the GM volumes were not statistically significantly different,
corroborating the idea that the synthetic t-statistics are closer in magnitude
to the real ones. Note that the VAE samples were also found not to be
statistically significant in the ADNI AD/CT subset, but this is primarily due
to the larger variance and the conservative Bonferroni correction.

##  5 Conclusion

In this work, we propose a scalable and high-resolution volumetric generative
model of the brain that preserves morphology. VBM [22] and GIF [23] were used
to assess the morphological preservation, while FID [12] and $bMMD^{2}$ [13]
to measure distribution alignment between synthetic and real samples. We have
shown that the synthetic samples preserve healthy and pathological morphology
and that they are realistic images that closely align with the distribution of
the real samples. Future work should address the lack of conditioning and the
top 1% pruning to increase diversity and provide sampling control.
Furthermore, the generative model could be extended for disease progression
modelling, disentanglement of style and content, have its privacy preserving
capabilities examined, and scaled to include multiple pathologies. To the best
of our knowledge, this is the first morphologically preserving generative
model of the brain, which paves the way for an unlimited amount of clinically
viable data without jeopardizing patient privacy.

####  5.0.1 Acknowledgements

WHLP, MG, PB, MJC and PN are supported by Wellcome [WT213038/Z/18/Z].PTD is
supported by the EPSRC Research Council, part of the EPSRC DTP [EP/R513064/1].
FV is supported by Wellcome/ EPSRC Centre for Medical Engineering
[WT203148/Z/16/Z], Wellcome Flagship Programme [WT213038/Z/18/Z], The London
AI Centre for Value-based Healthcare and GE Healthcare. PB is also supported
by Wellcome Flagship Programme [WT213038/Z/18/Z] and Wellcome EPSRC CME
[WT203148/Z/16/Z]. PN is also supported by the UCLH NIHR Biomedical Research
Centre. The models in this work were trained on NVIDIA Cambridge-1, the UK’s
largest supercomputer, aimed at accelerating digital biology.

##  6 Appendix

###  6.1 VQ-VAEs

The VQ-VAE model has a similar architecture with [33] but in 3D. The encoder
uses strided convolutions with stride 2 and kernel size 4. There are four
downsamplings in this VQ-VAE, giving the downsampling factor $f=2^{4}$. After
the downsampling layers, there are three residuals blocks ($3\times 3\times 3$
Conv, ReLU, 1x1x1 Conv, ReLU). The decoder mirrors the encoder and uses
transposed convolutions with stride 2 and kernel size 4. All convolution
layers have 256 kernels. The $\beta$ in Eq. 1 is 0.25 and the $\gamma$ in Eq.
2 is 0.5. The codebook size was 2048 while each element’s size was 32.

###  6.2 Transformers

Performer’s111Implementation used: https://github.com/lucidrains/performer-
pytorch [19] has $L=24$ layers, $d=256$ embedding size, 16 multi-head
attention modules (8 are local attention heads with window size of 420), and
ReZero gating [34]. Before the raster style ordering input was RAS+ canonical
voxel representation oriented.

###  6.3 Losses

VQ-VAE’s pixel-space loss weight is 1.0, perceptual loss’ weight is 0.001,
frequency loss’ weight is 1.0. The LPIPS uses AlexNet. Adam has been used as
optimizer with an exponential decay of 0.99999. VQ-VAE’s learning rate was
0.000165, discriminator’s learning rate was 0.00005 and Performer’s
CrossEntropy learning rate was 0.001.

###  6.4 Datasets

All datasets have been split into training and testing sub-sets. The VQ-VAE
UKB sub-sets had 31740 and 3970 subjects respectively, while VQ-VAE ADNI had
648 and 82. All datasets have been first processed with a rigid body
registration such that they roughly fit the same field of view. Afterwards,
all samples are passed through the following transformations before being fed
into the VQ-VAE during training: first, they are being normalized to [0,1],
then tightly spatially cropped resulting in an image of size (160,224,160),
random affine (rotation range 0.04, translation range 2, scale range 0.05),
random contrast adjustment (gamma [0.99, 1.01]), random intensity shift
(offsets [0.0,0.05]), random Gaussian noise (mean 0.0, standard deviation
0.02), and finally, the images were thresholded to be in the range [0, 1.0].
For the Transformer, the UKB and ADNI datasets were split into sub-
populations. UKB was split into small ventricles (6388 and 108), big
ventricles (6321 and 156), young (6633 and 113), old (5137 and 106), while
ADNI was split into cognitively normal (118 and 29) and Alzheimer’s disease
(151 and 36). For the Transformer training, each ADNI sample has been
augmented 100 times and each augmentation’s index-based representation was
used for training it.

###  6.5 VBM analysis

For the Voxel-Based Morphometry (VBM), Statistical Parametric Mapping(SPM)
[35] package version 12.7486 was used with MATLAB R2019a. Before running the
statistical tests, the images must first undergo unified segmentation where
they were spatially normalized to a common template and simultaneously
segmented into the Gray Matter (GM), White Matter (WM), and Cerebrospinal
fluid (CSF) tissue segments based on prior probability maps and voxel
intensities. The unified segmentation was done with the default parameters:
Bias Regularisation (light regularisation 0.001), Bias FWHM (60mm cutoff), MRF
Parameter (1), Clean Up (Light Clean), Warping Regularisation ([0, 0.001, 0.5,
0.05, 0.2]), Affine Regularisation (ICBM space template - European brains),
Smoothness (0), Sampling Distance (3). As per standard practice when using
VBM, the group-aligned segmentations were modulated to preserve tissue volume,
and a smoothing kernel was applied to the modulated tissue compartments to
make the data conform to the Gaussian field model that underlines VBM and to
increase the sensitivity to detect structural changes. The smoothing was also
done with the default parameters with FWHM ([8, 8, 8]). For the VBM analysis,
a Two-sample t-test Design was used, with the following parameters:
Independence (Yes), Variance (Unequal), Grand mean scaling (No) and ANCOVA
(No). No covariates, masking or global normalisation have been used.

Appendix F - Additional Samples

Figure 3: Synthetic samples

## References

  * [1] Cathie Sudlow  “UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age”  In _PLoS medicine_ 12.3 Public Library of Science, 2015, pp. e1001779
  * [2] Clifford R Jack Jr  “The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods”  In _Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine_ 27.4 Wiley Online Library, 2008, pp. 685–691
  * [3] Amber L Simpson  “A large annotated medical image dataset for the development and evaluation of segmentation algorithms”  In _arXiv preprint arXiv:1902.09063_ , 2019
  * [4] Chee Keong Chong and Eric Tatt Wei Ho  “Synthesis of 3D MRI brain images with shape and texture generative adversarial deep neural networks”  In _IEEE Access_ 9 IEEE, 2021, pp. 64747–64760
  * [5] Wanyun Lin  “Bidirectional mapping of brain MRI and PET with 3D reversible GAN for the diagnosis of Alzheimer’s disease”  In _Frontiers in Neuroscience_ 15 Frontiers, 2021, pp. 357
  * [6] Filip Rusak  “3D brain MRI GAN-based synthesis conditioned on partial volume maps”  In _International Workshop on Simulation and Synthesis in Medical Imaging_ , 2020, pp. 11–20  Springer
  * [7] Alice Segato  “Data augmentation of 3D brain environment using Deep Convolutional Refined Auto-Encoding Alpha GAN”  In _IEEE Transactions on Medical Robotics and Bionics_ 3.1 IEEE, 2020, pp. 269–272
  * [8] Gihyun Kwon  “Generation of 3D brain MRI using auto-encoding generative adversarial networks”  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 2019, pp. 118–126  Springer
  * [9] Shibo Xing  “Cycle Consistent Embedding of 3D Brains with Auto-Encoding Generative Adversarial Networks”  In _Medical Imaging with Deep Learning_ , 2021
  * [10] Li Sun  “Hierarchical amortized training for memory-efficient high resolution 3D GAN”  In _arXiv preprint arXiv:2008.01910_ , 2020
  * [11] Zhou Wang  “Multiscale structural similarity for image quality assessment”  In _The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003_ 2, 2003, pp. 1398–1402  Ieee
  * [12] Martin Heusel  “Gans trained by a two time-scale update rule converge to a local nash equilibrium”  In _Advances in neural information processing systems_ 30, 2017
  * [13] Arthur Gretton  “A kernel two-sample test”  In _The Journal of Machine Learning Research_ 13.1 JMLR. org, 2012, pp. 723–773
  * [14] Ali Razavi  “Generating diverse high-fidelity images with vq-vae-2”  In _Advances in neural information processing systems_ 32, 2019
  * [15] Patrick Esser  “Taming transformers for high-resolution image synthesis”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 12873–12883
  * [16] Jiahui Yu  “Vector-quantized image modeling with improved vqgan”  In _arXiv preprint arXiv:2110.04627_ , 2021
  * [17] Aaron Van Den Oord and Oriol Vinyals  “Neural discrete representation learning”  In _Advances in neural information processing systems_ 30, 2017
  * [18] Ashish Vaswani  “Attention is all you need”  In _Advances in neural information processing systems_ 30, 2017
  * [19] Choromanski Krzysztof  “Rethinking attention with performers”  In _Proceedings of ICLR_ , 2021
  * [20] James Jordon  “Synthetic Data–what, why and how?”  In _arXiv preprint arXiv:2205.03257_ , 2022
  * [21] Cristóbal Esteban  “Real-valued (medical) time series generation with recurrent conditional gans”  In _arXiv preprint arXiv:1706.02633_ , 2017
  * [22] John Ashburner and Karl J Friston  “Voxel-based morphometry—the methods”  In _Neuroimage_ 11.6 Elsevier, 2000, pp. 805–821
  * [23] M Jorge Cardoso  “Geodesic information flows: spatially-variant graphs and their application to segmentation and fusion”  In _IEEE transactions on medical imaging_ 34.9 IEEE, 2015, pp. 1976–1988
  * [24] Yi Tay  “Long Range Arena: A Benchmark for Efficient Transformers”  In _International Conference on Learning Representations_ , 2020
  * [25] Mark S Graham  “Transformer-based out-of-distribution detection for clinically safe segmentation”  In _Medical Imaging with Deep Learning_ , 2022
  * [26] Prafulla Dhariwal  “Jukebox: A generative model for music”  In _arXiv preprint arXiv:2005.00341_ , 2020
  * [27] Richard Zhang  “The unreasonable effectiveness of deep features as a perceptual metric”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 586–595
  * [28] Phillip Isola  “Image-to-image translation with conditional adversarial networks”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134
  * [29] Xudong Mao  “Least squares generative adversarial networks”  In _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2794–2802
  * [30] Petru-Daniel Tudosiu  “Neuromorphologicaly-preserving Volumetric data encoding using VQ-VAE”  In _arXiv preprint arXiv:2002.05692_ , 2020
  * [31] Ishaan Gulrajani  “Improved training of wasserstein gans”  In _Advances in neural information processing systems_ 30, 2017
  * [32] Gerard R Ridgway  “The problem of low variance voxels in statistical parametric mapping; a new hat avoids a ‘haircut”’  In _Neuroimage_ 59.3 Elsevier, 2012, pp. 2131–2141
  * [33] Walter Hugo Lopez Pinaya  “Unsupervised Brain Anomaly Detection and Segmentation with Transformers”  In _Medical Imaging with Deep Learning_ , 2021, pp. 596–617  PMLR
  * [34] Thomas Bachlechner  “Rezero is all you need: Fast convergence at large depth”  In _Uncertainty in Artificial Intelligence_ , 2021, pp. 1352–1361  PMLR
  * [35] John Ashburner  “SPM12 manual”  In _Wellcome Trust Centre for Neuroimaging, London, UK_ 2464, 2014, pp. 4