# Image Inpainting Guided by Coherence Priors of Semantics and Textures

Liang Liao1 Jing Xiao2 Zheng Wang1 Chia-Wen Lin3 Shin’ichi Satoh1
1National Institute of Informatics  2Wuhan University  3National Tsing Hua
University

###### Abstract

Existing inpainting methods have achieved promising performance in recovering
defected images of specific scenes. However, filling holes involving multiple
semantic categories remains challenging due to the obscure semantic boundaries
and the mixture of different semantic textures. In this paper, we introduce
coherence priors between the semantics and textures which make it possible to
concentrate on completing separate textures in a semantic-wise manner.
Specifically, we adopt a multi-scale joint optimization framework to first
model the coherence priors and then accordingly interleavingly optimize image
inpainting and semantic segmentation in a coarse-to-fine manner. A Semantic-
Wise Attention Propagation (SWAP) module is devised to refine completed image
textures across scales by exploring non-local semantic coherence, which
effectively mitigates mix-up of textures. We also propose two coherence losses
to constrain the consistency between the semantics and the inpainted image in
terms of the overall structure and detailed textures. Experimental results
demonstrate the superiority of our proposed method for challenging cases with
complex holes.

##  1 Introduction

High-quality image inpainting aims to fill in missing regions with synthetic
content [1, 2, 5]. It requires both semantically meaningful structures and
visually pleasing textures. To this end, deep learning-based methods [22, 36,
39, 41, 43, 44] resort to encoder-decoder based networks to infer the context
of a corrupted image and then refine the texture details in the initial
inference of a missing region by some tools, such as non-local algorithms.
Although current image inpainting methods have made significant progress, it
still poses technical challenges in completing complex holes, particularly
when a missing region involves multiple sub-regions with different semantic
classes. The main reason falls in the failure of modeling the prior
distributions of a mixture of different semantic regions, which usually result
in blurry boundaries and unrealistic textures [15].

A feasible approach is to adopt structural information, such as edges [12,
19], contours [35], and smooth images [24], as guidance to complete missing
structures and textures in two steps. The assumption is that structures offer
semantic clues for inferring an unknown scene, making them suitable for
guiding the filling of textures. However, we notice that the correspondence
between structural information and textures is not apparent, making the filled
textures still highly rely on the local correlation around the missing region.
Figure 1 demonstrates the ambiguity of the mapping from mid-level structures
(e.g., the edges) to the textures, which can significantly degrade the visual
authenticity of the generated textures.

Figure 1: Upper part: Mapping between image textures and edges/semantics (Dot
arrow - extraction of edges/semantics; solid arrow - texture generation).
Notice that two similar edge patches in a green circle could be mapped to
completely different semantic textures, but one semantic will be clearly
mapped to a certain texture category. Lower part: an example showing two
similar edge patches map to two different semantic textures.

Compared with mid-level structures, high-level semantic information offers
more vital semantic clues to the object textures. For example, in Figure 1,
the semantic animal leads to fluffy fur while the semantic tree leads to green
leaf in the image, which cannot be usually distinguished solely from their
mid-level structures. In contrast, object textures have been shown to provide
sufficient information about the semantic classes [7, 11]. Thus we
characterize the relationships between the semantics and textures of objects
as coherence priors and build our inpainting method on the coherence priors to
complete complex holes while ensuring the mutual consistency between the
predicted semantics and textures.

Based on the above motivation, we propose to utilize coherence priors between
semantics and textures to facilitate joint optimization of semantic
segmentation and image inpainting. To this end, our framework extracts a
shared feature to represent the common information of the two tasks, and
characterize the interaction between scales to enable the utilization of
coherence priors to optimize the two tasks jointly. Specifically, two novel
designs are proposed: 1) A Semantic-Wise Attention Propagation (SWAP) module
is used to explicitly capture the semantic relevance between an unknown
(missing) area and the known regions. As a result, when mapping semantics to
image textures, filling in an unknown patch only refers to those known patches
with the same semantic, rather than to the entire image, to avoid irrelevant
texture filling. 2) We devise two loss terms to learn the global and local
coherence relationships, respectively. The image-level structure coherence
loss is used to supervise the structural matching between the inpainted image
and the corresponding segmentation map to generate clear boundaries in the
inpainted image. Besides, the non-local patch-level coherence loss aims to
assess the distribution of patch textures in the semantic domain to encourage
the generated textures to be as similar as the matched known patch of the same
semantics.

Unlike the existing semantics-guided inpainting methods, such as SPG-Net [26]
and SGE-Net [15], which synthesize textures by convolution to involve the
local semantic information, our proposed method predicts the textures and the
semantics simultaneously, and borrows the known texture feature of the same
semantic to fill in a missing region by a semantics-guided non-local means,
which not only ensures realistic textures but also is valuable for semantic
recognition. For a better understanding of our method, we summarized the
comparison to other methods in Table 1. The main contributions of our paper
are three-fold:

  * •

We introduce coherence priors that highlight the mutual consistency between
the semantics and textures in image inpainting and devise two coherence losses
to boost the consistency between semantic information and inpainted image in
the global structure level and local texture level.

  * •

We propose a semantic-wise attention propagation module, which generates
semantically realistic textures by capturing distant relationship and
referring to the texture feature of the same semantic in the feature maps.

  * •

Our approach outperforms existing state-of-the-art image inpainting methods,
including semantics-based approaches [15, 19, 26, 40, 42] on completing
complex hole with multiple semantic regions in terms of the sharpness of
boundaries and the coherence and visual plausibility of textures.

##  2 Related Work

###  2.1 Image inpainting

Deep learning-based inpainting approaches have recently been proposed by
understanding the images, which can generate meaningful content for filling in
the missing region. Context Encoder [22] was first proposed to employ a
generative adversarial network and demonstrated its potential for inpainting
tasks. Based on it, efforts have been made to enhance the inpainting
performance, including introducing specific losses [6, 32, 44], building
recursive architectures for progressive refinement [13, 37], and involving the
structural priors as guidance in a two-stage framework for structural
consistency [14, 19, 24, 38]. However, these methods lack the ability to model
long-term correlations between distant context, leading to blurry textures.

Table 1: Comparison of inpainting methods based on three dimensions: non-local
correlation, structural guidance and framework.

| Non-local | Structures | Framework
---|---|---|---
GatedConv [40] | ✓ | N/A | Two-steps
PEN-Net [26] | ✓ | N/A | One-forward
Edgeconnect [19] | ✗ | Edge | Two-steps
SPG-Net [26] | ✗ | Semantic | Two-steps
SGE-Net [15] | ✗ | Semantic | Joint-learning
Ours | ✓ | Semantic | Joint-learning

To better refine the inpainted image textures, non-local algorithms are
adopted to borrow distant features from a known region, which contains fine
textures, to the missing region. [39] firstly proposed to compute textural
affinity within the same image to ﬁll the corrupted area with more realistic
texture patches from the available area. [42] devised a pyramid of contextual
attention at multiple layers to refine the textures from high-level to low-
level. [17] used a coherent semantic attention layer to ensure semantic
relevance between nearby filled features. [34] extends the single attention
map to bidirectional attention maps and re-normalizes the features to let the
decoder concentrate on filling the holes. Although these methods have
delivered considerable improvements, they failed to address the semantic
ambiguity as they try to measure the texture affinity across all semantics.

---
(a) Overall Framework (b) Semantic-wise Attention Propagation (SWAP) Module
Figure 2: Proposed network architecture. At each scale, both the inpainted
image and segmentation map are output from two task-specific heads to control
the predicted structures of the shared features. SWAP is added between scales
to progressively optimize the texture details of contextual feature.

###  2.2 Semantic-guided image processing

Recent research reveals that the semantic priors of the high-level vision
tasks are useful in the guidance of low-level vision tasks [4, 16, 30]. One
hot research topic is semantic-guided image generation. It covers several
research directions, including image translation from segmentation maps to
realistic images [10, 29] and semantic image synthesis [4, 27, 30]. To avoid
the vanishing of semantic priors in the generation process, [21] proposed a
spatially adaptive normalization layer to propagate the semantic information
to the synthesized images. With the useful semantic priors, it can generate
high-quality images.

The semantic priors have also been applied to promote many conditional low-
level vision tasks and demonstrated its effectiveness in constraining the
plausible solution space in the ill-posed problems. For example, they are
involved in the tasks of super-resolution [31], dehazing [23], denoising [16],
style transfer [18, 28] and image manipulation [9, 20]. Inspired by the
successful assistance from semantic priors in conditional image generation, we
also exploit the semantic guidance in image inpainting.

##  3 Proposed Method

How to achieve high-quality inpainting results on both semantically reasonable
structures and visually pleasing textures? We argue that such results should
not only be able to reconstruct the structures of the semantic objects for the
global structure, but also the textures of them should look realistic with
same semantic in the image for local pixel continuity. To this end, we build a
multi-task learning framework on the coherence priors to explicitly
reconstruct both the structures and textures of semantic objects. Moreover, we
propose a new SWAP module to optimize the textures by semantically binding the
textures between an inpainted region and the known regions based on the
coherence priors. we also devise two losses to guide the learning of coherence
relationships both in the global structure level and in the local patch level,
respectively.

###  3.1 Framework Overview

We build our network on an alternating-optimization architecture to utilize
the coherence priors to mutually assist image inpainting and semantic
segmentation for a corrupted image. Specially, we propose a multi-task
learning framework by sharing features in the decoders for the two tasks (as
shown in Figure 2). The encoder encodes the corrupted image and its mask into
hierarchical contextual features, which are then fed into the decoder to
predict the inpainted images and semantic segmentation maps across scales.
Prior to feeding the encoded feature of the last layer into the decoder, we
initially complete the feature via a Context Inference Module (CIM) based on
the contextual inference method [15, 33].

In the decoder, the contextual feature at each scale is processed by two task-
specific heads to predict the inpainted image and the segmentation map,
respectively. Different from the method proposed in [15] that updates the
contextual features by spatial adaptive normalization to capture the common
properties of the same semantic, we propose a SWAP module to stress the
realistic texture of each semantic patch by referring to the semantic relevant
features from the known regions. In this way, the contextual features are
learnt to represent the global structure and refined with the semantic-aware
texture details.

For brevity, we adopt the following notations. $\varphi^{l}$ and $\phi^{l}$
denote the features from the encoder and from the decoder at scale $l$,
respectively; $\hat{I}^{l}$ and $\hat{S}^{l}$ respectively represent the
inpainted image and the predicted $k$-channel segmentation maps from the
inpainting head $h(\cdot)$ and from the segmentation head $g(\cdot)$ at scale
$l$, where $k$ is the total number of semantic labels; $l$ ranges from 1 (the
coarsest layer) to 5 (the finest layer).

###  3.2 Semantic-wise Attention Propagation (SWAP)

The SWAP module is designed to optimize the contextual features by enhancing
the semantic authenticity of textures based on coherence priors. As shown in
Figure 2(b), SWAP takes four inputs: two of them are the current-scale feature
$\phi^{l}$ and the next-scale skip feature $\varphi^{l+1}$ from the encoder.
The third is the predicted segmentation probability map $\hat{S}^{l}$, that is
used to guide the separation of features. The last is the missing-region mask
$M$. The propagation process can be formulated as follows:

$\centering\phi^{l+1}=swap(\phi^{l},\varphi^{l+1},\hat{S}^{l},M),\@add@centering$
(1)

where $swap(\cdot)$ is the process of refining the contextual features in
SWAP.

---
(a) Non-local Patch Coherence Loss
(b) Structure Coherence Loss
Figure 3: Proposed coherence losses. (a) non-local patch coherence loss
encourages the generated texture to be as similar as any known patch of the
same semantic in the real image; (b) structure coherence loss ensures the
structural consistency between the entire segmentation map and the inpainted
image.

The attention-based approaches in [25, 34, 39] resort to contextual attention
to pick known regions as references to complete a missing region, which,
however, cannot distinguish patches of different semantics and thereby leads
to blurry boundaries and semantic confusion during attention propagation.
Unlike these approaches, SWAP calculates the attention score by matching
semantic-aware features of missing patches and the known patches based on the
coherence priors. Specifically, we first split the contextual features
$f^{l}$, which is generated from $\phi^{l}$ and $\varphi^{l+1}$, into
different semantic parts according to the semantic labels in the predicted
segmentation map $\hat{S}^{l}$ of the $l$-th layer. We subsequently drop the
superscript $l$ for the sake of notational simplicity.

Within each semantic part, each patch’s attention score is evaluated by the
patch affinity between the missing region and a known region using the
normalized inner product followed by a softmax operation:

$Df^{c}_{i,j}=\langle\frac{p^{c}_{i}}{\left|\left|p^{c}_{i}\right|\right|_{2}},\frac{p^{c}_{j}}{\left|\left|p^{c}_{j}\right|\right|_{2}}\rangle,$
(2)

$\omega^{c}_{j,i}=\frac{\exp(Df^{c}_{i,j})}{\sum_{i=1}^{N}\exp(Df^{c}_{i,j})},$
(3)

where $p^{c}_{i}$ is the $i$-th patch extracted from semantic feature $f^{c}$
of class $c$ in the known region, $p^{c}_{j}$ is the $j$-th patch extracted
from $f^{c}$ in the missing region. $Df^{c}_{i,j}$ is the affinity between
them, and $\omega^{c}_{j,i}$ is the attention score representing the
normalized affinities for each patch.

After obtaining the attention score from the known region, the feature of the
$j$-th missing patch is updated by

$p^{c}_{j}=\sum_{i=1}^{N^{c}}\omega^{c}_{j,i}~{}p^{c}_{i}.$ (4)

where $N^{c}$ is the total number of patches of semantic class $c$ in the
known region.

The output feature of SWAP is generated by merging the updated features of all
semantics, followed by four groups of dilated convolutions with different
rates to improve the structural coherence in the final reconstructed features.

###  3.3 Coherence Losses

We devise new coherence losses between the semantics and textures as
supervisions to guide the image inpainting and semantic segmentation to meet
the following requirements: 1) the overall structure between the inpainted
image and segmentation map should match each other; 2) the predicted textures
of a certain semantic class should have the same distribution as that of the
semantic textures in the known region. Under these considerations, we propose
two coherence losses shown in Figure 3, a non-local patch coherence loss and a
structure coherence loss, to respectively evaluate the patch similarity and
the structural matching.

Non-local Patch Coherence Loss. Given the final inpainted image $\hat{I}$ and
the predicted segmentation map $\hat{S}$, we aim to maximize the texture
similarity between $\hat{I}$ and $I$. This is, the generated patches
attributed to a specific class $c$ should be similar to the realistic patches
of the same class in the ground-truth image.

Similar to the attention propagation process, we first split $\hat{I}$ and $I$
into different semantic images and extract patches to build the corresponding
semantic patch sets $P^{c}=\\{p_{j}^{c}\\}$ and $B^{c}=\\{b_{i}^{c}\\}$
according to $\hat{S}$ and $S$. For each patch in $P^{c}$, we randomly select
one patch from $B^{c}$ with the same semantic, and compute the pairwise cosine
distances between them as

$Di^{c}_{i,j}=1-\langle\frac{p^{c}_{i}-\mu_{b}^{c}}{\left|\left|p^{c}_{i}-\mu_{b}^{c}\right|\right|_{2}},\frac{b^{c}_{j}-\mu_{b}^{c}}{\left|\left|b^{c}_{j}-\mu_{b}^{c}\right|\right|_{2}}\rangle,$
(5)

where $\mu_{b}^{c}=\frac{1}{N_{c}}\sum_{j}b_{j}^{c}$, $N_{c}$ is the number of
patches in $B_{c}$.

The non-local patch coherence loss for each semantic class $c$ aims to
maximize the similarity between the patch couples:

$\mathcal{L}_{\text{nlc}}^{c}(P^{c},B^{c})=-\log(\frac{1}{N_{P}^{c}}(\sum_{i}~{}Di^{c}_{i,j})),$
(6)

where $N_{P}^{c}$ is the cardinality of the set of the generated patches with
class label $c$. Our objective is defined as the sum of all the single-class
non-local patch coherence losses over different classes found in $\hat{S}$:

$\mathcal{L}_{\text{nlco}}(I,\hat{I},S,\hat{S})=\sum_{c}~{}\mathcal{L}_{\text{nlco}}^{c}(P^{c},B^{c}),$
(7)

where $c$ assumes all the class labels of mask in $\hat{S}$. Note that if the
label value in $\hat{S}$ is not found in $S$, the coherence loss of the
corresponding semantic patch set is set to 0.

Structure Coherence Loss. Besides the local patch similarity, we adopt a
structure coherence loss to encourage the structural coherence between the
inpainted image and the predicted segmentation map. In this work, we use two
conditional discriminators to judge whether the semantics and the same image’s
textures are coherence. The texture-conditioned discriminator $D_{t}$ is
introduced to detect the predicted segmentation map’ “fakes” given the real
image, while the semantics-conditioned discriminator $D_{s}$ is trained to
detect the inpainted image’s “fakes” given the real segmentation map. The
structure coherence loss can be expressed as:

$\displaystyle\mathcal{L}_{\text{sco}}(I,\hat{I},S,\hat{S})=\mathcal{L}_{c_{s}}(I,\hat{I},S)+\mathcal{L}_{c_{t}}(S,\hat{S},I){}{}{}{}{}{}{}{}{}{}{}{}$
(8) $\displaystyle~{}~{}~{}=\mathbb{E}_{I,S}[\log
D_{s}(I,S)]+\mathbb{E}_{\hat{I},S}[\log(1-D_{s}(\hat{I},S))]$
$\displaystyle~{}~{}~{}+\mathbb{E}_{S,I}[\log
D_{t}(S,I)]+\mathbb{E}_{\hat{S},I}[\log(1-D_{t}(\hat{S},I))].$

###  3.4 Objective Functions

We design appropriate supervised loss terms for learning the inpainting and
segmentation tasks at each scale to obtain multi-scale predictions. We adopt
the reconstruction loss, the adversarial loss and the proposed coherence
losses to promote the fidelity of the inpainted images. The cross entropy loss
is adopted for ensuring the accuracy of the predicted segmentation maps.

Reconstruction Loss. We use the $\mathcal{L}_{1}$ loss to encourage per-pixel
reconstruction accuracy at all scales.

$\mathcal{L}_{1}(I,\hat{I})=\sum_{l}~{}\left|\left|I-up(\hat{I}_{l})\right|\right|.$
(9)

where $l$ is the scale. 5 scales are adopted in this work

Adversarial Loss. We use a multi-scale PatchGAN [30] to classify the global
and local patches of an image at different resolutions. The discriminator at
each scale is identical and only the input is a differently scaled version of
an image.

$\begin{split}\mathcal{L}_{\alpha}(I,\hat{I})=\sum_{k=1,2,3}(\mathbb{E}_{I}[\log
D(p^{k}_{I})]+\mathbb{E}_{\hat{I}}[(1-\log D(p^{k}_{\hat{I}})]),\end{split}$
(10)

where $D(\cdot)$ is the discriminator, $p^{k}_{I}$ and $p^{k}_{\hat{I}}$ are
the patches in the $k$-th scaled versions of $I$ and $\hat{I}$.

Cross-Entropy Loss. This loss is used to penalize the segmentation
performance.

$\mathcal{L}_{\text{xe}}(S,\hat{S})=-\sum_{l}~{}\sum_{p\in
S}S(p)\log(up(\hat{S}^{l})(p)),$ (11)

where $p$ is the pixel index for segmentation map $S$.

Overall Training Loss. The overall training loss function for our network is
defined as the weighted sum of the above mentioned losses.

$\begin{split}\mathcal{L}_{Final}=\mathcal{L}_{1}(I,\hat{I})+\lambda_{\alpha}\mathcal{L}_{\alpha}(I,\hat{I})+\lambda_{\text{xe}}\mathcal{L}_{\text{xe}}(S,\hat{S})\\\
+\lambda_{\text{co}}(\mathcal{L}_{\text{nlco}}(I,\hat{I},S,\hat{S})+\mathcal{L}_{\text{sco}}(I,\hat{I},S,\hat{S})),\end{split}$
(12)

where $\lambda_{\alpha}$, $\lambda_{se}$ and $\lambda_{\text{co}}$ are the
weights for the adversarial loss, cross-entropy loss and coherence loss,
respectively.

|  |  |  |  |  |
---|---|---|---|---|---|---
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
(a) Input | (b) GatedConv | (c) PEN-Net | (d) EdgeConnect | (e) SGE-Net | (f) Our method | (g) Ground-truth
Figure 4: Qualitative comparison of inpainting results on image samples from
Outdoor Scenes and Cityscapes. Due to limited space, the comparison with SPG-
Net is included in the supplementary material

##  4 Results

###  4.1 Experimental Settings

We evaluate our method on the Outdoor Scenes [31] and Cityscapes [45]
datasets. Outdoor Scenes contains 9,900 training images and 300 test images.
Cityscapes contains 5,000 street-view images in total. In order to enrich the
training set of Cityscapes, we use 2,975 images from the training set and
1,525 images from the test set for training, and test on the 500 images from
the validation set. Since the test set lacks human-labeled semantic
annotations, we generate the annotations for training by using the state-of-
the-art segmentation model Deeplab [3]. We resize each training image to
ensure its minimal height/width to be 256 for Outdoor Scenes and 512 for
Cityscapes, and then randomly crop sub-images of size $256\times 256$ as
inputs to our model. The fine annotations of segmentation labels for both
datasets are also provided for training, in which Outdoor Scenes and
Cityscapes are annotated to 8 and 20 categories, respectively. Please note
that the annotations can also be replaced by the extracted segmentation maps
from the state-of-the-art segmentation models.

We compare our method with the following four learning-based inpainting
methods: 1) GatedConv [40]: Contextual attention for leveraging the
surrounding textures and structures. 2) EdgeConnect [19]: Two-stage inpainting
framework with edges as low-level structural information. 3) PEN-Net [42]:
Cross-layer attention transfer and pyramid filling in a multi-scale framework.
4) SPG-Net [26]: Two-step inpainting with a semantic segmentation map as high-
level structural information. 5) SGE-Net [15]: Semantic guidance for
inpainting based on spatially adaptive normalization [21].

Table 2: Objective quality comparison of six methods in terms of PSNR, SSIM,
and FID on Outdoor Scenes and Cityscapes ($\uparrow$: Higher is better;
$\downarrow$: Lower is better). The two best scores are colored in red and
blue, respectively

| Outdoor Scenes | Cityscapes
---|---|---
| centering holes | irregular holes | centering holes | irregular holes
| PSNR$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$
GatedConv | 19.06 | 0.73 | 42.34 | 18.47 | 0.74 | 44.15 | 21.13 | 0.74 | 20.03 | 17.13 | 0.67 | 43.14
PEN-Net | 18.58 | 0.75 | 44.12 | 17.56 | 0.69 | 48.95 | 20.48 | 0.72 | 22.34 | 16.37 | 0.66 | 47.87
EdgeConnect | 19.32 | 0.76 | 41.25 | 19.12 | 0.74 | 42.27 | 21.71 | 0.76 | 19.87 | 17.63 | 0.72 | 39.04
SPG-Net | 18.04 | 0.70 | 45.31 | 17.85 | 0.74 | 50.03 | 20.14 | 0.71 | 23.21 | 16.41 | 0.67 | 43.63
SGE-Net | 20.53 | 0.81 | 40.67 | 19.46 | 0.76 | 39.14 | 23.41 | 0.85 | 18.67 | 17.78 | 0.74 | 41.45
Ours | 21.18 | 0.81 | 38.15 | 20.31 | 0.80 | 36.74 | 23.89 | 0.84 | 18.14 | 17.86 | 0.76 | 38.18

###  4.2 Qualitative Comparisons

Figure 4 shows the qualitative comparisons of our method with all the
baselines. The corrupted area is simulated by sampling a central hole
($128\times 128$ for Outdoor Scenes and $96\times 96$ for Cityscapes) or
randomly placing multiple irregular masks based on [40]. As shown in the
figure, the baselines usually suffer from artifacts and unsatisfactory
boundaries while completing complex holes. GatedConv and PEN-Net adopt
contextual attention to bring in the features of the known region, but they
usually distort the structures when referencing to incorrect semantic textures
from the surrounding, especially in completing the complex holes. EdgeConnect
and SGE-Net are able to recover correct structures owning to the use of
structure priors. However, EdgeConnect may generate mixed edges, making it
difficult to generate correct textures, whereas the textures of SGE-Net are
often over-smoothed without texture refinement. In contrast, our method
generates more realistic textures and better boundaries delineating semantic
regions than all the baselines thanks to the coherence priors between
semantics and textures.

|  |  |  |
---|---|---|---|---
|  |  |  |
Figure 5: Comparisons between different attention modules. From left to right: input, the most matched patches in existing attention module, result from existing attention module, the most matched patches in SWAP module, and result from SWAP module. The arrows in columns 2 and 4 indicate the matched patch from known region to the missing region. |  |  |  |
---|---|---|---|---
|  |  |  |
Figure 6: Visual quality comparisons between EdgeConnect and our method. From
left to right: input, reconstructed edges and images by EdgeConnect,
reconstructed segmentation maps and images by our method. The red arrows
highlight the unrealistic regions generated from Edgeconnect compared with
ours.

###  4.3 Quantitative Comparisons

Table 2 shows the quantitative comparisons on Outdoor Scenes and Cityscapes
datasets based on three quality metrics: Peak Signal-to-Noise Ratio (PSNR),
Structural Similarity Index (SSIM) and Fréchet Inception Distance (FID) [8].
In general, the proposed method achieves significantly better objective scores
than the baselines, especially in PSNR and SSIM.

###  4.4 User Study

We randomly select 100 images from the two datasets (50 from Outdoor Scenes
and 50 from Cityscapes) and invite 20 subjects with image processing expertise
to rank the subjective visual qualities of images inpainted by the five
inpainting methods (GatedConv, PEN-Net, EdgeConnect, SPG-Net, SGE-Net, and our
method). They are not informed of any mask information. For each test image,
its five inpainting results are presented in a random order, and each subject
is asked to rank the five methods from the best to the worst. The result shows
that our method receives 51.8 % favorite votes (i.e., the top-1 in 1,036 out
of 2,000 comparisons), surpassing 19.3 % with SEG-Net, 11.3 % with
EdgeConnect, 7.8 % with GatedConv, 5.6 % with PEN-Net, and 4.2 % with SPG-Net.
Hence, our method outperforms the other methods.

###  4.5 Ablation Studies

####  4.5.1 Effectiveness of SWAP

We verify the effectiveness of SWAP by comparing it with the contextual
attention module from GatedConv [40]. To show the difference, we highlight the
location of the best-match patch for a patch in the missing area. As shown in
Figure 5, since the existing attention module refers to the whole known region
without any semantic guide, it usually matches wrong texture patches to the
missing area, leading to ambiguous textures. In contrast, benefiting from
SWAP, our method matches the patches within the same semantic class, which
effectively improves the fidelity of matched reference textures so as to
generate more realistic textures.

|  |  |  |
---|---|---|---|---
|  |  |  |
Figure 7: Visual quality comparisons on four variants to show the
effectiveness of SWAP and Coherence Loss. From left to right: input, results
of Ours (Base), Ours (Att), Ours (SWAP), and Ours (Full). Table 3: Comparisons
on the performance gains with SWAP and Coherence Loss (Co-Loss) in terms of
three metrics.

| SWAP | Co-Loss | PSNR$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$
---|---|---|---|---|---
Ours (Base) | ✗ | ✗ | 17.43 | 0.65 | 57.31
Ours (Att) | ✗ | ✗ | 19.77 | 0.76 | 43.54
Ours (SWAP) | ✓ | ✗ | 20.58 | 0.79 | 39.46
Ours (Full) | ✓ | ✓ | 21.18 | 0.81 | 38.15

####  4.5.2 Edge vs. Semantic Segmentation

Our work assumes that semantic segmentation labels offer tighter clues to the
textures than edges. To validate it, we compare the reconstructed structures
of EdgeConnect and our model in Figure 6. We find that the edges of different
objects may be mixed up in the edge maps, making EdgeConnect fill in incorrect
texture details for some missing areas. In contrast, the inferred semantic
segmentation labels from our model help well delineate the layout of images,
and the semantics can guide the filling of the textures, result in more photo-
realistic results.

####  4.5.3 Comparison of Segmentation Accuracy

Table 4: Statistical comparison on semantic segmentation accuracy between the
semantic-guided inpainting methods, namely SPG-Net [26], SGE-Net[15] and our
method on Outdoor Scenes and Cityscapes.

Outdoor Scenes | Cityscapes
---|---
Methods | mIoU% | Methods | mIoU%
| SPG-Net
---
0.51 |  | SPG-Net
---
0.39
SGE-Net | 0.68 | SGE-Net | 0.53
Ours | 0.71 | Ours | 0.57

In order to further validate the coherence priors between semantics and the
textures, We also conduct experiments to compare the generated segmentation
maps from SPG-Net, SGE-Net and our method. Due to the alternative optimization
of the inpainting and the segmentation tasks, we can generate high quality
segmentation maps, which in turn improve the inpainting results. Table 4 shows
that our method outperforms the SPG-Net and the SGE-Net in semantic
segmentation. The once-forward process from SPG-Net is hard to generate
reliable semantic labels for large missing area, while the SGE-Net does not
explicitly exploit the interaction between segmentation and inpainting.

####  4.5.4 Performance Gains with SWAP and Coherence Losses

In our method, the two core components, SWAP and coherence loss, are devised
to improve the inpainting performance. In order to investigate their
effectiveness, we conduct an ablation study on four variants: a) Ours (Base),
with only joint optimization of inpainting and segmentation in a multi-scale
framework; b) Ours (Att), adopting the attention module [39] to measure the
texture affinity across all semantics; c) Ours (SWAP), with SWAP; d) Ours
(Full), with both SWAP and Coherence loss.

The visual and numeric comparisons on Outdoor Scenes are shown in Figure 7 and
Table 3, respectively. In general, the inpainting performance increases with
the number of added modules. Specifically, the joint framework helps learn a
more accurate scene layout, and the contextual attention does a good job of
generating detailed content. Our SWAP can identify more relevant textures
thanks to the predicted semantics. Moreover, the coherence losses further
improves the texture details of inpainted regions.

|  |  |  |
---|---|---|---|---
|  |  |  |
Figure 8: Visual quality comparisons on image samples from Places2. From left
to right: input, results of GatedConv, EdgeConnect, and Our method, ground-
truth.

####  4.5.5 Additional Results on Places2

We also conduct performance evaluation on the Places2 dataset [45] without
semantic annotation, which was used in the assessment by both GatedConv and
EdgeConnect. We use our model trained on Outdoor Scenes to complete the images
with similar scenes in Places2. The subjective results in Figure 8 show that
our model is still able to generate proper semantic structures and textures,
owing to the supervision of the coherence loss, which provides better prior
knowledge about the scenes.

##  5 Conclusion

We proposed a novel joint optimization framework of semantic segmentation and
image inpainting to exploit the coherence priors existed between semantics and
textures for solving the complex holes inpainting problem. To address the
irrelevant texture filling, we proposed a semantic-wise attention propagation
module to optimize the predicted textures from the same semantical region and
two coherence losses to constrain the consistency of the semantic and texture
in the same image. Experimental results demonstrate that our method can
effectively generate promising semantic structures and texture details.

## References

  * [1] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B. Goldman.  PatchMatch: a randomized correspondence algorithm for structural image editing.  ACM Trans. Graph., 28(3):24, 2009.
  * [2] Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester.  Image inpainting.  In Proc. ACM SIGGRAPH, pages 417–424, 2000.
  * [3] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proc. ECCV, 2018.
  * [4] Qifeng Chen and Vladlen Koltun.  Photographic image synthesis with cascaded refinement networks.  In Proc. ICCV, pages 1520–1529, 2017.
  * [5] Antonio Criminisi, Patrick Pérez, and Kentaro Toyama.  Region filling and object removal by exemplar-based image inpainting.  IEEE Trans. Image Process., 13(9):1200–1212, 2004.
  * [6] Alexey Dosovitskiy and Thomas Brox.  Generating images with perceptual similarity metrics based on deep networks.  In Proc. NeurIPS, 2016.
  * [7] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  In Proc. ICLR, 2019.
  * [8] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs trained by a two time-scale update rule converge to a local Nash equilibrium.  In Proc. NeurIPS, 2017.
  * [9] Seunghoon Hong, Xinchen Yan, Thomas E. Huang, and Honglak Lee.  Learning hierarchical semantic image manipulation through structured representations.  In Proc. NeurIPS, pages 2713–2723, 2018.
  * [10] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks.  In Proc. CVPR, pages 5967–5976, 2017.
  * [11] Christoph Kamann and Carsten Rother.  Benchmarking the robustness of semantic segmentation models.  In Proc. CVPR, pages 8825–8835. IEEE, 2020.
  * [12] Jingyuan Li, Fengxiang He, Lefei Zhang, Bo Du, and Dacheng Tao.  Progressive reconstruction of visual structure for image inpainting.  In Proc. ICCV, pages 5961–5970. IEEE, 2019.
  * [13] Jingyuan Li, Ning Wang, Lefei Zhang, Bo Du, and Dacheng Tao.  Recurrent feature reasoning for image inpainting.  In Proc. CVPR, pages 7757–7765, 2020.
  * [14] Liang Liao, Ruimin Hu, Jing Xiao, and Zhongyuan Wang.  Edge-aware context encoder for image inpainting.  In Proc. IEEE ICASSP, 2018.
  * [15] Liang Liao, Jing Xiao, Zheng Wang, Chia-Wen Lin, and Shin’ichi Satoh.  Guidance and evaluation: Semantic-aware image inpainting for mixed scenes.  In Proc. ECCV, 2020.
  * [16] Ding Liu, Bihan Wen, Jianbo Jiao, Xianming Liu, Zhangyang Wang, and Thomas S. Huang.  Connecting image denoising and high-level vision tasks via deep learning.  IEEE Trans. Image Process., 29:3695–3706, 2020.
  * [17] Hongyu Liu, Bin Jiang, Yi Xiao, and Chao Yang.  Coherent semantic attention for image inpainting.  arXiv preprint arXiv:1905.12384, 2019.
  * [18] Zhuoqi Ma, Jie Li, Nannan Wang, and Xinbo Gao.  Semantic-related image style transfer with dual-consistency loss.  Neurocomputing, 406:135–149, 2020.
  * [19] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi.  EdgeConnect: Generative image inpainting with adversarial edge learning.  arXiv preprint arXiv:1901.00212, 2019.
  * [20] Evangelos Ntavelis, Andrés Romero, Iason Kastanis, Luc Van Gool, and Radu Timofte.  SESAME: semantic editing of scenes by adding, manipulating or erasing objects.  CoRR, abs/2004.04977, 2020.
  * [21] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.  Semantic image synthesis with spatially-adaptive normalization.  In Proc. CVPR, 2019.
  * [22] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proc. CVPR, 2016.
  * [23] Wenqi Ren, Jingang Zhang, Xiangyu Xu, Lin Ma, Xiaochun Cao, Gaofeng Meng, and Wei Liu.  Deep video dehazing with semantic segmentation.  IEEE Trans. Image Process., 28(4):1895–1908, 2019.
  * [24] Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li.  StructureFlow: Image inpainting via structure-aware appearance flow.  In Proc. ICCV, 2019.
  * [25] Yuhang Song, Chao Yang, Zhe Lin, Xiaofeng Liu, Qin Huang, Hao Li, and C-C Jay Kuo.  Contextual-based image inpainting: Infer, match, and translate.  In Proc. ECCV, 2018.
  * [26] Yuhang Song, Chao Yang, Yeji Shen, Peng Wang, Qin Huang, and C-C Jay Kuo.  SPG-Net: Segmentation prediction and guidance network for image inpainting.  arXiv preprint arXiv:1805.03356, 2018.
  * [27] Hao Tang, Dan Xu, Yan Yan, Philip H. S. Torr, and Nicu Sebe.  Local class-specific and global image-level generative adversarial networks for semantic-guided scene generation.  In Proc. CVPR, pages 7867–7876, 2020.
  * [28] Matteo Tomei, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara.  Art2real: Unfolding the reality of artworks via semantically-aware image-to-image translation.  In Proc. CVPR, pages 5849–5859, 2019.
  * [29] Chaoyue Wang, Chang Xu, Chaohui Wang, and Dacheng Tao.  Perceptual adversarial networks for image-to-image transformation.  IEEE Trans. Image Process., 27(8):4066–4079, 2018.
  * [30] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro.  High-resolution image synthesis and semantic manipulation with conditional gans.  In Proc. CVPR, pages 8798–8807, 2018.
  * [31] Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy.  Recovering realistic texture in image super-resolution by deep spatial feature transform.  In Proc. CVPR, 2018.
  * [32] Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia.  Image inpainting via generative multi-column convolutional neural networks.  In Proc. NeurIPS, pages 329–338, 2018.
  * [33] Jing Xiao, Liang Liao, Qiegen Liu, and Ruimin Hu.  CISI-Net: Explicit latent content inference and imitated style rendering for image inpainting.  In Proc. AAAI, 2019.
  * [34] Chaohao Xie, Shaohui Liu, Chao Li, Ming-Ming Cheng, Wangmeng Zuo, Xiao Liu, Shilei Wen, and Errui Ding.  Image inpainting with learnable bidirectional attention maps.  In Proc. ICCV, pages 8857–8866, 2019.
  * [35] Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, and Jiebo Luo.  Foreground-aware image inpainting.  In Proc. CVPR, 2019.
  * [36] Zhaoyi Yan, Xiaoming Li, Mu Li, Wangmeng Zuo, and Shiguang Shan.  Shift-Net: Image inpainting via deep feature rearrangement.  In Proc. ECCV, 2018.
  * [37] Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li.  High-resolution image inpainting using multi-scale neural patch synthesis.  In Proc. CVPR, 2017.
  * [38] Jie Yang, Zhiquan Qi, and Yong Shi.  Learning to incorporate structure knowledge for image inpainting.  In Proc. AAAI, pages 12605–12612, 2020.
  * [39] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Generative image inpainting with contextual attention.  In Proc. CVPR, 2018.
  * [40] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Free-form image inpainting with gated convolution.  In Proc. ICCV, 2019.
  * [41] Tao Yu, Zongyu Guo, Xin Jin, Shilin Wu, Zhibo Chen, Weiping Li, Zhizheng Zhang, and Sen Liu.  Region normalization for image inpainting.  In Proc. AAAI, pages 12733–12740, 2020.
  * [42] Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo.  Learning pyramid-context encoder network for high-quality image inpainting.  In Proc. CVPR, 2019.
  * [43] Yu Zeng, Zhe Lin, Jimei Yang, Jianming Zhang, Eli Shechtman, and Huchuan Lu.  High-resolution image inpainting with iterative confidence feedback and guided upsampling.  In Proc. ECCV, pages 1–17, 2020.
  * [44] S. Zhang, R. He, Z. Sun, and T. Tan.  Demeshnet: Blind face inpainting for deep meshface verification.  IEEE Trans. Inf. Forensics Security, 13(3):637–647, 2018.
  * [45] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  IEEE Trans. Pattern Anal. Mach. Intell., 40(6):1452–1464, 2017\.


# Bayesian analysis of verbal autopsy data using factor models
with age- and sex-dependent associations between symptoms

Tsuyoshi Kunihama  Department of Economics, Kwansei Gakuin University  Zehang
Richard Li  Department of Statistics, University of California, Santa Cruz
Samuel J. Clark  Department of Sociology, Ohio State University  Tyler H.
McCormick  Department of Statistics and Department of Sociology, University of
Washington

(March, 2024)

###### Abstract

Verbal autopsies (VAs) are extensively used to investigate the population-
level distributions of deaths by cause in low-resource settings without well-
organized vital statistics systems. Computer-based methods are often adopted
to assign causes of death to deceased individuals based on the interview
responses of their family members or caregivers. In this article, we develop a
new Bayesian approach that extracts information about cause-of-death
distributions from VA data considering the age- and sex-related variation in
the associations between symptoms. Its performance is compared with that of
existing approaches using gold-standard data from the Population Health
Metrics Research Consortium. In addition, we compute the relevance of
predictors to causes of death based on information-theoretic measures.

Key words: Bayesian factor models; Causes of death distribution; Multivariate
data; Verbal autopsies; Survey data.

##  1 Introduction

Cause-of-death distributions provide fundamental information about the current
dynamics of a population, which is crucial for designing, monitoring, and
evaluating public health actions. However, this information is inaccessible to
decision-makers in low-resource regions, where many deaths occur outside
hospitals and are not officially registered. A survey-based method called
verbal autopsy (VA) has been developed as a practical solution; it involves
inferring the cause of death without medical certification and estimating
cause-specific mortality rates in populations without complete-coverage civil
registration and vital statistics systems (Maher et al., 2010; Sankoh and
Byass, 2012; Nkengasong et al., 2020). In VA, information about the cause of
death of an individual is collected by interviewing a person close to them
with questions about their demographics, medical history, and signs, and
symptoms in the period leading to their death. The VA interview needs to be
analyzed to assign a probable cause of death, and a common practice is to have
a panel of physicians interpret the interview data. However, this consumes
physicians’ time for patients and is difficult to scale up for massive amounts
of data. In recent years, as an alternative, algorithmic and statistical
methods have been developed for cause-of-death assignment using VA data (Byass
et al., 2003; Serina et al., 2015; McCormick et al., 2016). Computer-based
approaches have the advantages of being scalable for large-scale data and
becoming free once their programming codes are publicly distributed. For
example, popular methods are easily implemented using the openly available R
package openVA (Li et al., 2023). Chandramohan et al. (2021) conducted a
comprehensive review of the VA developments over the last several decades.

Regarding VA interview questions as predictors, statistical methods assign a
cause of death based on the probability of causes given these predictors
$P(\text{cause}\,|\,\text{predictors})$. Standard parametric models, such as
multinomial probit/logit regression, can be applied to this conditional
probability. However, VA surveys usually contain large amounts of missing
values and thus require the challenging task of imputing high-dimensional data
with complex interactions. The common approach is to use an indirect
expression based on Bayes theorem as
$P(\text{cause}\,|\,\text{predictors})\propto
P(\text{predictors}\,|\,\text{cause})P(\text{cause})$, which makes it easy to
accommodate abundant missing values under the missing-at-random assumption.
Popular methods assume the conditional independence of predictors given a
cause (Byass et al., 2012, 2019; Miasnikof et al., 2015; McCormick et al.,
2016), but VA surveys can contain more than a hundred of questionnaire items
and may violate this assumption. Recent studies investigated approaches to
expressing complex dependent structures among predictors (Li et al., 2020;
Kunihama et al., 2020; Moran et al., 2021).

VA surveys collect various information relevant to causes of death, ranging
from the demographic backgrounds to the symptoms, signs, or medical history
(henceforth collectively called symptoms) of deceased individuals. All survey
items are treated equally as predictors in existing methods, although some of
them may play different roles in cause-of-death assignment. Demographic
characteristics, especially age and sex, are key factors in many public health
settings. For example, worldwide differences by sex and age have been
identified in disability-adjusted life years (World Health Organization, 2020)
and the cancer burden (Ferlay et al., 2020). In addition, World Health
Organization (2022) reported inequalities in the distribution of COVID-19
cases and deaths across different demographic profiles such that males are
more likely than females to die and older people are more likely to develop
serious illness. The report also shows age and sex differences in preventive
behaviors during the pandemic, such as social distancing, and mask wearing.
Considering the key roles of age and sex in public health, we explicitly
differentiate them from other predictors and assume they affect the
associations between symptoms for each cause of death. Then, assuming
predictors consist of symptoms, age and sex, we consider the expression

$\displaystyle P($ $\displaystyle\text{cause}\,|\,\text{predictors})\propto
P(\text{predictors}\,|\,\text{cause})P(\text{cause})$ $\displaystyle\propto
P(\text{symptoms, age, sex}\,|\,\text{cause})P(\text{cause})$
$\displaystyle\propto P(\text{symptoms}\,|\,\text{age, sex,
cause})P(\text{age, sex}\,|\,\text{cause})P(\text{cause}).$ (1)

Moran et al. (2021) developed a framework where the distribution of symptoms
differs by covariates, namely, $P(\text{cause}\,|\,\text{predictors})\propto
P(\text{symptoms}\,|\,\text{covariates, cause})P(\text{cause})$. In comparison
to (1), with age and sex set as covariates, the assumption of independence
between (age, sex) and causes of death is implicitly imposed; that is,
$P(\text{age, sex}\,|\,\text{cause})=P(\text{age, sex})$. Given that age and
sex are often associated with causes of death (World Health Organization,
2020), avoiding the independence assumption is appropriate.

###  1.1 PHMRC VA survey data

Using gold-standard VA data, we examine the following points: 1. The
associations between symptoms vary by age and sex. 2. The distributions of age
and sex differ by cause of death. We analyze neonatal VA data collected by the
Population Health Metrics Research Consortium (PHMRC) project (Murray et al.,
2011; PHMRC, 2013). These data consist of medically certified causes of death
and VA survey responses collected by interviewers who were blinded to the
causes of death assigned in hospitals. We use five neonatal causes of death
and 97 predictors for 1,619 babies, and all predictors are binarized based on
the transformation by Murray et al. (2011). Figure 1 shows a histogram of the
causes of death in our study, and the predictors are listed in the
supplementary materials.

As for the associations between symptoms, we compute Cramér’s $V$ (Cramér,
1946) that measures strength of associations between two nominal variables,
producing a value between 0 (no association) and 1 (complete association).
Figure 2 shows Cramér’s V values of all pairs of symptoms for deaths by birth
asphyxia between age groups and between sexes. If age and sex do not affect
the associations between symptoms, then the corresponding points will be close
to the diagonal lines. Figure 3 displays the absolute difference from each of
the points to the diagonal line in Figure 2, indicating that some symptom
pairs show relatively large differences between age groups and between sexes.
Therefore, it is worth pursuing a direction of modeling the age- and sex-
dependent associations between symptoms to estimate the causes-of-death
distribution.

Figure 4 shows the proportions of age groups and sexes for each cause of
death. The bar patterns differ between causes. For example, most of the babies
who died due to birth asphyxia are in the early age group, and the ratio of
females is higher than that of males for pneumonia and preterm delivery.
Hence, age and sex are considered not independent of causes of death.

The remainder of this paper is organized as follows. Section 2 shows the
proposed Bayesian approach to estimating cause-of-death distributions using
factor models with age- and sex-dependent associations between symptoms.
Section 3 describes the Markov chain Monte Carlo (MCMC) algorithm of the
proposed method. In Section 4, we assess the proposed approach in comparison
to existing methods and evaluate the relevance of the predictors to the causes
of death using the PHMRC neonate dataset. Section 5 concludes this article.

Figure 1: Histogram of causes of death in our PHMRC neonate dataset.

Figure 2: Plots of Cramér’s V values for each pair of symptoms for birth
asphyxia between age groups (left) and between sexes (right).

Figure 3: Histograms of absolute difference in Cramér’s V values for each pair
of symptoms for birth asphyxia between age groups (left) and between sexes
(right). Figure 4: Proportions of age (early and late stages) and sex (male
and female) for each cause of death in our PHMRC neonate dataset.

##  2 Proposed Bayesian approach

We develop the Bayesian models in (1) for estimating the population-level
distribution of the causes of death using the VA survey data. The proposed
approach divides the VA predictors into basic demographic information (age and
sex) and the other questionnaire items (symptoms), and the associations
between symptoms differs by age and sex. In addition, we measure the relevance
of each predictor to the causes of death using the proposed models. For the
notations below, let $y_{i}\in\\{1,\ldots,L\\}$ be the cause of the $i$th
person’s death, where $i=1,\ldots,n$, and
$x_{i}=(x_{i1},\ldots,x_{ip})^{\prime}$ be the binary indicators of the
symptoms, where $x_{ij}\in\\{0,1\\}$ for $j=1,\ldots,p$. Moreover,
$\text{age}_{i}$ and $\text{sex}_{i}$ indicate the age and sex, respectively,
of the $i$th baby.

###  2.1 Modeling of $P(\text{symptoms}\,|\,\text{age, sex, cause})$ in (1)

First, we develop models for symptoms in which the dependence structure varies
between the age and sex groups for each cause of death. Let
$z_{i}=(z_{i1},\ldots,z_{ip})^{\prime}$ be a latent continuous variable for
$x_{i}$. The Bayesian factor (BF) model (Kunihama et al., 2020) is expressed
as

$\displaystyle x_{ij}$ $\displaystyle=1(z_{ij}>0),\ \ j=1,\ldots,p,$
$\displaystyle z_{i}$
$\displaystyle=\mu_{y_{i}}+\Lambda_{y_{i}}\eta_{i}+\varepsilon_{i},\ \
\varepsilon_{i}\sim N(0,I_{p}),$

where $1(\cdot)$ is an indicator function; $\mu_{y}$ and $\Lambda_{y}$ denote
the mean and the loading matrix, respectively, for cause $y$; and $\eta_{i}$
is a latent factor, where $\eta_{i}\sim N(0,I_{K})$. By integrating out the
factor $\eta_{i}$, the dependence is induced with
$\text{cov}(z_{i})=\Lambda_{y}\Lambda^{\prime}_{y}+I_{p}$. By borrowing the
idea of covariance regression (Hoff and Niu, 2012; Niu and Hoff, 2019), we
place additional factors in $z_{i}$, leading to age- and sex-dependent
associations between the symptoms.

$\displaystyle x_{ij}$ $\displaystyle=1(z_{ij}>0),\ \ j=1,\ldots,p,$ (2)
$\displaystyle z_{i}$
$\displaystyle=B_{y_{i}}w_{i}+\Lambda_{y_{i}}\eta_{i}+C_{y_{i}}(w_{i}\otimes\gamma_{i})+\varepsilon_{i},\
\ \varepsilon_{i}\sim N(0,I_{p}),$ (3)

where $w_{i}=(1,\text{age}_{i},\text{sex}_{i})^{\prime}$ and
$C_{y}=[C_{y}^{(1)}\ C_{y}^{(2)}\ C_{y}^{(3)}]$, where $C_{y}^{(q)}$ is a
$p\times G$ matrix for $q=1,2,3$; $\gamma_{i}$ and $\eta_{i}$ are latent
factors, where $\eta_{i}\sim N(0,I_{K})$ and $\gamma_{i}\sim N(0,I_{G})$,
respectively. The Kronecker product is expanded, and (3) is expressed as

$\displaystyle z_{i}$
$\displaystyle=B_{y_{i}}w_{i}+\tilde{\Lambda}_{y_{i}}\tilde{\eta}_{i}+\varepsilon_{i},\
\ \varepsilon_{i}\sim N(0,I_{p}),$ (4)

where $\tilde{\eta}_{i}=(\eta^{\prime}_{i},\gamma^{\prime}_{i})^{\prime}$ is a
$K+G$-dimensional latent factor, where $\tilde{\eta}_{i}\sim N(0,I_{K+G})$,
and $\tilde{\Lambda}_{y}=[\Lambda_{y}\ D_{y}]$ is a factor loading, where
$\Lambda_{y}$ is constant against age and sex and $D_{y}$ is an age- and sex-
dependent matrix given by

$\displaystyle
D_{y}=C_{y}^{(1)}+C_{y}^{(2)}\text{age}_{i}+C_{y}^{(3)}\text{sex}_{i}.$

For example, if $\text{age}_{i}\in\\{0,1\\}$ and $\text{sex}_{i}\in\\{0,1\\}$
correspond to {young, old} and {male, female}, respectively, then

$\displaystyle D_{y}=\begin{cases}C_{y}^{(1)}&(\text{young male})\\\
C_{y}^{(1)}+C_{y}^{(2)}&(\text{old male})\\\
C_{y}^{(1)}+C_{y}^{(3)}&(\text{young female})\\\
C_{y}^{(1)}+C_{y}^{(2)}+C_{y}^{(3)}&(\text{old female})\\\ \end{cases}$

By integrating $\tilde{\eta}_{i}$ out in (4), we obtain the following
covariance matrix,

$\displaystyle\text{cov}(z_{i})=\tilde{\Lambda}_{y}\tilde{\Lambda}^{\prime}_{y}+I_{p}=\Lambda_{y}\Lambda^{\prime}_{y}+D_{y}D^{\prime}_{y}+I_{p},$

where $\Lambda_{y}\Lambda^{\prime}_{y}$ denotes rank-$K$ associations shared
between all age and sex groups and $D_{y}D^{\prime}_{y}$ denotes rank-$G$
associations changing by age and sex. To avoid complexity, we assume $K=G$ for
the dimensions of the latent factors and select an appropriate value via
cross-validation in our application. In general, the loading matrix need to be
constrained to identify latent factors, but we use factor modeling to reduce
the dimensions of the covariance matrix, in which case identification is
unnecessary.

###  2.2 Modeling of $P(\text{age, sex}\,|\,\text{cause})$ and
$P(\text{cause})$ in (1)

In terms of modeling of a cause of death, age and sex, it may be standard to
develop a probability of a cause given age and sex,
$P(\text{cause}\,|\,\text{age},\text{sex})$, because a cause is the variable
of main interest in the VA analysis. However, our objective is not in
proposing an exact data generating process of a cause given only age and sex
but in developing a predictive function of a cause given all predictors.
Therefore, we consider modeling $P(\text{age},\text{sex}\,|\,\text{cause})$,
which is a part of $P(\text{cause}\,|\,\text{predictors})$ derived from the
expression (1).

We analyze the dichotomized version of VA data in our application and apply
the Dirichlet distribution to $P(\text{age, sex}\,|\,\text{cause})$
independently for each cause.

$\displaystyle\left\\{P(\text{age}_{i}=a,\text{sex}_{i}=b\,|\,y_{i}),\
a,b\in\\{0,1\\}\right\\}\sim\text{Dirichlet}(a_{1},\ldots,a_{4}),$

where $a_{1},\ldots,a_{4}$ are concentration parameters. Likewise, regarding
the prior distribution of causes $P(\text{cause})$, we assume the Dirichlet
distribution

$\displaystyle\left\\{P(y_{i}=l),\
l\in\\{1,\ldots,L\\}\right\\}\sim\text{Dirichlet}(a_{1},\ldots,a_{L}).$

In cases with little prior information, we assume a uniform prior with the
concentration parameters that are equal to one. For example, we assume
$P(y_{i}=l)\propto 1$ for $l=1,\ldots,L$ if little prior information is
available about the distribution of causes in a new VA study site.

###  2.3 Measurement of predictor relevance

VA surveys consist of many questionnaire items, which are conceptually useful
for predicting a cause of death. However, the degree of association with
causes of death can vary much from question to question, so it is of interest
to investigate the relevance of each predictor to causes of death in VA survey
data.

Using the proposed models in (1), we compute the strength of the associations
between a cause of death and each predictor based on information-theoretic
measures (MacKay, 2003; Cover and Thomas, 2006). Let $H(y)$ denote the entropy
of $y$, which quantifies the expected amount of information about $y$;
$H(y)=-\sum_{y}P(y)\log P(y)$. Mutual information (MI) is a measure of the
mutual dependence between two random variables. Letting
$\tilde{x}=(\text{age},\text{sex},x)$, MI is defined by

$\displaystyle
I(y;\tilde{x}_{j})=\sum_{y}\sum_{\tilde{x}_{j}}P(y,\tilde{x}_{j})\log\frac{P(y,\tilde{x}_{j})}{P(y)P(\tilde{x}_{j})}.$
(5)

This denotes the amount of information obtained about a cause $y$ by observing
the $j$th predictor $\tilde{x}_{j}$. Theoretically, (5) takes a nonnegative
value and equals zero if and only if the two random variables are independent.
To improve our interpretation, we normalize the MI to take a value in the
$[0,1]$ interval, we normalize MI by dividing it by the entropy of $y$; that
is $I(y;\tilde{x}_{j})/H(y)$, which indicates the amount of information about
causes added by the $j$th predictor relative to all the information about
causes.

VA questionnaires often have multiple questions on each topic under symptoms.
For example, the PHMRC survey on neonatal death asks six questions each about
suckling and breathing. If these questions provide highly overlapped
information about causes, then some of them may show high MI values despite
adding little new information about causes given the other questions.
Considering this point, we compute the conditional MI (CMI) along with the MI.
The CMI indicates the amount of information each predictor can add given the
other items.

$\displaystyle
I(y;\tilde{x}_{j}\,|\,\tilde{x}_{-j})=\sum_{y}\sum_{\tilde{x}}P(y,\tilde{x})\log\frac{P(y,\tilde{x}_{j}\,|\,\tilde{x}_{-j})}{P(y\,|\,\tilde{x}_{-j})P(\tilde{x}_{j}\,|\,\tilde{x}_{-j})},$
(6)

where $\tilde{x}_{-j}$ indicates all the predictors except the $j$th one. We
normalize the CMI to $I(y;\tilde{x}_{j}\,|\,\tilde{x}_{-j})/H(y)$, which
assumes a value in the $[0,1]$ interval and indicates the amount of additional
information about causes we can obtain by knowing $x_{j}$ given all the other
predictors relative to the total amount of information related to causes. The
CMI can capture the strength of the conditional association between the
predictors and causes of death, but relying only on CMI may result in a
misleading conclusion. For example, if two predictors are strongly associated
with causes but share most of their information, then both of them will show
small CMI values despite being useful for predicting the causes of death.
Therefore, the relevance of predictors should be evaluated by investigating
both the MI and CMI values. The computation of the MI and CMI using the
proposed models is detailed in Section 3.

##  3 Posterior computation

We develop an Markov-chain Monte Carlo (MCMC) algorithm for the proposed
method in Section 2. Let $m_{i}=(m_{i1},\ldots,m_{ip})^{\prime}$ be a vector
of indicators denoting missing values for the $i$th person such that
$m_{ij}=1$ if $x_{ij}$ is missing and $m_{ij}=0$ if $x_{ij}$ is observed,
where $j=1,\ldots,p$. We define the notation $[m_{i}]$ such that $a_{[m_{i}]}$
and $A_{[m_{i}]}$ (where $a$ and $A$ are a vector and a matrix with $p$ rows,
respectively) denote a subvector and a submatrix consisting of components with
$m_{ij}=0$ for $j=1,\ldots,p$. In addition, $A_{j\cdot}$ shows the $j$th row
of the matrix $A$.

For the prior distributions on $B_{y}$, $\Lambda_{y}$ and $C_{y}$, we use
Cauchy distributions. It has high density around zero and heavy tails, which
reduce the effects of redundant elements while capturing meaningful signals.
Based on the expression of the Cauchy distribution via normal-gamma
distributions, we assume $B_{yjq}\sim N(0,\phi^{-1}_{Bjq})$, $\phi_{Bjq}\sim
Ga(0.5,0.5)$, $\Lambda_{yjk}\sim N(0,\phi^{-1}_{\Lambda j})$, $\phi_{\Lambda
j}\sim Ga(0.5,0.5)$, $C^{(q)}_{yjk}\sim N(0,\phi^{-1}_{Cjq})$, $\phi_{Cjq}\sim
Ga(0.5,0.5)$ where $q=1,2,3$, $j=1,\ldots,p$, $k=1,\ldots,K$, and $Ga(a,b)$
denotes a gamma distribution with a mean $a/b$. The latent variables
$\phi_{Bjq}$, $\phi_{\Lambda j}$ and $\phi_{Cjq}$ are shared between causes
and factors to reduce the number of model parameters.

The proposed MCMC algorithm is below.

  1. 1.

Update
$\beta_{yj}\equiv(B_{yj\cdot},\Lambda_{yj\cdot},C^{(0)}_{yj\cdot},C^{(1)}_{yj\cdot},C^{(2)}_{yj\cdot})^{\prime}$
from $N(\mu_{*},\Sigma_{*})$ for $y=1,\ldots,L$ and $j=1,\ldots,p$ with

$\displaystyle\mu_{*}=\Sigma_{*}\left(\sum_{i:y_{i}=y,m_{ij}=0}a_{i}z_{ij}\right),\
\
\Sigma_{*}=\left(\sum_{i:y_{i}=y,m_{ij}=0}a_{i}a^{\prime}_{i}+\Sigma_{0}^{-1}\right)^{-1}$

where
$a_{i}=(w^{\prime}_{i},\eta^{\prime}_{i},\gamma^{\prime}_{i},\text{age}_{i}\gamma^{\prime}_{i},\text{sex}_{i}\gamma^{\prime}_{i})^{\prime}$
and
$\Sigma_{0}^{-1}=\text{diag}(\phi_{Bj1},\phi_{Bj2},\phi_{Bj3},\phi_{\Lambda
j}1_{K},\phi_{Cj1}1_{K},\phi_{Cj2}1_{K},\phi_{Cj3}1_{K})$, where $1_{K}$
denote a $K$-dimensional vector with all elements one.

  2. 2.

Update $\tilde{\eta}_{i}=(\eta^{\prime}_{i},\gamma^{\prime}_{i})^{\prime}$
from $N(\tilde{\mu},\tilde{\Sigma})$ for $i=1,\ldots,n$ with

$\displaystyle\tilde{\mu}=\tilde{\Sigma}\tilde{\Lambda}^{\prime}_{y_{i}[m_{i}]}(z_{i}-B_{y_{i}}w_{i})_{[m_{i}]},\
\
\tilde{\Sigma}=\left(\tilde{\Lambda}^{\prime}_{y_{i}[m_{i}]}\tilde{\Lambda}_{y_{i}[m_{i}]}+I_{2K}\right)^{-1}.$

  3. 3.

Update $\phi_{Bjq}$ from $Ga(0.5(L+1),0.5(\sum_{y=1}^{L}B^{2}_{yjq}+1))$ for
$q=1,2,3$ and $j=1,\ldots,p$.

  4. 4.

Update $\phi_{\Lambda j}$ from
$Ga(0.5(LK+1),0.5\sum_{y=1}^{L}\sum_{k=1}^{K}\Lambda^{2}_{yjk}+1))$ for
$j=1,\ldots,p$.

  5. 5.

Update $\phi_{Cjq}$ from
$Ga(0.5(LK+1),0.5\sum_{y=1}^{L}\sum_{k=1}^{K}C^{(q)2}_{yjk}+1))$ for $q=1,2,3$
and $j=1,\ldots,p$.

  6. 6.

Update $z_{ij}$ with $m_{ij}=0$ for $i=1,\ldots,n$ and $j=1,\ldots,p$ from

$\displaystyle\begin{cases}N_{+}(B_{y_{i}j\cdot}w_{i}+\tilde{\Lambda}_{y_{i}j\cdot}\tilde{\eta}_{i},1)&\text{if
$x_{ij}=1$},\\\
N_{-}(B_{y_{i}j\cdot}w_{i}+\tilde{\Lambda}_{y_{i}j\cdot}\tilde{\eta}_{i},1)&\text{if
$x_{ij}=0$},\\\ \end{cases}$

where $N_{+}$ and $N_{-}$ denote truncated normals with support $[0,\infty)$
and $(-\infty,0]$.

  7. 7.

Update $\left\\{P(\text{age}_{i}=a,\text{sex}_{i}=b\,|\,y_{i}=y),\
a,b\in\\{0,1\\}\right\\}$ from $\text{Dirichlet}(b_{00},b_{01},b_{10},b_{11})$
where
$b_{ab}=1+\sum_{i=1}^{n}1(y_{i}=y,\ \text{age}_{i}=a,\ \text{sex}_{i}=b)$ for
$y=1,\ldots,L$.

  8. 8.

Impute $\text{age}_{i}$ and $\text{sex}_{i}$ if missing from

$\displaystyle
P(\text{age}_{i}=a\,|\,y_{i},x_{i},\text{sex}_{i},\tilde{\eta}_{i})$
$\displaystyle=\frac{P(x_{i}\,|\,y_{i},\text{age}_{i}=a,\text{sex}_{i},\tilde{\eta}_{i})P(\text{age}_{i}=a,\text{sex}_{i}\,|\,y_{i})}{\sum_{l=0}^{1}P(x_{i}\,|\,y_{i},\text{age}_{i}=l,\text{sex}_{i},\tilde{\eta}_{i})P(\text{age}_{i}=l,\text{sex}_{i}\,|\,y_{i})},\
\ a=0,1,$ $\displaystyle
P(\text{sex}_{i}=b\,|\,y_{i},x_{i},\text{age}_{i},\tilde{\eta}_{i})$
$\displaystyle=\frac{P(x_{i}\,|\,y_{i},\text{age}_{i},\text{sex}_{i}=b,\tilde{\eta}_{i})P(\text{age}_{i},\text{sex}_{i}=b\,|\,y_{i})}{\sum_{l=0}^{1}P(x_{i}\,|\,y_{i},\text{age}_{i},\text{sex}_{i}=l,\tilde{\eta}_{i})P(\text{age}_{i},\text{sex}_{i}=l\,|\,y_{i})},\
\ b=0,1.$

where $P(x_{i}\,|\,y_{i},\text{age}_{i},\text{sex}_{i},\tilde{\eta}_{i})$
corresponds to the probability from (2) and (4).

  9. 9.

For $i\in S$ where $S$ is target data, predict $y_{i}$ with

$\displaystyle
P(y_{i}=y\,|\,x_{i},\text{age}_{i},\text{sex}_{i})=\frac{P(x_{i}\,|\,y_{i}=y,\text{age}_{i},\text{sex}_{i})P(\text{age}_{i},\text{sex}_{i}\,|\,y_{i}=y)P(y_{i}=y)}{\sum_{l=1}^{L}P(x_{i}\,|\,y_{i}=l,\text{age}_{i},\text{sex}_{i})P(\text{age}_{i},\text{sex}_{i}\,|\,y_{i}=l)P(y_{i}=l)},$

with $y=1,\ldots,L$ where
$P(x_{i}\,|\,y_{i},\text{age}_{i},\text{sex}_{i})=\int
P(x_{i}\,|\,y_{i},\text{age}_{i},\text{sex}_{i},\tilde{\eta})f(\tilde{\eta})d\tilde{\eta}$
is evaluated via a Monte Carlo approximation with $\tilde{\eta}_{(r)}\sim
N(0,I_{2K})$ for $r=1,\ldots,R$,

$\displaystyle
P(x_{i}\,|\,y_{i},\text{age}_{i},\text{sex}_{i})\approx\frac{1}{R}\sum_{r=1}^{R}\left\\{\prod_{j:m_{ij}=0}P(x_{ij}\,|\,\tilde{\eta}_{(r)},y_{i},,\text{age}_{i},\text{sex}_{i})\right\\}.$
(7)

Then, compute the population distribution of the causes of death as

$\displaystyle\left(\frac{1}{n_{S}}\sum_{i\in
S}1(y_{i}=1),\ldots,\frac{1}{n_{S}}\sum_{i\in S}1(y_{i}=L)\right)$

where $n_{S}$ is the number of observations in the test data. For the
computation of the measures of importance of the predictors, Step 9 is
replaced as follows:

  10. 9.

Update $\left\\{P(y_{i}=y),\ y\in\\{1,\ldots,L\\}\right\\}$ from

$\displaystyle\text{Dirichlet}\left(1+\frac{1}{n}\sum_{i=1}^{n}1(y_{i}=1),\ldots,1+\frac{1}{n}\sum_{i=1}^{n}1(y_{i}=L)\right).$

  11. 10.

Compute the MI and CMI for each predictor in (5) and (6). Because the
summation in (6) is intractable for a nonsmall $p$, it is approximated using
the Monte Carlo sample $(y_{(r)},\tilde{x}_{(r)})\sim
P(y,\tilde{x})=P(x\,|\,y,\text{age},\text{sex})P(\text{age},\text{sex}\,|\,y)P(y)$,
where $r=1,\ldots,\tilde{R}$.

$\displaystyle
I(y;\tilde{x}_{j}\,|\,\tilde{x}_{-j})\approx\frac{1}{\tilde{R}}\sum_{r=1}^{\tilde{R}}\log\frac{P(y_{(r)},\tilde{x}_{(r)j}\,|\,\tilde{x}_{(r)-j})}{P(y_{(r)}\,|\,\tilde{x}_{(r)-j})P(\tilde{x}_{(r)j}\,|\,\tilde{x}_{(r)-j})}.$
(8)

The programming codes for the proposed method are distributed on GitHub
(https://github.
com/kunihama/BF-AS).

##  4 Results

In this section, we evaluate the performance of the proposed approach using
the PHMRC neonatal VA data. Our primary interest is in estimating the
distribution of deaths by cause in a target population. In VA studies, it is
often extremely challenging to obtain gold-standard training data from a
target site where the VA survey questions are tied to the medically certified
cause of death of each individual. Therefore, the cause-of-death distribution
in a target site is predicted using training data collected from different
geographic locations, so the obtained fractions of death by cause vary widely
between testing and training data. Considering this point, we consider
realistic scenarios where each PHMRC study site is treated as a target site
and all other sites are jointly regarded as training data. Thus, we have six
cases where the targets are Andhra Pradesh, India (case 1), Bohol, Philippines
(case 2), Dar es Salaam, Tanzania (case 3), Mexico City, Mexico (case 4),
Pemba Island, Tanzania (case 5) and Uttar Pradesh, India (case 6).

We assess VA classification methods based on the accuracy of their cause-
specific mortality fractions (CSMFs) (Murray et al., 2011).

$\displaystyle\text{CSMF
accuracy}=1-\frac{\sum_{y=1}^{L}\left|P_{0}(y_{i}=y)-P(y_{i}=y)\right|}{2\\{1-\min_{1\leq
y\leq L}P_{0}(y_{i}=y)\\}}$

where $P_{0}$ and $P$ are the true and estimated distributions, respectively,
of the causes of death in a target site. The CSMF measures the closeness
between the two distributions, taking a value in the [0,1] interval, and a
larger value means better performance. In our study, the true distribution
$P_{0}$, is approximated using the empirical distribution of causes in the
test data, whereas $P$ corresponds to the distribution of causes estimated
using a classification tool.

The proposed method (BF models with age and sex dependence [BF-AS]) is
compared with recent VA classification tools. Popular methods in the VA
literature are available in the R package, such as openVA (Li et al., 2023)
with InterVA (Byass et al., 2012), the tariff method (James et al., 2011;
Serina et al., 2015), the naive Bayes classifier (NBC) (Miasnikof et al.,
2015) and InSilicoVA (McCormick et al., 2016). These approaches are based on
the conditional independence of symptoms given a cause, whereas the BF models
of Kunihama et al. (2020) and the Bayesian hierarchical factor regression
(FARVA) model of Moran et al. (2021) describe the dependence between symptoms
for each cause. FARVA allows the associations between symptoms to vary by
covariate, which we set as age and sex. The settings of the competitor methods
are detailed in the supplementary materials. For the proposed method, we
standardized $\text{age}_{i}$ and $\text{sex}_{i}$ in $w_{i}$ in (3) with the
mean zero and the standard deviation one. And we selected the number of latent
factors $K$ from the set $\\{1,2,3,4,5\\}$ via fivefold cross-validation and
set $R=1,000$ in (7) and $\tilde{R}=10,000$ in (8) as the number of random
samples for the Monte Carlo approximation. We generated 10,000 MCMC samples
after the initial 1,000 burn-in iterations, and every 20th sample was saved.
We observed that the sample paths were stable, and the sample autocorrelations
dropped smoothly. The trace plot and the autocorrelation plot of the CSMF in
the target site are in the supplementary materials.

First, as an illustrative example, Figure 5 shows the CSMF estimation results
for case 2. All methods except InterVA work reasonably well, capturing the
true mortality fractions closely; InterVA underestimates birth asphyxia while
overestimating pneumonia. The figures for the other cases are in the
supplementary materials. Table 1 reports the CSMF accuracy of all methods,
with the bold numbers indicating the top three results in each case. No single
method dominates its competitors all the time; the approach with the highest
accuracy differs from case to case. However, as the bold numbers show, the
proposed method consistently performs well in all cases. In addition, Figure 6
shows the average CSMF accuracy of each method over the six cases. InterVA
lags by a large margin, and although the BF models produce the highest
accuracy among the competing methods, BF-AS outperforms them.

Regarding uncertainty, the coverage rate of the true fractions by the proposed
method in a 95% interval is 0.6. Although it is larger than those of the BF
models (0.43), FARVA (0.4), and InSilicoVA (0.3) (Tariff, InterVA, and NBC
produce only point estimates), certain gaps need to be addressed in future
work, as discussed in Section 5.

Figure 5: CSMF estimation results for case 2. The red asterisks show the true values, and each circle with an interval corresponds to the mean and the 95% interval, respectively, of each statistical method. BF-AS: BF models with age and sex dependence (proposed method), BF: Bayesian factor models, FARVA: Bayesian hierarchical factor regression model, NBC: naive Bayes classifier. Tariff, InterVA, and NBC produce point estimates only. Method | Case 1 | Case 2 | Case 3 | Case 4 | Case 5 | Case 6
---|---|---|---|---|---|---
BF-AS | 0.757 | 0.932 | 0.769 | 0.893 | 0.845 | 0.822
BF | 0.741 | 0.942 | 0.742 | 0.788 | 0.792 | 0.801
FARVA | 0.734 | 0.891 | 0.768 | 0.806 | 0.740 | 0.746
InsilicoVA | 0.671 | 0.810 | 0.706 | 0.707 | 0.628 | 0.884
Tariff | 0.777 | 0.880 | 0.811 | 0.566 | 0.800 | 0.740
InterVA | 0.471 | 0.683 | 0.463 | 0.612 | 0.659 | 0.539
NBC | 0.759 | 0.840 | 0.716 | 0.837 | 0.685 | 0.877
Table 1: CSMF accuracy. The bold numbers indicate the top three results in
each case. BF-AS: BF models with age and sex dependence (proposed method), BF:
Bayesian factor models, FARVA: Bayesian hierarchical factor regression model,
NBC: naive Bayes classifier. The posterior means are used as the estimate of
the CSMF for BF-AS, BF, FARVA and InsilicoVA. Figure 6: Average CSMF accuracy
of each method in six cases.

About the relevance of the predictors, Figure 7 shows the estimated
standardized MI for the top 15 predictors. The first two predictors, weight
and size at the time of delivery, have more than 10% of the information about
the causes of death. Early/late delivery, physical abnormality at birth and
several items related to crying and suckling also have high MI values. As for
the demographic variables, age is on the list, but sex is out.

As discussed in Section 2.3, the MI of a predictor does not reflect the
effects of other predictors. Figure 8 reports the estimated CMI for the top 15
predictors. Many of the predictors with large MI values appear again on the
CMI list of CMI, such as weight, size, early/late delivery, physical
abnormality, birth order and symptoms related to crying and suckling. The
estimated values of CMI values for these predictors are smaller than the
estimated MI values, indicating that a certain amount of their information
about the causes of death is shared with the other predictors. In addition, we
observe a remarkable difference –the position of sex– between Figures 7 and 8.
Sex shows a small posterior mean of the MI (0.15%), ranking 96th out of the 97
predictors (not shown in this figure), but leads the CMI list by a wide
margin. It is also the only predictor whose CMI value is larger than its MI
value. Thus, although one does not obtain so much information about causes
only by knowing the sex of a baby, it can indirectly deliver unique
information about the cause of death through the other predictors.

Finally, we investigate the variation in the association between a cause and
each symptom across different age and sex groups. The mutual information,
conditional on age and sex, is expressed as the expected value of the
Kullback–Leibler (KL) divergence between the joint distribution and the
product of marginal distributions,

$\displaystyle I(\text{cause};\text{symptom}\,|\,\text{age},\text{sex})$
$\displaystyle=E_{\text{age},\text{sex}}\left[D_{\text{KL}}\\{P(\text{cause},\text{symptom}\,|\,\text{age},\text{sex})\,||\,P(\text{cause}\,|\,\text{age},\text{sex})P(\text{symptom}\,|\,\text{age},\text{sex})\\}\right],$
$\displaystyle=\sum_{\text{age}}\sum_{\text{sex}}P(\text{cause},\text{symptom},\text{age},\text{sex})\log\frac{P(\text{cause},\text{symptom}\,|\,\text{age},\text{sex})}{P(\text{cause}\,|\,\text{age},\text{sex})P(\text{symptom}\,|\,\text{age},\text{sex})}.$

To examine the differences between age and sex groups, Figure 9 presents a
comparison of the KL divergence for symptoms that are highly relevant to
causes of death, revealing diverse patterns. For instance, the KL divergence
indicates larger values for older age groups (pregnant end early/late, stop
sucking and first cry within 5 minutes) and for females (small or very small).
Conversely, there is less variation in the KL divergence by age and sex
concerning the physically abnormal symptom. Additionally, females in the early
age stage are associated with relatively smaller KL divergence values (birth
order and first cry more than 30 minutes). These findings suggest that the
degree of association between a cause of death and each symptom varies
significantly by age and sex.

Figure 7: Posterior mean of standardized MI for top 15 predictors. The error
bars correspond to standard deviations. Figure 8: Posterior mean of
standardized CMI for top 15 predictors. The error bars correspond to standard
deviations. Figure 9: Posterior means of the KL divergence for cause of death
and symptom associations across age and sex groups.

##  5 Conclusion

In this article, we develop an approach to estimating cause-of-death
distributions using VA survey data. Considering the differences between basic
demographic information (age and sex) and the other types of predictors, we
propose BF models with age- and sex-dependent associations between symptoms
for each cause of death. In addition, the relevance of predictors to causes of
death is computed based on information-theoretic measures. In our analysis of
PHMRC neonate data, BF-AS performs well in estimating the cause-of-death
distributions in the target sites compared with existing methods in the VA
literature. However, as the proposed approach has low coverage rates of the
true fractions of causes in the 95% interval, it needs further development for
a better understanding of the relations between causes of death and VA survey
items.

One research direction is to allow cause-predictor relations to differ
flexibly by location. In the proposed method, the distribution of predictors
given a cause is independent of the VA survey site, but a recent work stated
that this conditional probability can be affected by the domain (Wu et al.,
2021; Li et al., 2021). One approach may be to treat the domain information
similarly to age and sex in the proposed model by adding site-related dummy
variables as covariates. However, this is challenging because target sites
often do not have training data, so domain effects on symptom distributions
can not be estimated directly. Domains may have similar patterns of the
relations between causes and VA questions, so borrowing information from other
sites for incorporation into the factor models is key.

Another study direction is to investigate whether the estimation of cause-of-
death distributions will be improved by allowing symptom associations to vary
by other demographic variables along with age and sex. For example,
educational backgrounds, income levels and marital status may affect the
symptom distribution for each cause. Adding information will increase the
flexibility of factor models, but sparsity may have to be incorporated into
the models to avoid overfitting, such as by using variable selection methods
for covariance regression (Niu and Hoff, 2019).

## Acknowledgment

This work was supported by JSPS KAKENHI Grant Numbers JP21K13274, JP20KK0292,
JP19H00588. The results are generated mainly using Ox (Doornik, 2007). The
programing codes for the proposed method and the Bayesian factor model by
Kunihama et al. (2020) are distributed on the GitHub website
(https://github.com/kunihama/BF-AS).

## References

  * Byass et al. (2012) Byass, P., D. Chandramohan, S. J. Clark, L. D’Ambruoso, E. Fottrell, W. J. Graham, A. J. Herbst, A. Hodgson, S. Hounton, K. Kahn, et al. (2012).  Strengthening standardised interpretation of verbal autopsy data: the new InterVA-4 tool.  Global Health Action 5, 19281.
  * Byass et al. (2003) Byass, P., D. L. Huong, and H. V. Minh (2003).  A probabilistic approach to interpreting verbal autopsies: methodology and preliminary validation i in Vietnam.  Scandinavian Journal of Public Health. Supplement 62, 32–37.
  * Byass et al. (2019) Byass, P., L. Hussain-Alkhateeb, L. D’Ambruoso, S. Clark, J. Davies, E. Fottrell, J. Bird, C. Kabudula, S. Tollman, K. Kahn, et al. (2019).  An integrated approach to processing WHO-2016 verbal autopsy data: the interVA-5 model.  BMC Medicine 17, 102.
  * Chandramohan et al. (2021) Chandramohan, D., E. Fottrell, J. Leitao, E. Nichols, S. J. Clark, C. Alsokhn, D. C. Munoz, C. AbouZahr, A. D. Pasquale, R. Mswia, et al. (2021).  Estimating causes of death where there is no medical certification: evolution and state of the art of verbal autopsy.  Global Health Action 14, 1982486.
  * Cover and Thomas (2006) Cover, T. M. and J. A. Thomas (2006).  Elements of information theory.  Wiley-Interscience.
  * Cramér (1946) Cramér, H. (1946).  Mathematical Methods of Statistics.  Princeton landmarks in mathematics and physics. Princeton University Press.
  * Doornik (2007) Doornik, J. A. (2007).  Object-Oriented Matrix Programming Using Ox, 3rd ed. London: Timberlake Consultants Press and Oxford.
  * Ferlay et al. (2020) Ferlay, J., M. Ervik, F. Lam, M. Colombet, L. Mery, M. Pineros, A. Znaor, I. Soerjomataram, and F. Bray (2020).  Global cancer observatory: Cancer today. lyon, france: International agency for research on cancer.  https://gco.iarc.fr/today.
  * Hoff and Niu (2012) Hoff, P. D. and X. Niu (2012).  A covariance regression model.  Statistica Sinica 22, 729–753.
  * James et al. (2011) James, S. L., A. D. Flaxman, and C. J. Murray (2011).  Performance of the tariff method: validation of a simple additive algorithm for analysis of verbal autopsies.  Population Health Metrics 9, 31.
  * Kunihama et al. (2020) Kunihama, T., Z. R. Li, S. J. Clark, and T. H. McCormick (2020).  Bayesian factor models for probabilistic cause of death assessment with verbal autopsies.  The Annals of Applied Statistics 14, 241–256.
  * Li et al. (2020) Li, Z. R., T. H. McComick, and S. J. Clark (2020).  Using Bayesian latent Gaussian graphical models to infer symptom associations in verbal autopsies.  Bayesian Analysis 15, 781–807.
  * Li et al. (2023) Li, Z. R., J. Thomas, E. Choi, T. H. McCormick, and S. J. Clark (2023).  The r journal: The openva toolkit for verbal autopsies.  The R Journal, 1.  https://journal.r-project.org/articles/RJ-2023-020/.
  * Li et al. (2021) Li, Z. R., Z. Wu, I. Chen, and S. Clark (2021).  Bayesian nested latent class models for cause-of-death assignment using verbal autopsies across multiple domains.  Working paper. https://arxiv.org/abs/2112.12186.
  * MacKay (2003) MacKay, D. J. C. (2003).  Information theory, inference, and learning algorithms.  Copyright Cambridge University Press.
  * Maher et al. (2010) Maher, D., S. Biraro, V. Hosegood, R. Isingo, T. Lutalo, P. Mushati, B. Ngwira, M. Nyirenda, J. Todd, and B. Zaba (2010).  Translating global health research aims into action: the example of the alpha network.  Tropical Medicine & International Health 15, 321–328.
  * McCormick et al. (2016) McCormick, T. H., Z. R. Li, C. Calvert, A. C. Crampin, K. Kahn, and S. J. Clark (2016).  Probabilistic cause-of-death assignment using verbal autopsies.  Journal of the American Statistical Association 111, 1036–1049.
  * Miasnikof et al. (2015) Miasnikof, P., V. Giannakeas, M. Gomes, L. Aleksandrowicz, A. Y. Shestopaloff, D. Alam, S. Tollman, A. Samarikhalaj, and P. Jha (2015).  Naive Bayes classifiers for verbal autopsies: comparison to physician-based classification for 21,000 child and adult deaths.  BMC Medicine 13, 286.
  * Moran et al. (2021) Moran, K. R., E. L. Turner, D. Dunson, and A. H. Herring (2021).  Bayesian hierarchical factor regression models to infer cause of death from verbal autopsy data.  Journal of the Royal Statistical Society Series C: Applied Statistics 70, 532–557.
  * Murray et al. (2011) Murray, C. J., A. D. Lopez, R. Black, R. Ahuja, S. M. Ali, A. Baqui, L. Dandona, E. Dantzer, V. Das, U. Dhingra, et al. (2011).  Population Health Metrics Research Consortium gold standard verbal autopsy validation study: design, implementation, and development of analysis datasets.  Population Health Metrics 9, 27.
  * Murray et al. (2011) Murray, C. J., R. Lozano, A. D. Flaxman, A. Vahdatpour, and A. D. Lopez (2011).  Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studies.  Population Health Metrics 9, 28.
  * Niu and Hoff (2019) Niu, X. and P. Hoff (2019).  Joint mean and covariance modeling of multiple health outcome measures.  Annals of Applied Statistics 13, 321–339.
  * Nkengasong et al. (2020) Nkengasong, J., E. Gudo, I. Macicame, X. Maunze, A. Amouzou, K. Banke, S. Dowell, and I. Jani (2020).  Improving birth and death data for African decision making.  The Lancet Global Health 8, e35–e36.
  * PHMRC (2013) PHMRC (2013).  Population health metrics research consortium gold standard verbal autopsy data 2005–2011.  https://ghdx.healthdata.org/record/ihme-data/population-health-metrics-research-consortium-gold-standard-verbal-autopsy-data-2005-2011.
  * Sankoh and Byass (2012) Sankoh, O. and P. Byass (2012).  The INDEPTH network: filling vital gaps in global epidemiology.  International Journal of Epidemiology 41, 579–588.
  * Serina et al. (2015) Serina, P., I. Riley, A. Stewart, S. L. James, A. D. Flaxman, R. Lozano, B. Hernandez, M. D. Mooney, R. Luning, R. Black, et al. (2015).  Improving performance of the tariff method for assigning causes of death to verbal autopsies.  BMC Medicine 13, 291.
  * World Health Organization (2020) World Health Organization (2020).  Global health estimates 2020: Deaths by cause, age, sex, by country and by region, 2000-2019. geneva.  https://www.who.int/data/global-health-estimates.
  * World Health Organization (2022) World Health Organization (2022).  World health statistics 2022: monitoring health for the sdgs, sustainable development goals.  https://www.who.int/publications/i/item/9789240051157.
  * Wu et al. (2021) Wu, Z., Z. R. Li, I. Chen, and M. Li (2021).  Tree-informed Bayesian multi-source domain adaptation: cross-population probabilistic cause-of-death assignment using verbal autopsy.  Working paper. https://arxiv.org/abs/2112.10978.