Pre-print to appear in IEEE Access

Global and Local Features through Gaussian Mixture Models on Image Semantic Segmentation
Darwin Saire,1 and Adín Ramírez Rivera,1,2
Senior Member, IEEE
[1]Institute of Computing, University of Campinas, Brazil (e-mail: )
[2]Department of Informatics, University of Oslo, Norway (e-mail: )
This work was financed in part by the São Paulo Research Foundation (FAPESP) under grants No. 2017/16597-7, 2019/07257-3, and 2019/18678-0, and in part by the Brazilian National Council for Scientific and Technological Development (CNPq) under grant No. 307425/2017-7.  The Laboratoire Lorrain de Recherche en Informatique et ses Applications (LORIA) institute provided, in part, infrastructure for this work.
Code available at <https://gitlab.com/mipl/phgmm>.

Saire and Ramírez Rivera: Global and Local Features through Gaussian Mixture Models on Image Semantic Segmentation
Saire and Ramírez Rivera: Global and Local Features through Gaussian Mixture Models on Image Semantic Segmentation

Corresponding author: Adín Ramírez Rivera (e-mail: ).

Explainable Latent Spaces, Context-aware features, Gaussian Mixture Models, Semantic Segmentation

The semantic segmentation task aims at dense classification at the pixel-wise level.
Deep models exhibited progress in tackling this task.
However, one remaining problem with these approaches is the loss of spatial precision, often produced at the segmented objects' boundaries.
Our proposed model addresses this problem by providing an internal structure for the feature representations while extracting a global representation that supports the former.
To fit the internal structure, during training, we predict a Gaussian Mixture Model from the data, which, merged with the skip connections and the decoding stage, helps avoid wrong inductive biases.
Furthermore, our results show that we can improve semantic segmentation by providing both learning representations (global and local) with a clustering behavior and combining them.
Finally, we present results demonstrating our advances in Cityscapes and Synthia datasets.

§ INTRODUCTION

Humans naturally (innate way) receive and understand a large amount of information (multiple samples) <cit.>.
Trying to replicate this process through computers with visual perception, , endowing the machines the ability to see, is called computer vision <cit.>, which is difficult due largely to variance.
A vision system is required to infer objects across huge variations in pose, appearance, viewpoint, illumination, and occlusion throughout the image.

Scene understanding is one of the most challenging tasks in computer vision.
It plays an essential role in many novels and future applications, ,  autonomous driving or object detection.
While humans can easily capture a scene at a glance (, perceive general and specific details), it is difficult for a machine.
Thus, semantic scene segmentation is one crucial step toward scene understanding, and it is one of the key challenges in computer vision.
In a broader view, semantic segmentation (SS) <cit.> is a high-level task that makes complete scene understanding possible.
The goal of scene understanding is to make machines see like humans, , to have a complete understanding of visual scenes.
SS aims to annotate each pixel of an image with a class label describing what this pixel represents.
SS task is also called dense prediction since each pixel in the image is classified.
How to extract and interpret the different levels of information details (, global and local context information) is still an open problem and even more for a dense pixel-wise classification.
Note that in solving the SS task, this one directly influences different applications, for example, self-drive vehicles <cit.>, segmentation on X-ray <cit.>, detect crown on dental X-ray <cit.>, brain tumor segmentation <cit.>, remote sensing <cit.>, among others.

(a) FastNet     (b) DUNet     (c) HRNet

(d) OCNet     (e) CCNet     (f) HRNet+OCR

The loss of spatial precision, often produced at the objects' boundary (red color), remains a problem in semantic segmentation.

Over the last decade, computer vision has benefited from significant developments in deep learning, particularly by convolutional neural networks (CNNs) that perform remarkably on complex vision tasks such as object classification and SS.
With the help of ever-growing amounts of labeled data, the deep networks can better generalize and model their context information than the traditional methods <cit.>.
Long  <cit.> showed that CNNs could be adapted for the segmentation task by adding upsampling layers to perform pixel-wise predictions.
However, with the deep learning approach, new problems have been observed (Fig. <ref>) that derive from using CNNs for the specific task of semantic segmentation.
These problems are <cit.>:
(i) low-resolution obtained in the output of the CNNs, and
(ii) loss of spatial precision of objects within the image.

These problems are not produced by a specific operation, , downsampling, but rather by a set of operations.
For instance, the absence of reconstruction and refinement methods, the excessive down-sampling, the gradient vanishing, or the lack of a better extractor of feature maps <cit.>.
Nowadays, different models <cit.> have addressed these problems, especially in the low resolution on the output maps.
However, the loss of spatial precision problem persists and is commonly visualized at the edge of segmented objects, Fig. <ref>.
In this work, we found (see Tables <ref> and <ref>) that to preserve spatial precision, we need mechanisms that provide general and specific details to segment, , methods for better extraction, modeling, and treatment of global and local context information.

Thus, some models <cit.> use heatmap refinement with post-processing steps—, conditional random field <cit.>—as well as models adjusting the maps from bounding boxes <cit.>.
Although post-processing helps to have a broader-view field of objects (, global features), the models lack effective global feature extractors embedded in the architecture.
The encoder-decoder models <cit.> carry out more detailed work to adjust the heat maps by adding operations in the reconstruction, , upsampling and deconvolution.
These models perform local and global feature extraction at some level.
However, they require a more robust combination of features than skip connections and concatenations.
Some models <cit.> work with samples at different scales (multi-scale models) to obtain a full context of the images, , global and local context information from upper and lower scales, respectively.
However, multiple-size inputs make the merging process more complicated than it needs to be.
In contrast, instead of resizing the inputs, models <cit.> increase the field of vision of the kernels (, receptive field) through multiple dilated convolutions <cit.>.
With the information from different scopes, they have the ability to tackle objects of varying sizes.
However, the sampling ranges distribution (, global features and context priors) cannot ensure that the information can be contained in particular ranges.
That is, they have drawbacks with objects of bigger-size than the convolution kernel pyramid.
Finally, models <cit.> are emerging, focusing on smart feature extraction through attention.
However, performing attention in limited regions of the image shows similar behavior to the previous models, , larger-size objects escape the focus of attention.

We address the loss of spatial precision problem for SS tasks by including specific structures to extract local and global features (, holistic and specific features extraction blocks).
Thus, we propose a Probabilistic Hourglass Gaussian Mixture Model (PHGMM), which combines Gaussian Mixture Model (GMM) <cit.> by assuming that the data points are generated from a Mixture-of-Gaussians and a Variational AutoEncoder (VAE) <cit.>.
Unlike the previous models, PHGMM learns two distributions by providing structures to the internal representations (, latent spaces) of the SS task given an input.
The first distribution extracts the global features (, global context) useful for coarse SS.
In contrast, the second distribution focuses on obtaining the specific features (, local context) for a fine SS.
Furthermore, in contrast to the previous models, we use a single sample for the multi-scale features extraction; our distributions (, our learning representation) extract information independently of the objects-size to segment, combining the information in one stage robust reconstruction.
Thus, we address the loss of spatial precision problem by combining the local and global features of both latent spaces and the spatial features of the encoder through skip connections and upsampling in the decoder.
Our main contributions are

* an end-to-end trainable deep model, PHGMM, that combines VAE and GMM for the features' internal representations;

* two latent spaces modeled as a GMM to extract global and local context information that jointly improve the detection of the semantic classes (, holistic and specific features extraction);

* pipeline for image reconstruction by merging the different scales of context information, recovering and improving the geometric information through the decoding stage; and

* a demonstration of the advantages of using coarse and fine modeling for the image information.

node distance=1.cm,
pic shift/.store in=,
pic shift=(0,0),
shift to anchor/.code=
pics/named scope code/.style 2 args=code=@fig@mustbenamed
[pic shift, local bounding box/.expanded=@fig@name, #1]#2
layer/.style=draw, text width=4pt, minimum width=4pt, rounded corners, minimum height=#1,
pics/encoder/.style = named scope code=node distance=4pt
[layer=.8cm, fill=cs-encoder, fill opacity=0.4] (-l2) ;
[layer=1.2cm, left=of -l2, fill=cs-encoder, fill opacity=0.4] (-l1) ;
[layer=.4cm, right=of -l2, fill=cs-encoder, fill opacity=0.4] (-l3) ;
[layer=1.6cm, left=of -l1, fill=cs-encoder, fill opacity=0.4] (-l0) ;
pics/encoderpost/.style = named scope code=node distance=4pt
[layer=.8cm] (-l2) ;
[layer=1.2cm, left=of -l2] (-l1) ;
[layer=.4cm, right=of -l2] (-l3) ;
[layer=1.6cm, left=of -l1] (-l0) ;
pics/decoder/.style = named scope code=node distance=4pt
[layer=.8cm, fill=cs-decoder, fill opacity=0.4] (-l1) ;
[layer=.4cm, left=of -l1, fill=cs-decoder, fill opacity=0.4] (-l0) ;
[layer=1.2cm, right=of -l1, fill=cs-decoder, fill opacity=0.4] (-l2) ;
[layer=1.6cm, right=of -l2, fill=cs-decoder, fill opacity=0.4] (-l3) ;
[layer=1.6cm, right=12pt of -l3] (-l4) ;
[rounded corners] (-l4.north west) – (-l4.south west) [sharp corners]– (-l4.south) – (-l4.north) [fill=black!50, rounded corners]– cycle;
pics/gmm/.style=named scope code=
[yscale=2.0,yshift=-3,xscale=0.1,domain=0:10] plot[id=gauss1]
[yscale=2.0,yshift=-3,xscale=0.1,domain=5:15] plot[id=gauss2]
[yscale=2.0,yshift=-3,xscale=0.1,domain=10:20] plot[id=gauss3]
pics/gmm posterior/.style=named scope code=
[yscale=2.0,yshift=-3,xscale=0.1,domain=0:8] plot[id=gauss4]
[yscale=2.0,yshift=-3,xscale=0.1,domain=5:11] plot[id=gauss5]
[yscale=2.0,yshift=-3,xscale=0.1,domain=7:20] plot[id=gauss6]
pics/gauss/.style=named scope code=
[yscale=2.0,yshift=-3,xscale=0.1,domain=0:10] plot[id=gauss]
node distance=5pt,
rounded corners,
shorten >= 5pt,
shorten <= 5pt,
rounded corners,
shorten >= 5pt,
shorten <= 5pt,

(img) ;

pic[right=2.5cm of img] (enc) encoder;
pic[below right=1.25cm and -25pt of enc] (enc2) encoderpost;

[right=of enc] (gmm0)

[fit=(gmm0), inner sep=-2mm, draw, rounded corners, fill=cs-localz, fill opacity=0.2] (gmm) ;

[right=of enc2] (gmm3)

[fit=(gmm3), inner sep=-2mm, draw, rounded corners] (gmm2) ;

[above=of gmm] (gauss0)

[fit=(gauss0), inner sep=-2mm, draw, rounded corners, fill=cs-globalg, fill opacity=0.2] (gauss) ;

[right=1.5cm of gmm] (dec) decoder;

[right=of dec] (pred) ;
[right=.25cm of pred] (gt) ;

[label, above=of enc] (enc-lbl) Encoder;
[label, below=of img] (img-lbl) Input;

[label, anchor=west] at (enc-lbl -| dec-l0) Decoder ($f$);
[label, below=of pred] (pred-lbl) Prediction;
[label, below=of gt] (gt-lbl) Ground Truth;

[label, below=of enc2] (enc2-lbl) PostNet;
[label] at (enc2-lbl -| gmm2) Posterior;

[label, below=0.1cm of gauss] (gauss-lbl) Global space ($g$);

[edge] (enc-l0.north) -| ($(enc-l3.east)+(5pt,-3pt)$);
[edge] (enc-l1.north) -| ($(enc-l3.east)+(5pt,-3pt)$);
[edge] (enc-l2.north) -| ($(enc-l3.east)+(5pt,-3pt)$);

[edge] (enc2-l0.north) -| ($(enc2-l3.east)+(5pt,-3pt)$);
[edge] (enc2-l1.north) -| ($(enc2-l3.east)+(5pt,-3pt)$);
[edge] (enc2-l2.north) -| ($(enc2-l3.east)+(5pt,-3pt)$);

[edge] (img) – (enc);
[edge] (img-lbl) |- ($(enc2.west)+(0,5pt)$);

[edge] (enc) – (gmm);
[edge] (enc2) – (gmm2);

[edge] (gmm) – (dec);
[edge] (dec) – (pred);

[edge] (enc.east) – ++(.5cm, .0cm) |- (gauss);
[edge] (gauss) -| ($(dec-l4.north)$);

(p-enc2) at ($(enc2-lbl.south)+(0,-5pt)$);
[edge] (gt-lbl) |- (p-enc2) – (p-enc2 -| img) |- ($(enc2.west)+(0,-5pt)$);

[thick, loss] ($(dec-l0.south)+(5pt,4pt)$) |- ($(dec-l3.south)+(15pt,-20pt)$);
[thick, loss] ($(dec-l1.south)+(5pt,4pt)$) |- ($(dec-l3.south)+(-15pt,-20pt)$);
[thick, loss] ($(dec-l2.south)+(5pt,4pt)$) |- ($(dec-l3.south)+(15pt,-20pt)$);
[edge, loss] ($(dec-l3.south)+(5pt,-20pt)$) -| ($(dec-l3.east)+(13pt,-20pt)$);

[thick] ($(enc-l2.south)+(0,3pt)$) |- ($(gmm.south)+(5pt,-5pt)$);
[edge] ($(gmm.south)+(-5pt,-5pt)$) -| ($(dec-l1.south)+(-5pt,3pt)$);

[thick] ($(enc-l1.south)+(0,3pt)$) |- ($(gmm.south)+(5pt,-10pt)$);
[edge] ($(gmm.south)+(-5pt,-10pt)$) -| ($(dec-l2.south)+(-5pt,3pt)$);

[thick] ($(enc-l0.south)+(0,3pt)$) |- ($(gmm.south)+(5pt,-15pt)$);
[edge] ($(gmm.south)+(-5pt,-15pt)$) -| ($(dec-l3.south)+(-5pt,3pt)$);

[label, fill=white, fill opacity=.75, text opacity=1, rounded corners] (gmm-lbl) at (img-lbl -| gmm) Local space ($z$);
[label, fill=white, fill opacity=.75, text opacity=1, rounded corners] at (enc-lbl -| dec-l4.north) Merger $(f_m)$;

[circle, loss, inner sep=1pt] (ls) at ($(pred.north)!.5!(gt.north)+(0pt,15pt)$) $\ell_s$;
[circle, loss, inner sep=1pt] (lz) at ($(gmm.east)!.5!(gmm2.east)+(15pt,-10pt)$) $\ell_z$;

[circle, draw] (prior) at (enc |-gauss) $\mathcal{N}$;
[label, left=of prior] (prior-lbl) Prior;

[circle, loss, inner sep=1pt] (lg) at ($(gauss.north)!.5!(prior.north)+(0,20pt)$) $\ell_g$;

[circle, loss, inner sep=1pt] (lz) at ($(gmm.east)!.5!(gmm2.east)+(15pt,-10pt)$) $\ell_z$;

[edge, loss] (ls) -| (pred);
[edge, loss] (ls) -| (gt);

[edge, loss] (lz) |- ($(gmm.south east)+(0,5pt)$);
[edge, loss] (lz) |- ($(gmm2.north east)+(0,-5pt)$);

[edge, loss] (lg) -| (prior);
[edge, loss] (lg) -| (gauss);

[use as bounding box] (img.west |- lg.north) rectangle (gt.east |-p-enc2);

Given an input image, we extract local, $z$, and global, $g$, features (orange box) to improve the prediction of the semantic classes.
We model the local features as GMM that fit a predicted posterior from the data (light blue box).
For the global features, we use a single Gaussian to hold the holistic information of the image (green box).
Note that we managed to recover the local features at the decoder stage by combining the GMMs of the latent space $z$ with the encoder features through skip connections and deconvolution operations (purple box).
Finally, the whole process is regularized by making the predictions for matching the ground truth, $\ell_s$, and making the local and global spaces similar to their respective priors, $\ell_z$, and $\ell_g$, respectively.

§ RELATED WORK

Semantic segmentation aims at dense pixel-wise classification, , assign labels to each pixel in fine-grained fashion.
It is repeatedly used as an intermediate step in computer vision applications by facilitating high-level image processing and analysis.
Currently, the deep segmentation approaches based on Fully Convolutional Network (FCN) have made remarkable progress.
However, this architecture is inherently limited to local receptive fields and short-range contextual information.
Due to this shortage of contextual information, the FCN model suffers from two main drawbacks: low-resolution output maps and a loss in spatial precision.

Existing models <cit.> present the first effort to address these problems by refining the output maps using Conditional Random Field (CRF) <cit.>.
However, most of the models that perform post-processing steps lack local and global feature extractors that are efficient enough to discriminate the object boundaries.
In contrast, instead of performing holistic post-processing, other models <cit.> focus on the refinement from bounding boxes, , they obtain a segmentation heat map in each region of interest.

However, the transfer of information (context) remains limited.
Recent works <cit.> verify the importance of context information in SS.
How to extract, merge, and employ this context information is the next step in SS.
Thus, encoder-decoder models, such as U-Net <cit.>, DeconvNet <cit.>, SegNet <cit.>, or DUNet <cit.>, add a decoder stage (, a set of deconvolution and unpooling operations) with skip connection, managing to merge, at some level, local context from low-level features and global context from high-level ones.
For instance, ParseNet <cit.> adds global information to layers through the global average pooling operation, and FC-DenseNet <cit.> combines knowledge by concatenating all the features of the previous output layers.
Although the encoder-decoder models show to focus on details, they still require additional detailed connections to improve their information reconstruction stage.

every node/.style=scale=0.8,
node distance=1.25cm,
pic shift/.store in=,
pic shift=(0, 0),
shift to anchor/.code=
@scan@one@point@firstofone(.@anchor)shift=(-@x, -@y)
pics/named scope code/.style 2 args=code=@fig@mustbenamed
[pic shift, local bounding box/.expanded=@fig@name, #1]#2
layer/.style=draw, text width=4pt, minimum width=4pt, rounded corners, minimum height=#1,
pics/encoder2/.style=named scope code=node distance=4pt
[layer=.8cm, fill=cs-encoder, fill opacity=0.4] (-l2) ;
[layer=1.2cm, above left=of -l2, fill=cs-encoder, fill opacity=0.4] (-l1) ;
[layer=.4cm, below right=of -l2, fill=cs-encoder, fill opacity=0.4] (-l3) ;
[layer=1.6cm, above left=of -l1, fill=cs-encoder, fill opacity=0.4] (-l0) ;
node distance=5pt,
rounded corners,
shorten >= 5pt,
shorten <= 5pt,
rounded corners,
shorten >= 5pt,
shorten <= 5pt,

pic[] (enc3) encoder2;

[fit=($(enc3-l0.north)+(0pt, 0pt)$) ($(enc3-l3.south)+(385pt, -48pt)$), right=0.5cm of enc3, inner sep=+2mm, draw, rounded corners, fill=cs-localz, fill opacity=0.2] (localspace) ;

[layer=1.6cm, right=2.2cm of enc3-l0.east, fill=white] (convsig0) ;
[layer=1.2cm, below=0.1cm of convsig0, fill=white] (convsig1) ;
[layer=0.8cm, below=0.1cm of convsig1, fill=white] (convsig2) ;
[layer=0.4cm, below=0.1cm of convsig2, fill=white] (convsig3) ;

[layer=1.6cm, right=0.8cm of convsig1, fill=white] (gap) ;
[linea] ($(gap.west)+(-5pt, 11pt)$) – ($(gap.east)+(5pt, 11pt)$);
[linea] ($(gap.west)+(-5pt, 0)$) – ($(gap.east)+(5pt, 0)$);
[linea] ($(gap.west)+(-5pt, -11pt)$) – ($(gap.east)+(5pt, -11pt)$);;

(5, 2)600.50.8cs-sky (gaussgmm0);
(5.3, 1)200.80.6cs-vege (gaussgmm1)
(4, 1)400.50.9cs-cyclist (gaussgmm2)
(4, 0)00.80.5cs-sign (gaussgmm3)
(4.2, 2)00.50.5cs-person (gaussgmm4)
(5.3, 0.1)100.60.7cs-car (gaussgmm5)

[edge] ($(gap.east)+(82pt, 0pt)$) – ($(gap.east)+(103pt, 0pt)$);
[linea] ($(gap.east)+(99pt, -40pt)$) – ($(gap.east)+(99pt, 40pt)$);
[linea] ($(gap.east)+(94pt, 35pt)$) – ($(gap.east)+(107pt, 35pt)$);
[linea] ($(gap.east)+(113pt, 35pt)$) – ($(gap.east)+(126pt, 35pt)$);
(mupi0) at ($(gap.east)+(110pt, 30pt)$) [cs-road, label] $\mu_z^0 \times \pi_z^0$;
(mupi1) at ($(gap.east)+(110pt, 15pt)$) [cs-vege, label] $\mu_z^1 \times \pi_z^1$;
(mupi2) at ($(gap.east)+(110pt, 0pt)$) [cs-person, label, opacity=0.6] $\mu_z^2 \times \pi_z^2$;
(mupidots) at ($(gap.east)+(110pt, -15pt)$) [label] $\vdots$;
(mupik) at ($(gap.east)+(110pt, -30pt)$) [cs-car, label, opacity=0.8] $\mu_z^k \times \pi_z^k$;
[linea] ($(gap.east)+(121pt, -40pt)$) – ($(gap.east)+(121pt, 40pt)$);
[linea] ($(gap.east)+(94pt, -35pt)$) – ($(gap.east)+(107pt, -35pt)$);
[linea] ($(gap.east)+(113pt, -35pt)$) – ($(gap.east)+(126pt, -35pt)$);

[cs-road, layer=0.8cm, rotate=90, right=1.6cm of mupi0.south east, fill=white] (proc-gmm0) ;
[cs-vege, layer=0.8cm, rotate=90, right=1.6cm of mupi1.south east, fill=white] (proc-gmm1) ;
[cs-person, opacity=0.6, layer=0.8cm, rotate=90, right=1.6cm of mupi2.south east, fill=white] (proc-gmm2) ;
[cs-car, opacity=0.8, layer=0.8cm, rotate=90, right=1.6cm of mupik.south east, fill=white] (proc-gmmk) ;

[cs-road, layer=0.8cm, rotate=90, right=3cm of mupi0.south east, fill=white] (proc-gmm-up0) ;
[cs-vege, layer=0.8cm, rotate=90, right=3cm of mupi1.south east, fill=white] (proc-gmm-up1) ;
[cs-person, opacity=0.6, layer=0.8cm, rotate=90, right=3cm of mupi2.south east, fill=white] (proc-gmm-up2) ;
[cs-car, opacity=0.8, layer=0.8cm, rotate=90, right=3cm of mupik.south east, fill=white] (proc-gmm-upk) ;

[layer=1.6cm, right=4cm of mupi2.east, fill=white] (concat) ;
[linea] ($(concat.west)+(-5pt, 14pt)$) – ($(concat.east)+(5pt, 14pt)$);
[linea] ($(concat.west)+(-5pt, 5pt)$) – ($(concat.east)+(5pt, 5pt)$);
[linea] ($(concat.west)+(-5pt, -5pt)$) – ($(concat.east)+(5pt, -5pt)$);
(dotsconcat) at ($(concat.west)+(5pt, -7.2pt)$) [label] $\vdots$;
[linea] ($(concat.west)+(-5pt, -14pt)$) – ($(concat.east)+(5pt, -14pt)$);

[layer=1.6cm, right=0.7cm of concat, fill=white] (proc-concat) ;

[edge] (enc3-l0) – node[label, below]conv $+$ sig ($(convsig0.west)+(3pt, 0)$);
[edge] (enc3-l1) – node[label, below]conv $+$ sig ($(convsig1.west)+(3pt, 0)$);
[edge] (enc3-l2) – node[label, below]conv $+$ sig ($(convsig2.west)+(3pt, 0)$);
[edge] (enc3-l3) – ($(convsig3.west)+(3pt, 0)$);

[edge] (convsig0) -| node[label, above]GAP (gap.north);
[edge] ($(convsig1.east)+(0, 5pt)$) – node[label, above]GAP ($(gap.west)+(0, 5pt)$);
[edge] (convsig2.east) – node[label, below right]GAP ++(10pt,0) |-  ($(gap.west)+(0, -5pt)$);
[edge] (convsig3) -| node[label, below]GAP ($(gap.south)+(0, -5pt)$);
[edge] ($(gap.east)+(-2pt, 0)$) – node[label, above]FC ($(gap.east)+(18pt, 0)$);

[edge] ($(mupi0.east)+(0, -2pt)$) – node[label, below]FC $+$ relu (proc-gmm0);
[edge] ($(mupi1.east)+(0, -2pt)$) – node[label, below]FC $+$ relu (proc-gmm1);
[edge] ($(mupi2.east)+(0, -2pt)$) – node[label, below]FC $+$ relu (proc-gmm2);
[edge] ($(mupik.east)+(0, -2pt)$) – node[label, below]FC $+$ relu (proc-gmmk);

[edge] (proc-gmm0) – node[label, below]up (proc-gmm-up0);
[edge] (proc-gmm1) – node[label, below]up (proc-gmm-up1);
[edge] (proc-gmm2) – node[label, below]up (proc-gmm-up2);
[edge] (proc-gmmk) – node[label, below]up (proc-gmm-upk);

[edge] (proc-gmm-up0.south) -| node[label, above] ($(concat.north)+(0, -5pt)$);
[edge] ($(proc-gmm-up1.south)+(0, -1pt)$) – node[label, above] ($(concat)+(-2pt, 12pt)$);
[edge] ($(proc-gmm-up2.south)+(0, -1pt)$) – node[label, above] ($(concat)+(-2pt, -2pt)$);
[edge] (proc-gmm-upk.south) -| node[label, below] ($(concat.south)+(0, 5pt)$);

[edge] (concat) – node[label, below]conv (proc-concat);
[edge] ($(proc-concat.east)+(-3pt, 0)$) – ($(proc-concat.east)+(30pt, 0)$);

[label] at (5, 2.1) $\mu_z^0, \sigma_z^0, \pi_z^0$;
[label] at (5.3, 1) $\mu_z^1, \sigma_z^1, \pi_z^1$;
[label] at (4, 1) $\mu_z^2, \sigma_z^2, \pi_z^2$;
[label] at (4.2, 1.8) $\mu_z^2, \sigma_z^2, \pi_z^2$;
[label] at (4, -0.1) $\mu_z^{k-1}, \sigma_z^{k-1}, \pi_z^{k-1}$;
[label] at (5.3, 0.2) $\mu_z^k, \sigma_z^k, \pi_z^k$;
[label] at ($(proc-concat.north)+(-20pt, 40pt)$) Local Space ($z$);

[label] at ($(enc3-l0.west)+(-17pt, 0pt)$) $\frac{h}{4} \times \frac{w}{4} \times 256$;
[label] at ($(enc3-l1.west)+(-17pt, 0pt)$) $\frac{h}{8} \times \frac{w}{8} \times 512$;
[label] at ($(enc3-l2.west)+(-20pt, 0pt)$) $\frac{h}{16} \times \frac{w}{16} \times 1024$;
[label] at ($(enc3-l3.west)+(-20pt, 0pt)$) $\frac{h}{32} \times \frac{w}{32} \times 2048$;

[label] at ($(convsig0.north)+(0pt, 5pt)$) $\frac{h}{4} \times \frac{w}{4} \times C$;
[label] at ($(convsig3.south)+(0pt, -5pt)$) $\frac{h}{32} \times \frac{w}{32} \times C$;
[label] at ($(gap.south)+(0pt, -5pt)$) $1 \times 256$;

[label] at ($(gap.south east)+(50pt, -30pt)$) Latent Space z;
[label] at ($(gap.south east)+(50pt, -37pt)$) $q_\theta(z | x)$;

[label] at ($(5.3, 0.1)+(45pt, -17pt)$) $K \times 1 \times 256$;
[label] at ($(proc-gmm2.center)+(0pt, -10pt)$) $1 \times 1 \times D$;
[label] at ($(proc-gmmk.center)+(0pt, -10pt)$) $1 \times 1 \times D$;
[label] at ($(proc-gmm-up2.center)+(0pt, -10pt)$) $\frac{h}{32} \times \frac{w}{32} \times D$;
[label] at ($(proc-gmm-upk.center)+(0pt, -10pt)$) $\frac{h}{32} \times \frac{w}{32} \times D$;
[label] at ($(concat.north)+(0pt, 15pt)$) $\frac{h}{32} \times \frac{w}{32} \times [K \times D]$;
[label] at ($(proc-concat.south)+(0pt, -7pt)$) $\frac{h}{32} \times \frac{w}{32} \times 512$;

Local Space diagram.
Note, the blocks and arrows represent the feature vectors and operations, respectively.
We extract features at different scales through convolution operations with a sigmoid activation function.
Next, we use the Global Average Pooling operation (GAP) to summarize the information and gather it into a concatenated feature vector.
To model the latent space $z$ through a GMM, we use the Fully Connected layer (FC) to extract $K$ different Gaussian mixtures.
Next, the mean of each Gaussian is used and stored in a list, and apply an FC with ReLU is to do further feature processing.
Finally, we return to our initial vector size (, $\frac{h}{32} \times \frac{w}{32}$).
We concatenate the $K$ Gaussians processed and perform the last convolution operation to fuse all the information from the different Gaussians.
Although we compress the information using GAP to model the Gaussians distribution, the information contained in the latent space $z$ plays an important role in adequately recovering the geometric information (background and contour of the segmented objects) through the decoder stage.

Although theoretically, in the layers with lower resolution (, size of $H / 32 \times W / 32$), there is a greater receptive field; empirical experiments showed that these fields are significantly smaller and are not enough to capture global context information (, global features) <cit.>.
One way to address these drawbacks is through multi-scale models <cit.> (, ICNet <cit.> is fed with different input sizes).
DeepLabv2 <cit.>, DeepLabv3 <cit.>, DeepLabv3++ <cit.>, DecoupleSegNets <cit.> and Gated-SCNN <cit.> preserve the spatial size of the features maps by proposing Atrous Spatial Pyramid Pooling (ASSP).
ASPP employs a set of atrous convolutions operations <cit.> and average pooling to capture several context information (from local to global features).
However, dilated convolutions can cause grinding problems <cit.>.
It can induce a loss of local information in the models and capture irrelevant information on a large scale
attention models:
A different approach is presented by PSPNet <cit.> and ESPNet <cit.>, operating not on the convolution kernels but on  sub-regions of the feature maps generated at different levels.
They use the Pyramid Pooling Module (PMM) to get global and local context information by reducing the spatial size of features.
Also, HRNetv2 <cit.> and HRNet+OCR <cit.> perform multi-scale feature extraction by sharing feature maps across different branches (scales), , broadcasting context information at various resolutions.
Nevertheless, how and where to combine multi-scale information still represents a challenge.

Although the previous context fusion models help capture different scales' features, The relationship between objects in a global view (essential to SS) is still limited.
Thus, attention mechanisms <cit.> can use to extract long-range contextual information.
Furthermore, CCNet <cit.> efficiently makes a feature from any position perceive the other features, , it adds contextual information in horizontal and vertical directions with sparse attention.
In contrast, DANet <cit.> proposes two attention modules (dual attentions), focused on spatial features and channels.
Finally, OCNet <cit.> extracts global and local context by grouping (by permutation), splitting, and operating each sub-region with a self-attention module (, fully matrix).

Unlike previous models, PHGMM extracts local and global context information from two specific latent spaces, which we provide an adequate structure and behavior to obtain this context information precisely. In addition to providing a decoding stage capable of recovering the geometric information of the objects (background and boundary) by combining both context information (local y global) with the decoder information through skip connections, deconvolutions, and upsampling.

§ GLOBAL AND LOCAL FEATURE MODELING

every node/.style=scale=0.75,
node distance=1.25cm,
pic shift/.store in=,
pic shift=(0,0),
shift to anchor/.code=
pics/named scope code/.style 2 args=code=@fig@mustbenamed
[pic shift, local bounding box/.expanded=@fig@name, #1]#2
layer/.style=draw, text width=4pt, minimum width=4pt, rounded corners, minimum height=#1,
pics/encoder2/.style = named scope code=node distance=4pt
[layer=.8cm, fill=cs-encoder, fill opacity=0.4] (-l2) ;
[layer=1.2cm, above left=of -l2, fill=cs-encoder, fill opacity=0.4] (-l1) ;
[layer=.4cm, below right=0.3cm of -l2.south east, fill=cs-encoder, fill opacity=0.4] (-l3) ;
[layer=1.6cm, above left=of -l1, fill=cs-encoder, fill opacity=0.4] (-l0) ;
pics/decoder/.style = named scope code=node distance=4pt
[layer=.4cm] (-l0) ;
[layer=.8cm, right=of -l0] (-l1) ;
[layer=1.2cm, right=of -l1] (-l2) ;
[layer=1.6cm, right=of -l2] (-l3) ;
node distance=5pt,
rounded corners,
shorten >= 5pt,
shorten <= 5pt,
rounded corners,
shorten >= 5pt,
shorten <= 5pt,
rounded corners,
shorten >= 5pt,
shorten <= 5pt,

pic[] (enc4) encoder2;

[fit=($(enc4-l0.north)+(0pt, 0pt)$) ($(enc4-l3.south)+(229pt, -65pt)$), right=0.3cm of enc4, inner sep=+2mm, draw, rounded corners, fill=cs-globalg, fill opacity=0.2] (globalspaceg) ;

[layer=1.6cm, right=2.2cm of enc4-l0.east, fill=white] (convsig0) ;
[layer=1.2cm, below=0.1cm of convsig0, fill=white] (convsig1) ;
[layer=0.8cm, below=0.1cm of convsig1, fill=white] (convsig2) ;
[layer=0.4cm, below=0.2cm of convsig2.south, fill=white] (convsig3) ;

[layer=1.6cm, right=0.8cm of convsig1, fill=white] (gap) ;
[linea] ($(gap.west)+(-5pt,11pt)$) – ($(gap.east)+(5pt,11pt)$);
[linea] ($(gap.west)+(-5pt,0)$) – ($(gap.east)+(5pt,0)$);
[linea] ($(gap.west)+(-5pt,-11pt)$) – ($(gap.east)+(5pt,-11pt)$);

[layer=1.6cm, right=0.7cm of gap, fill=white] (lsg) ;

(5.3,1)200.80.7 (gaussg0)

[layer=1.2cm, right=2.6cm of lsg, fill=white] (procg0) ;

[layer=1.2cm, right=0.7cm of procg0, fill=white] (procg1) ;

[edge] (enc4-l0) – node[label,below]conv $+$ sig ($(convsig0.west)+(3pt, 0)$);
[edge] (enc4-l1) – node[label,below]conv $+$ sig ($(convsig1.west)+(3pt, 0)$);
[edge] ($(enc4-l2.east)+(0pt, 3pt)$) – node[label,below]conv $+$ sig ($(convsig2.west)+(3pt, 3pt)$);
[edge] (enc4-l3) – ($(convsig3.west)+(3pt, 0)$);

[edge] (convsig0) -| node[label,above]GAP (gap.north);
[edge] ($(convsig1.east)+(0, 5pt)$) – node[label,above]GAP ($(gap.west)+(0, 5pt)$);
[edge] (convsig2.east) – node[label,below right]GAP ++(10pt,0) |- ($(gap.west)+(0, -5pt)$);
[edge] (convsig3) -| node[label,below]GAP ($(gap.south)+(0, -5pt)$);
[edge] (gap) – node[label,above]FC (lsg);
[edge] (lsg) – node[label,above]FC ($(lsg.east)+(20pt, 0pt)$);
[edge] ($(lsg.east)+(52pt, 0pt)$) – node[label,above]FC (procg0);
[edge] (procg0) – node[label,above]up (procg1);

[edge] (procg1) – ($(procg1.east)+(40pt, 0pt)$);

[label] at ($(procg1.north)+(-15pt, 45pt)$) Global Space ($g$);

[label,cs-white] at (5.3,1.0) $\mu_g, \sigma_g$;

[label] at ($(enc4-l0.north)+(8pt, 5pt)$) $\frac{h}{4} \times \frac{w}{4} \times 256$;
[label] at ($(enc4-l1.north)+(8pt, 5pt)$) $\frac{h}{8} \times \frac{w}{8} \times 512$;
[label] at ($(enc4-l2.north)+(10pt, 5pt)$) $\frac{h}{16} \times \frac{w}{16} \times 1024$;
[label] at ($(enc4-l3.north)+(8pt, 5pt)$) $\frac{h}{32} \times \frac{w}{32} \times 2048$;

[label] at ($(convsig0.north)+(0pt, 5pt)$) $\frac{h}{4} \times \frac{w}{4} \times C$;
[label] at ($(convsig3.south)+(0pt, -5pt)$) $\frac{h}{32} \times \frac{w}{32} \times C$;
[label] at ($(gap.south)+(0pt, -5pt)$) $1 \times [4 \times C]$;

[label] at ($(lsg.south)+(0pt, -5pt)$) $1 \times 256$;

[label] at ($(gap.south east)+(65pt, -10pt)$) Latent Space g;
[label] at ($(gap.south east)+(65pt, -17pt)$) $q_\theta(g|x)$;

[label] at ($(procg0.south)+(0pt, -5pt)$) $1 \times M$;

[label] at ($(procg1.south)+(0pt, -5pt)$) $\frac{h}{32}\times \frac{w}{32} \times M$;

Global Space diagram.
The blocks and arrows represent the feature vectors and operations, respectively.
First, we extract features at different scales through convolution operations with a sigmoid activation function.
Next, we use the Global Average Pooling operation (GAP) to summarize the information and gather it into a concatenated feature vector.
Subsequently, we carry out two Fully Connected layers (FC) for a better features extraction, obtaining our Normal Gaussian.
Then another FC transforms the mean of the Gaussian into a vector feature that is scaled to $\frac{h}{32} \times \frac{w}{32}$.

In segmentation, global features can generalize entire objects or a set in a compact representation, while local features, on the other hand, compute representations for particular (more focused) parts of the image.
However, due to the sensitivity to occlusion in global features or noise in local ones, it is not straightforward to combine both features reliably <cit.>.
Consequently, we propose a model capable of merging these local and global features in a suitable manner.
Our contribution comes from combining the features in two stages.
First, we decode the local features representation, and then add the global ones once the image structure has been recovered in the decoder.
We recover structural information, , background and contour of the objects, by merging local and global features (latent spaces) with decoder features through skip connections and upsampling in the decoding stage.
Thus, we obtain diverse types of contextual information, reducing overfitting and addressing the loss of spatial precision problem.

Current semantic segmentation models work by extracting features and pooling them to reduce the representation dimension.
Then, while decoding the encoded features, these models pass part of the encoded information to aid the decoder in producing better features.
In a way, these models use local features and share them across the encoding and decoding tasks to improve the final prediction <cit.>.
Differently, we propose to use local and global features to improve the classes' representation by adding information to the shared knowledge between these two stages.
Our proposal is to create a global model that holds the information of macro objects of the scene, while using the traditional local features, as well as the helpful combination of these and encoder features; through the decoder.
We propose to use Gaussian Mixture Models to represent both spaces (, global and local), since we need to hold multi-modal representations due to the different classes present in a scene (Fig. <ref>).

Our objective is to model two latent spaces to perform a better and more robust extraction of local (Fig. <ref>) and global (Fig. <ref>) features and an effective way to combine them through the decoder (Fig. <ref>) to recover the geometric information of the objects.
Thus, we propose using an end-to-end model that fits and employs two latent spaces suitable for the segmentation task.
In order to provide an internal structure to the learned representation (, latent space), we merge VAE with ResNet-101 <cit.> (for an encoder-decoder stage for SS).
A VAE allows us to represent the latent space through a structure, , a predefined distribution.
Furthermore, the ResNet-101 backbone provides robust processing (, residual block) to extract the basic features that will turn into the distributions.
As a result, our model obtains a local context for the discrimination of segmented objects (, class-level information) as a mixture of Gaussians.
The model mixes the structure of the local-global features by employing concatenation operation and upsampling throughout the deconvolution stage.
In addition, the decoder stage is responsible for recovering the geometric information of the objects.
Combining the context information (local-global structure features) previously extracted with the features coming from the encoder through skip connections in the different scales.
Integrating these two latent spaces provides the neural network with a powerful representation (Fig. <ref>).
The latent spaces provide complementary information that improves the semantic segmentation (Tables <ref> and <ref>).

We present an overview of our PHGMM model in Fig. <ref>, and detail the inference and generation processes in Sections <ref> and <ref>, respectively.
Also, in Section <ref>, we split the description of the operations used for local and global context extraction.
The local and global extraction processes are shown in Figs. <ref> and <ref>, respectively.
Fig. <ref> shows that our decoder is in charge of intelligently merging the previously extracted features so that PHGMM recovers the geometric information of the segmented objects addressing the problem of loss of spatial precision (see Table. <ref>).
Finally, in Section <ref>, we describe how our model is trained.

every node/.style=scale=0.75,
node distance=1.25cm,
pic shift/.store in=,
pic shift=(0,0),
shift to anchor/.code=
pics/named scope code/.style 2 args=code=@fig@mustbenamed
[pic shift, local bounding box/.expanded=@fig@name, #1]#2
layer/.style=draw, text width=4pt, minimum width=4pt, rounded corners, minimum height=#1,
pics/encoder2/.style = named scope code=node distance=10pt
[layer=1.6cm, fill=cs-encoder, fill opacity=0.4] (-l0) ;
[layer=1.2cm, below right=of -l0, fill=cs-encoder, fill opacity=0.4] (-l1) ;
[layer=.8cm, below right=of -l1, fill=cs-encoder, fill opacity=0.4] (-l2) ;
[layer=.4cm, below right=of -l2, fill=cs-encoder, fill opacity=0.4] (-l3) ;
pics/decoder/.style = named scope code=node distance=10pt
[layer=.4cm, fill=cs-decoder, fill opacity=0.4] (-l0) ;
[layer=.8cm, above right=of -l0, fill=cs-decoder, fill opacity=0.4] (-l1) ;
[layer=1.2cm, above right=of -l1.north, fill=cs-decoder, fill opacity=0.4] (-l2) ;
[layer=1.6cm, above right=of -l2.north, fill=cs-decoder, fill opacity=0.4] (-l3) ;
node distance=5pt,
rounded corners,
shorten >= 5pt,
shorten <= 5pt,
rounded corners,
shorten >= 5pt,
shorten <= 5pt,

pic[] (enc3) encoder2;

[layer=1.7cm, rotate=90, below right=0.8cm of enc3-l3, fill=cs-localz, fill opacity=0.2] (lszlocal) ;
[layer=1.7cm, rotate=90, below=0.5cm of lszlocal.north, fill=cs-globalg, fill opacity=0.2] (lsgglobal) ;

[right=2.4cm of enc3-l3] (dec) decoder;

[draw, right=15.7pt of dec-l3, minimum width=18pt, rounded corners, minimum height=1.8cm, fill=cs-fm, fill opacity=0.2] (deconcat) ;
[draw, right=18.8pt of dec-l3, minimum width=2pt, minimum height=1.8cm] (deconcat1) ;
[draw, right=21.2pt of dec-l3, minimum width=2pt, minimum height=1.8cm] (deconcat2) ;

[edge, draw=cs-fenc] (enc3-l3) – (dec-l0);
[edge, draw=cs-fenc] (enc3-l2) – (dec-l1);
[edge, draw=cs-fenc] (enc3-l1) – (dec-l2);
[edge, draw=cs-fenc] (enc3-l0) – (dec-l3);

[edge, draw=black!50] ($(enc3-l0.south)+(2pt, 4pt)$) |- ($(enc3-l1.north)+(0pt, -4pt)$);
[edge, draw=black!50] ($(enc3-l1.south)+(2pt, 4pt)$) |- ($(enc3-l2.north)+(0pt, -4pt)$);
[edge, draw=black!50] ($(enc3-l2.south)+(2pt, 4pt)$) |- ($(enc3-l3.north)+(0pt, -4pt)$);

[edge, draw=black!50] ($(dec-l2.north)+(0pt, -3pt)$) -| ($(dec-l3.south)+(0pt, 4pt)$);
[edge, draw=black!50] ($(dec-l1.north)+(0pt, -3pt)$) -| ($(dec-l2.south)+(0pt, 4pt)$);
[edge, draw=black!50] ($(dec-l0.north)+(0pt, -2pt)$) -| ($(dec-l1.south)+(0pt, 4pt)$);

[linea] ($(enc3-l0.south)+(0pt, 4pt)$) |- ($(lszlocal.west)+(-25pt, -3pt)$);
[linea] ($(enc3-l1.south)+(0pt, 4pt)$) |- ($(lszlocal.west)+(-25pt, -3pt)$);
[linea] ($(enc3-l2.south)+(0pt, 4pt)$) |- ($(lszlocal.west)+(-25pt, -3pt)$);
[linea] ($(enc3-l3.south)+(0pt, 4pt)$) |- ($(lszlocal.west)+(-25pt, -3pt)$);
[edge] ($(lszlocal.west)+(-30pt, -8pt)$) |- ($(lszlocal)+(-14pt, 0pt)$);
[edge] ($(lszlocal.west)+(-30pt, -8pt)$) |- ($(lsgglobal)+(-14pt, 0pt)$);

[edge] ($(lszlocal.center)+(14pt, 0pt)$) -| ($(dec-l0)+(0pt, 0pt)$);
[edge, draw=cs-fm] ($(dec-l3)+(0pt, 0pt)$) – ($(deconcat)+(-3pt, 0pt)$);
[edge, draw=cs-fm] ($(dec-l2)+(0pt, 0pt)$) -| ($(deconcat.south)+(-3pt, 5pt)$);
[edge, draw=cs-fm] ($(dec-l1)+(0pt, 0pt)$) -| ($(deconcat.south)+(0pt, 5pt)$);
[edge, draw=cs-fm] ($(dec-l0)+(0pt, 0pt)$) -| ($(deconcat.south)+(3pt, 5pt)$);
[edge, draw=cs-fm] ($(lsgglobal)+(14pt, 0pt)$) -| ($(deconcat.south)+(6pt, 5pt)$);

[label] at ($(lszlocal.center)+(0pt, 0pt)$) $z$;
[label] at ($(lsgglobal.center)+(0pt, 0pt)$) $g$;

[label] at ($(enc3-l0.north)+(8pt, 5pt)$) $\frac{h}{4} \times \frac{w}{4} \times 256$;
[label] at ($(enc3-l1.north)+(8pt, 5pt)$) $\frac{h}{8} \times \frac{w}{8} \times 512$;
[label] at ($(enc3-l2.north)+(10pt, 5pt)$) $\frac{h}{16} \times \frac{w}{16} \times 1024$;
[label] at ($(enc3-l3.north)+(8pt, 5pt)$) $\frac{h}{32} \times \frac{w}{32} \times 2048$;

[label] at ($(dec-l0.north)+(-11pt, 5pt)$) $\frac{h}{32} \times \frac{w}{32} \times [3 \times C]$;
[label] at ($(dec-l1.north)+(-16pt, 5pt)$) $\frac{h}{16} \times \frac{w}{16} \times [3 \times C]$;
[label] at ($(dec-l2.north)+(-14pt, 5pt)$) $\frac{h}{8} \times \frac{w}{8} \times [3 \times C]$;
[label] at ($(dec-l3.north)+(-13pt, 4pt)$) $\frac{h}{4} \times \frac{w}{4} \times [3 \times C]$;

[label] at ($(deconcat.north)+(-3pt, 9pt)$) $\frac{h}{4} \times \frac{w}{4} \times [3 \times C \times M]$;

[label] at ($(lszlocal.west)+(-55pt, -8pt)$)conv $+$ sig;
[label] at ($(lsgglobal)+(60pt, 5pt)$)conv $+$ up;
[label] at ($(dec-l2)+(18pt, 5pt)$)conv $+$ relu;
[label] at ($(dec-l2)+(15pt, -5pt)$)$+$ up;
[label] at ($(dec-l1)+(26pt, 5pt)$)conv $+$ relu;
[label] at ($(dec-l1)+(26pt, -5pt)$)$+$ up;
[label] at ($(dec-l0)+(40pt, 5pt)$)conv $+$ relu $+$ up;

[label, above=of dec-l3 -| lszlocal] (c3-lbl) conv;
[label, above=of dec-l2 -| lszlocal] (c2-lbl) conv;
[label, above=of dec-l1 -| lszlocal] (c1-lbl) conv;
[label, above=of dec-l0 -| lszlocal] (c0-lbl) conv;

[nodecircle, inner sep=0.5pt] (zop) at ($(lszlocal)+(30pt, 0pt)$) $||$;
[nodecircle, inner sep=0.5pt] (gop) at ($(lsgglobal)+(94pt, 0pt)$) $||$;
[nodecircle, inner sep=0.5pt] (decl0op) at ($(dec-l0)+(56pt, 0pt)$) $||$;
[nodecircle, inner sep=0.5pt] (decl1op) at ($(dec-l1)+(41pt, 0pt)$) $||$;
[nodecircle, inner sep=0.5pt] (decl2op) at ($(dec-l2)+(30pt, 0pt)$) $||$;

[nodecircle, inner sep=0.5pt] (encl0op) at ($(dec-l0)+(-20pt, 0pt)$) $+$;
[nodecircle, inner sep=0.5pt] (encl1op) at ($(dec-l1)+(-20pt, 0pt)$) $+$;
[nodecircle, inner sep=0.5pt] (encl2op) at ($(dec-l2)+(-20pt, 0pt)$) $+$;
[nodecircle, inner sep=0.5pt] (encl3op) at ($(dec-l3)+(-20pt, 0pt)$) $+$;

[layer=0.4cm, rotate=90, above right=1.5cm of enc3-l0.north, fill=cs-encoder, fill opacity=0.4] (legendenc) ;
[label] at ($(legendenc.center)+(15pt, 0pt)$)Encoder;
[layer=0.4cm, rotate=90, below=0.4cm of legendenc.north, fill=cs-decoder, fill opacity=0.4] (legenddec) ;
[label] at ($(legenddec.center)+(19pt, 0pt)$)Decoder ($f$);
[layer=0.4cm, rotate=90, right=1.6cm of legendenc.south west, fill=cs-localz, fill opacity=0.2] (legendz) ;
[label] at ($(legendz.center)+(24pt, 0pt)$)Local space $(z)$;
[layer=0.4cm, rotate=90, below=0.4cm of legendz.north, fill=cs-globalg, fill opacity=0.2] (legendg) ;
[label] at ($(legendg.center)+(25pt, 0pt)$)Global space $(g)$;
[edge, draw=cs-fenc] ($(legendz)+(50pt, 0pt)$) – ($(legendz)+(70pt, 0pt)$);
[label] at ($(legendz)+(79pt, 0pt)$)Skip $(f_{\text{enc}})$;
[layer=0.4cm, rotate=90, right=2.3cm of legendg.south west, fill=cs-fm, fill opacity=0.2] (legendfm) ;
[edge, draw=cs-fm] ($(legendg)+(50pt, 0pt)$) – ($(legendg)+(70pt, 0pt)$);
[label] at ($(legendg)+(91pt, 0pt)$)Merger $(f_m)$;

The decoding diagram is fed by the previously processed feature vector  (, local space $z$).
Next, the decoder takes the skip connections, deconvolution, and upsampling from the encoding.
Finally, in its last layer, it concatenates the information of the different levels (, multi-scale information) with the global information (, global space $g$).
Note, we call the latent space $g$ a global representation because this space summarizes features of objects in a holistic way (, a global view of the object).
Also, our decoding stage is in charge of merging the different context information extracted at the different levels of our network.
This smart fusion recovers the geometric information of the segmented objects.

§.§ Inference Process

We jointly model the global information of the scene $g$ and the local features $z$ of the image $x$ and assume that they are independent.
We set the distribution that models this relation as $q_\theta(z, g \given x) = q_\theta(z \given x) q_\theta(g \given x)$.
The local distribution

\begin{equation}
\label{eq:latent_space_z}
q_\theta(z \given x) = \sum_{k=1}^K  \pi^k(x) \mathcal{N} \left( z; \mu_z^k(x), \sigma_z^k(x)^2 \mathbf{I} \right)
\end{equation}

is a mixture of Gaussians that models $K$ clusters of related features that are helpful in obtaining different classes for the segmentation.
The functions $\mu_z^k(\cdot)$, $\sigma_z^k(\cdot)$, and $\pi^k(\cdot)$ are the parameters of the $k$-th component of the mixture and depend on the given image $x$ (see Fig. <ref>).
All are approximated through a ResNet-101 backbone with separate projection heads that output the corresponding parameter.
On the other hand, the global distribution

\begin{equation}
\label{eq:latent_space_g}
q_\theta(g \given x) = \mathcal{N} \left( g; \mu_g(x), \sigma_g(x)^ 2 \mathbf{I} \right)
\end{equation}

is a single Gaussian that models a global context that holds all the image $x$ information.
The parameter functions $\mu_g(\cdot)$ and $\sigma_g(\cdot)$ are computed in a similar way as the previous parameters.

The local features $z$ hold information about the features, while the global ones $g$ serve as a latent variable to model holistic information from the whole scene.
Additionally, due to the way we engineer the decoding stage (see Fig. <ref>), the global features also serve to spread the gradient back to the encoder.

§.§ Generative Process

To obtain the classes $y$ for the segmentation, we model the joint probability $p(x, y, z, g)$ of the data, the images $x$ and the labels $y$, and the latent variables, the local $z$ and global $g$.
We factorize the joint as $p_{\theta,\phi}(x, y, z, g) = p_{\phi}(y \given z, g) p_\theta(z \given x) p_\theta(g \given x) p(x)$.

To implement the generative process from $p(y \given z, g)$, we use the decoder $f(\bar{\mu}_z)$, where $\bar{\mu}_z = \Cat_{k=1}^K \mu_z^k$ is the concatenation of all the means from the mixture.
Later, a merge function $f_m(f(\bar{\mu}_z), \mu_g, f_{\text{enc}})$ combines the partially decoded segmentations with the parameters from the global context and the encoder features, $f_{\text{enc}}$, from the skip connections.
Hence, the predicted labels are $y' = f_m(f(\bar{\mu}_z), \mu_g, f_{\text{enc}})$.
For a better interpretation, we present the associated diagram in Fig. <ref>, in later sections.

§ IMPLEMENTATION

In this section, we explain in detail the PHGMM architecture, , local space $z$ and global space $g$, as well as the operations involved in modeling latent spaces.
In addition, we explain how to merge the different local-global context information and the encoder features to recover the details of the segmented objects (, the geometric information).
Finally, we perform the loss functions used to adjust the PHGMM parameters and show the hyperparameters used for our model.

§.§ Architecture

We use a ResNet-101 as a backbone for the PHGMM model due to its good behavior, , the trade-off between good feature extraction and the number of parameters used, that is, most of the residual blocks and depths of the layers.
The samples that PHGMM receives as input are passed through convolution with kernel $7 \times 7$ and depth $64$ to carry out the information compression, thus converting the samples in size $\frac{h}{4} \times \frac{w}{4} \times 64$.
Next, we use a defined decoder in four groups of blocks of depth $256$, $512$, $1024$, and $2048$ respectively.
Each group presents a set of $3$, $4$, $23$, and $3$ internal residual blocks.
Notice that our feature vector shows a reduction in the size of $(\frac{h}{4} \times \frac{w}{4})$, $(\frac{h}{8} \times \frac{w}{8})$, $(\frac{h}{16} \times \frac{w}{16})$, and $(\frac{h}{32} \times \frac{w}{32})$ as it passes through the decoder.

One of the issues we address in this paper is how to take advantage of this multi-scale information to improve the edges of the segmented objects and thus address the problem of the loss of spatial precision.
To address this, we create local space $z$ (Fig. <ref>) to model the latent space through a mixture of Gaussians, focusing on local features and global space $g$ (Fig. <ref>) to extract global information through Gaussian modeling of a second latent space.
Additionally, we use skip connections to pass complemental multi-scale information from the encoder to the decoder.

Thus, in local space $z$ (see Fig. <ref>), we use $3 \times 3$ kernel convolutions together with a non-linear sigmoid function to compress the information of the different levels, reducing the depth from $256$, $512$, $1024$, and $2048$ for the number of classes ($C$).
We concatenate the feature vectors to merge the multi-scale information through Global Average Pooling (GAP).
To model the latent space $z$, we use a Fully Connected layer (FC) and produce $K$ mixtures of Gaussians (, $\mu_z^k$, $\sigma_z^k$, $\pi_z^k$).
Note that the concatenation vector and the Gaussian parameters (, $\mu$, $\sigma$) have a size of $1 \times 256$ (chosen empirically).
We combine the local information from the GMMs by stacking the $K$ vectors (, $\mu_z^k \times \pi_z^k$).
For each item in the stack, we perform an FC with a non-linear function RELu, transforming the features of size $1 \times 256$ to $1 \times 1 \times D$.
Then we scale the vectors achieving $\frac{h}{32} \times \frac{w}{32} \times D$.
Finally, we perform a non-linear combination on the concatenation vector of size $\frac{h}{32} \times \frac{w}{32} \times [K \times D]$, reaching a depth of $512$.
These feature vector locales will be used as input for our encoder.
Note that even though we use GAPs in the process of converting the samples to Gaussian distributions, we call local latent space to the local space $z$ because, from this feature vector, the geometric reconstruction is performed to recover and refine the details of the edges of segmented objects.

On the other hand, the global space $g$ (see Fig. <ref>) performs the treatment of multi-scale information to a certain extent similar to the local space $z$, through the concatenation (of size $1 \times [4 \times C]$) of feature vectors summarized by GAP operations.
Next, we use two FC to model the latent space $g$ and give it the structure of a Normal Gaussian with parameters $\mu_g$ and $\sigma_g$, both of size $1 \times 256$ (chosen empirically).
Finally, the information is combined by taking $\mu_g$ and passing it through an FC, obtaining a vector of global features of size $1 \times M$, followed by an upscale operation achieving a vector size of $\frac{h}{32} \times \frac{w}{32} \times M$.
Note that we use the latent space $g$ as a global representation because $g$ is in charge of extracting the holistic information of the objects and then concatenates it in the last reconstruction layer of the decoder.

We merge both latent spaces ($z$ and $g$), and decoder features through the decoding stage.
The decoder (see Fig. <ref>) obtains as input the latent space $z$ and has four groups of residual units, where each group has $3$, $23$, $4$, and $3$ residual units, all with $3 \times C$ depth and a feature vector of size $\frac{h}{32} \times \frac{w}{32}$, $\frac{h}{16} \times \frac{w}{16}$, $\frac{h}{8} \times \frac{w}{8}$ and $\frac{h}{4} \times \frac{w}{4}$ respectively.
In each residual unit group of the decoder, our PHGMM model performs a geometric information retrieval by combining the local features coming from the GMM (latent space $z$) with the features extracted in the encoding through an addition operation.
We transform the encoder features through nonlinear combination, , convolution operations and non-linearity functions, using a $3 \times 3$ kernel with depth $C$.
These features are passed from the different levels of the encoder to the decoder (, multi-level information) with the use of skip connections.
In the last layer, we concatenate the decoder's multi-scale reconstruction information through convolution operations with kernel $3 \times 3$ and a non-linearity function RELu and upscale applied to each level (, group) of the decoder.
Besides, a convolution operation of $3 \times 3$ with upscale to latent space $g$.
The concatenation of all this information gives a resulting feature vector of size $\frac{h}{4} \times \frac{w}{4} \times [3 \times C \times M]$.
Finally, for the output of PHGMM, a convolution operation with kernel $8 \times 8$ and a Softmax are applied to combine all the information and obtain our final heat map.

table-format = 2.2,

[Results on Cityscapes validation]IoU results on Cityscapes validation set for semantic segmentation, using $11$ classes and with crop size of $384 \times 768$. The classes are: cs-sky1 Sky, cs-build1 Building, cs-road1 Road, cs-sidewalk1 Sidewalk, cs-fence1 Fence, cs-vege1 Vegetation, cs-pole1 Pole, cs-car1 Car, cs-sign1 Sign, cs-person1 Person, cs-cyclist1 Cyclist.

Model     cs-sky1.1     cs-build1.1     cs-road1.1     cs-sidewalk1.1     cs-fence1.1     cs-vege1.1     cs-pole1.1     cs-car1.1     cs-sign1.1     cs-person1.1     cs-cyclist1.1     mIoU (%)

SegNet <cit.>     73.74     79.29     92.70     59.88     13.63     81.89     26.18     78.83     31.44     45.03     43.46     56.92

DeconvNet <cit.>     89.38      83.08      95.26      68.07      27.58     85.80      34.20     85.01      27.62     45.11      41.11      62.02

DeepLab v2 <cit.>     74.28     81.66     90.86     63.3     26.29     84.33     27.96     86.24     44.79     58.89     60.92     63.59

FCN8 <cit.>     76.51      83.97      93.82      67.67      24.91      86.38      31.71      84.80      50.92      59.89     59.11      65.43

FastNet <cit.>     77.69     86.25     94.97     72.99     31.02     88.06     38.34     88.42     52.34     61.76     61.83     68.52

ParseNet <cit.>     90.76340732     85.20362826     92.00606416     63.49442359     38.99991853     88.20904710     46.81669972     89.19935668     61.90421068     63.91352437     62.72517683     71.2032233889497

ESPNet  <cit.>     91.78638812     86.35574516     95.73185838     71.84438459     48.52102068     88.4447178     49.06453645     87.29345279     54.60143022     61.82588989     57.51110799     72.08913927909090

FC-DenseNet67 <cit.>     92.19346337765376     86.76988562706273     96.59884889225275     75.39657905029527     41.551418122907954     88.07405997289057     52.923282245565694     87.09217726828671     63.88987642687341     60.48381609817361     52.92274378685763     72.5360137153472

BiSeNet <cit.>     91.63755455     87.42330637     96.48050315     75.41466065     44.04604326     89.07274113     39.50127053     89.33916815     58.62617238     66.80642806     63.07673701     72.85678047715217

ENet <cit.>     91.62527061     87.48086007     96.43995471     75.34161092     48.44360288     89.23168657     43.13950425     89.24173128     56.03890594     65.12868147     63.7010055     73.25571038043044

DeepLab v3 <cit.>     92.82     89.02     96.74     78.13     41.00     90.81     49.74     91.02     64.48     66.52     66.98     75.21

PSPNet <cit.>     91.94289991     89.93398895     96.93718646     78.37073789     53.6440468     90.18757238     43.47155108     92.11920061     64.40262511     70.70955096     70.94303548     76.60567233102417

DANet <cit.>     92.24915714     90.26078544     97.24916248     79.95381396     51.33334745     90.59671054     45.20256646     92.50120637     66.37661578     71.4706024     71.24568649     77.1308776837044

AdapNet++ <cit.>     93.06606033717834     89.46445452619719     97.05622901868213     80.02680655465022     49.46480016553393     90.57898007902187     52.099869078274374     92.22088910324935     66.2616099406663     72.88264400519273     70.6234837933447     77.61325696381738

CCNet <cit.>     90.96516     89.00806     96.58898     77.36207     42.48835     91.36251     58.24336     91.21552     71.17928     74.80497     70.57466     77.61753851250874

OCNet <cit.>     92.71713413     90.73445133     97.39139266     80.80080867     54.58097682     90.85808692     45.59792652     92.6447007     67.35220639     71.80571641     71.97828486     77.86015321793398

CGBNet <cit.>     92.97664103982117     89.39958276438676     96.66423105831508     77.59557837050379     42.80119490412156     91.88485470080798     57.52385700042729     91.13894547133451     73.28850836068722     75.2576623028372     71.1815062265886     78.15568747271192

DUNet <cit.>     93.3292409     91.04735692     97.27624635     80.17870018     55.14552055     91.3528578      53.6978367     92.88570205     68.3278322     73.33193179     72.29190738     78.9877393463894

HRNet <cit.>     94.56263803392699     90.9782601331852     97.4834383731036     82.46280689648792     50.269730202459684     92.35495660868773     61.571180462866714     93.96316213743727     73.14349328079304     78.55382561397901     75.44369460350559     80.98065330422115

HRNet + OCR <cit.>     94.4358545529273     91.56636315886351     97.5902237064308     82.99885185368369     55.464824930830105     92.49025959872975     62.67544657375434     93.97781763087684     75.66912451675692     78.69563372557992     75.34932305850215     81.90124757335776

PHGMM     94.63233601001029    91.60106905124312     97.69572777111502     83.64714307025132     55.26833420934679     92.51727151053036     62.71005972946803     94.2014523189135     75.34570228830016     79.21058371057573     76.45046505096894     82.11637679279302

table-format = 2.2,

[Results on Synthia validation]IoU results on Synthia validation set for semantic segmentation, using $11$ classes and with crop size of $384 \times 768$. The classes are: st-sky1 Sky, st-build1 Building, st-road1 Road, st-sidewalk1 Sidewalk, st-fence1 Fence, st-vege1 Vegetation, st-pole1 Pole, st-car1 Car, st-sign1 Sign, st-person1 Person, st-cyclist1 Cyclist.

Model     st-sky1.1     st-build1.1     st-road1.1     st-sidewalk1.1     st-fence1.1     st-vege1.1     st-pole1.1     st-car1.1     st-sign1.1     st-person1.1     st-cyclist1.1     mIoU (%)

SegNet <cit.>     91.90     87.19     83.72     80.94     50.02     71.63     26.12     71.31     1.01     52.34     32.64     58.98

FCN8 <cit.>     92.36      91.92     88.94     86.46     48.22     77.41     36.02     82.63     30.37     57.10     46.84     67.11

DeconvNet <cit.>     95.88     93.83     92.85     90.79     66.40     81.04     48.23     84.65     0.00     69.46     52.79     70.54

ParseNet <cit.>     93.80     93.09     91.05     88.98     53.22     79.48     46.15     85.37     36.00     63.30     50.82     71.02

DeepLab v2 <cit.>     94.07     93.34     88.07     88.93     55.57     80.22     45.97     85.87     38.73     64.40     52.54     71.61

FC-DenseNet67 <cit.>     92.73814556976933     89.9395336203245     83.35090004988923     85.72130330179895     74.47123045060108     70.46750674345112     51.13256847382862     84.20938649729955     35.10596107373156     68.7789144013552     54.543095772903094      71.85986781408657

FastNet <cit.>     92.21     92.41     91.85     89.89     56.64     78.59     51.17     84.75     32.03     69.87     55.65     72.28

ESPNet <cit.>     95.43823286     93.15770559     91.5216543     89.62311607     65.97587661     80.11211991     48.08913894     84.9277357     39.62219107     66.80256737     53.91453925     73.56226160672742

BiSeNet <cit.>     95.03744954     92.97008983     91.24806482     89.07589772     63.72464811     79.66567326     48.43960044     84.76681067     45.67717217     65.88504126     54.01226264     73.68206458813599

ENet <cit.>     94.80623702     93.01018077     91.51169991     89.67088157     65.56147227     79.03701313     50.78473701     85.2168258     42.08862661     67.67363271     54.13601256     73.9543017600822

ICNet <cit.>     95.7844326     94.31929631     92.61893427     90.76403926     64.91143153     83.17492561     54.37105364     87.52909058     51.15457222     68.02915624     56.4693035     76.28420325029799

DeepLab v3 <cit.>     95.30     92.75     93.58     91.56     73.37     80.71     55.83     88.09     44.17     75.65     60.15     77.38

DANet <cit.>     96.93447143     96.00161362     95.21915878     93.82209322     70.73170426     87.36041471     64.00945124     92.6669109     61.39760396     76.18953394     65.70174948     81.82133686747711

PSPNet <cit.>     96.92212494     96.00194732     95.26771804     93.84681381     71.17356701     87.29600412     65.19687298     92.97791942     62.7326132     76.62439962     66.24129414     82.20738859888307

CGBNet <cit.>     97.06787     96.20894     95.4536     94.06276     71.59252     87.90565     66.27628
93.27113     64.45714     77.08918     66.96545     82.7591380704133

AdapNet++ <cit.>     97.09202     96.23805     95.47351     94.09333     71.78852     87.97434     66.23204
93.32698     64.92306     77.15782     67.20667     82.86421346545318

CCNet <cit.>     97.22151909     96.17349084     95.44657886     93.88143657     72.13568526     88.36453937     66.73581338     93.19174904     67.74466553     76.88199048     67.69651846     83.22490789873517

OCNet <cit.>     97.39121591     96.49751938     95.68929423     94.37198817     72.49477248     88.83951937     67.49981915     93.63770127     67.92060139     77.5858862      67.68750816     83.60143870072893

HRNet <cit.>     96.72645     94.75738     94.828     93.94678     86.99789     83.62414     72.81181     91.90143     61.74201     85.30969     71.08358     84.8844687057615

HRNet + OCR <cit.>     96.7443     94.82684     94.29712     93.46798     87.79663     84.33265     75.36277
92.54169     62.77966     87.13963     72.88337      85.65205732467531

DUNet <cit.>     97.57455889     96.47748927     96.11347477     95.04700978     80.56647131     88.08831442     73.95292407     93.70595222     67.47618125     82.3821732     71.0970727     85.68014744482672

PHGMM     97.00813502905118     95.89240129113708     96.2347068203646     95.17129574558614     85.0293086295568     84.9081549899404     71.77954488726537     94.2647961456622     68.26379118088354     85.49675649722813     72.41706457197729     86.04235961715027

§.§ Training

The target function used to train our PHGMM model $\mathcal{L}$, per data sample, is a linear combination of the specific loss functions $\ell_g$, $\ell_z$, and $\ell_s$ functions, defined by

\begin{equation}
\label{eq:loss_general}
\mathcal{L} = \lambda_g \ell_g + \lambda_z \ell_z + \lambda_s \ell_s,
\end{equation}

where $\ell_g$ and $\ell_z$ are bound to provide a useful structure (, helpful embedding) for the latent spaces,
and $\ell_s$ on the loss function $\mathcal{L}$ focuses on dense pixel-level classification.
Also, $\lambda_{g, z, s}$ are hyperparameters of each term, respectively.
Note, to train with a batch of data, we aggregate the losses over it.

For learning the global representation $g$, we need to find a relevant embedding for the segmentation task (, find useful low-level features).
Consequently, we use Kullback-Leibler divergence (KL) <cit.> to provide a structure to $g$ its prior, ,

\begin{equation}
\label{eq:KL_g}
\ell_g = \kl[\big]{q_\theta(g \given x)}{p(g)}.
\end{equation}

The KL works as a regularizer over $g$ and measures the divergence between the encoder distribution $q(g \given x)$ and the prior $p(g)$.
We specify $p(g)$ as a standard Normal distribution $p(g) = \mathcal{N}(g; 0, \text{I})$, and define $q(g \given x)$ as before (<ref>).

Our second term provides the latent space $z$ with a clustering behavior.
Instead of using a default prior, we infer a conditional $p(z \given x, y)$ from data with another net, called PostNet (see Fig. <ref>).
This PostNet uses the ground truth labels and image data to produce the posterior distribution while training to serve as regularizer.
The posterior is also a mixture of Gaussians defined by

\begin{equation}
p_\gamma(z \given x, y) = \sum_{k=1}^K  \pi^k_{z^*}(x, y) \mathcal{N} \left( z; \mu_{z^*}^k(x, y), \sigma_{z^*}^k(x, y)^2 \mathbf{I} \right).
\end{equation}

We fit the predicted distribution with the infer one through

\begin{equation}
\label{eq:KL_z}
\ell_z = \kl[\big]{q_\theta(z \given x)}{p_\gamma(z \given x, y)}.
\end{equation}

Recall that our PHGMM model learns to densely predict the semantic classes given an input image $x$.
Thus, the reconstruction loss function is expressed as a cross-entropy loss between the prediction $y'$ and its ground-truth $y$.
Penalizing the pixel-wise prediction for each class is not enough.
Thus, we penalize the segmented objects' contours as well.
We employ the loss function soft intersection-over-union <cit.>.
So our third term, $\ell_s$, is

\begin{equation}
\label{eq:loss_recons}
\ell_s  = -\frac{1}{C}\sum_{c=1}^C y_c \log(y'_c) + 1 - \frac{1}{C}\sum_{c=1}^C \left(\frac{y'_c \cap y_c }{y'_c \cup y_c}\right),
\end{equation}

where $y'_c$ is our prediction of the ground truth $y_c$ for the class $c$.

Finally, We provide an algorithm in Alg. <ref> to better understand the training framework, which summarizes the PHGMM steps process.

Training the PHGMM model

$\theta_{\text{Enc}}, \phi_{\text{Dec}}, \gamma_{\text{PosNet}} \gets $ Initialize networks parameters
$X \gets$ mini-batch image from dataset
$Y \gets$ mini-batch label from dataset
$X'_1, X'_2, X'_3, X'_4 \gets$ Encoder$(X)$ Multi-level features
$Z, q_\theta(z|x) \gets $ LocalSpace$(X'_1, X'_2, X'_3, X'_4)$
$p_\gamma(z|x,y) \gets$ PostNet$(X,Y)$
$G, q_\theta(g|x) \gets $ GlobalSpace$(X'_1, X'_2, X'_3, X'_4)$
$\ell_z \gets \kl[\big]{q_\theta(z \given x)}{p_\gamma(z \given x, y)}$ Eq. <ref>
$\ell_g \gets \kl[\big]{q_\theta(g \given x)}{p(g)}$ Eq. <ref>
$X''_1, X''_2, X''_3, X''_4 \gets $ Decoder$(Z)$ Multi-level features
$Y' \gets$ convolution of $[X''_1, X''_2, X''_3, X''_4, G]$
$\ell_s \gets \text{CrossEntropy}(Y', Y) + \text{IoU}_{\text{soft}}(Y', Y)$ Eq. <ref>
$\mathcal{L} = \lambda_g \ell_g + \lambda_z \ell_z + \lambda_s \ell_s$ Eq. <ref>

Update parameters according to gradients
$\theta_{\text{Enc}} \stackrel{+}\leftarrow - \nabla_{\theta_{\text{Enc}}}(\ell_z + \ell_g + \ell_s)$
$\phi_{\text{Dec}} \stackrel{+}\leftarrow - \nabla_{\phi_{\text{Dec}}}(\ell_g + \ell_s)$
$\gamma_{\text{PostNet}} \stackrel{+}\leftarrow - \nabla_{\gamma_{\text{PostNet}}}(\ell_g + \ell_z)$
static stopping criterion (epoch = 100) Number stactic of epochs

§.§ Setup

The ResNet inspires PHGMM's architecture because it presents an extensive feature extraction process in the encoding stage.
The encoder has four blocks of depths $256$, $512$, $1024$, and $2048$.
The convolution, atrous convolution operations, and ReLU activation functions are performed within each block.
Because of our limited computational resources, we reduce our input image by $\frac{1}{4}$.
We use pooling and convolution with a stride size of two to achieve it.
To produce the parameters of the latent spaces, we concatenate the output of each block.
Thus, we use convolution and a Sigmoid function to adjust the depth (, number of classes).
Then, by mean reduction over the height and width, we obtain a feature vector of each block.
We concatenate all of them and pass these through a Fully Connected layer (FC) producing our parameters ($\mu_{*}$ or $\sigma_{*}$) in size $1\times1\times256$ (empirically chosen).
Note, we use different parameters (weights) in FC to infer each cluster $k$ (, $\mu^k_z$, $\sigma^k_z$).
The same configuration is also applied for PostNet.

In the decoder, we concatenate the $\mu^k_z$ of every $K$ clusters.
Then, we use FC to bring our GMM vector (from $z$) from 1D to 2D and feed our decoder with it.
The decoding stage presents operations similar to the encoder adding a bilinear interpolation operation.
Moreover, it also has four blocks, all with depth equal to the number of classes.
Note that in each block, the PHGMM model retrieves the geometric information (background and borders of the objects) by merging the local-context features $z$ with the convolution operations, upscale, and the features coming from the encoder through skip connections.
Finally, we add a vector of specific features from $\mu_g$ (passed through FC operations and bilinear interpolation) to the decoder output.
In the end, we perform two last convolution operations followed by the softmax function.
Furthermore, we set our hyperparameters $\lambda_{g, z, s}$ to $1.1$, $0.4$, and $0.4$, respectively.
Finally, we use a static stopping criterion when training our model. We define a static number of epochs (one hundred epochs) to stop the training.

§ EXPERIMENTS

In this section, we introduced datasets for SS used in the training and testing of our PHGMM model and the evaluation metrics used to compare the different existing models.
We use the dataset with reduced classes proposed by Valada  <cit.>.
It is due to our limited computational resources.
To make the comparison fair (see Table <ref> and <ref>), we train all models from scratch and each model with the same number of classes and same resolution, and use their respective hyperparameters from each model.

group style=
group size=3 by 1,
horizontal sep=1.1cm,
vertical sep=15pt,
xlabels at=edge bottom,
x descriptions at=edge bottom,
minimal plot grid,
cycle list/Dark2,
cycle multiindex* list=
mark listDark2linestyles,
/tikz/mark options=solid,
/tikz/mark repeat=1,
/tikz/mark phase=1,
major grid style=dashed,
xtick=2, 4, 8, 12, 16,
x tick label style=/pgf/number format/precision=0,
y tick label style=/pgf/number format/precision=0,
ylabel near ticks,
xlabel near ticks,
legend pos=outer north center,
legend columns=2,
legend style=
table[x=k, y=z, col sep=comma, header=true]img/ablation_study_IoU_z.csv;
table[x=k, y=zg, col sep=comma, header=true]img/ablation_study_IoU_z_g.csv;

xlabel=Number of cluster $k$ in $z$,
table[x=k, y=z, col sep=comma, header=true]img/ablation_study_Prec_z.csv;
table[x=k, y=zg, col sep=comma, header=true]img/ablation_study_Prec_z_g.csv;

table[x=k, y=z, col sep=comma, header=true]img/ablation_study_Rec_z.csv;
table[x=k, y=zg, col sep=comma, header=true]img/ablation_study_Rec_z_g.csv;

[Ablative study on $z$ and $g$]Ablation study on the latent spaces $z$ (number of clusters $k$ in the GMM) and $g$ (include or exclude from the model) using the Cityscape validation dataset.

group style=
group size=1 by 2,
xlabels at=edge bottom,
ylabels at=edge left,
x descriptions at=edge bottom,
y descriptions at=edge left,
horizontal sep=1.0cm,
vertical sep=45pt,
minimal plot grid,
cycle list/Dark2,
cycle multiindex* list=
mark listDark2linestyles,
/tikz/mark options=solid,
/tikz/mark repeat=3,
/tikz/mark phase=2,
major grid style=dashed,
ylabel=Pixelwise Class. Error ($\%$),
xlabel=Trimap Width (in pixels),
xtick=1, 10, 20, 30,
x tick label style=/pgf/number format/precision=0,
y tick label style=/pgf/number format/precision=0,
ylabel near ticks,
xlabel near ticks,
legend columns=3,
legend style=

ytick=25, 36, 47, 58,
yticklabels=25, 36, 47, 58,
legend to name=legend2,
xlabel style=align=center,
xlabel=Trimap Width (in pixels)
(c1) at (rel axis cs:1, 1);

table[x=width, y=trimap, col sep=comma, header=true]img/cityscape_adapnet_pp.csv;

table[x=width, y=trimap, col sep=comma, header=true]img/cityscape_ocnet.csv;

table[x=width, y=trimap, col sep=comma, header=true]img/cityscape_dunet.csv;

table[x=width, y=trimap, col sep=comma, header=true]img/cityscape_hrnet.csv;

table[x=width, y=trimap, col sep=comma, header=true]img/cityscape_hrnet+ocr.csv;

table[x=width, y=trimap, col sep=comma, header=true]img/cityscape_phgmm.csv;

yticklabels=21, 31, 40, 49, 58,
ytick=21, 31, 40, 49, 58,
legend to name=legend,
xlabel style=align=center,
xlabel=Trimap Width (in pixels)
table[x=width, y=trimap, col sep=comma, header=true]img/synthia_adapnet_pp.csv;

table[x=width, y=trimap, col sep=comma, header=true]img/synthia_ocnet.csv;

table[x=width, y=trimap, col sep=comma, header=true]img/synthia_dunet.csv;

table[x=width, y=trimap, col sep=comma, header=true]img/synthia_hrnet.csv;

table[x=width, y=trimap, col sep=comma, header=true]img/synthia_hrnet+ocr.csv;

table[x=width, y=trimap, col sep=comma, header=true]img/synthia_phgmm.csv;

[above] at ($(c1.north)+(-80pt, 20pt)$) legend;

[Pixelwise classification error trimap width]
Pixelwise classification error (lower is better) trimap width (to measure the pixel boundary labeling accuracy) for AdapNet++ <cit.>, OCNet <cit.>, DUNet <cit.>, HRNet <cit.>, HRNet+OCR <cit.>, and our PHGMM on the validation datasets.

§.§ Datasets

We evaluated several types of urban/forest scenarios datasets. They are Cityscapes <cit.> and Synthia <cit.>.

Cityscapes: The dataset has 5000 samples.
However, we used crop for augmenting the training set (, transformations of contrast, brightness, and horizontal flip) to generate 17500 samples.
For comparison with other models,  we employ the original validation set with a resolution of $768\times384$ (resize).
To facilitate the models comparison, AdapNet++ <cit.> proposed the class reduction produced by combining some types of classes.
Thus, we use 11 classes: sky, building, road, sidewalk, fence, vegetation, pole, car/truck/bus, traffic sign, person, rider/bicycle/motorbike, and background.

Synthia: This dataset contains realistic photo images from a virtual city. We use the original 9000 samples (7000 for training and 2000 testing) resized to $768\times384$ resolution.
The classes of object labels are the same as the Cityscapes mentioned above label set.

§.§ Evaluation Metrics

To evaluate our results on segmentation, we chose accuracy and intersection-over-union metrics as validation measures <cit.>.
The intersection-over-union (IoU) is defined by

\begin{equation}
\label{eq:metric_iou}
\text{IoU} = \sum_{i}^N \frac{P_i \cap Y_i}{P_i \cup Y_i} = \sum_{i}^N \frac{\mathit{TP}_i}{\mathit{TP}_i + \mathit{FP}_i + \mathit{FN}_i},
\end{equation}

the precision (Prec) is

\begin{equation}
\label{eq:metric_prec}
\text{Prec} = \sum_{i}^N \frac{\mathit{TP}_i}{\mathit{TP}_i + \mathit{FP}_i},
\end{equation}

and the recall (Rec) is

\begin{equation}
\label{eq:metric_rec}
\text{Rec} = \sum_{i}^N \frac{\mathit{TP}_i}{\mathit{TP}_i + \mathit{FN}_i}.
\end{equation}

We assume that $P_i$ is the set of pixels predicted as the $i$th class, $Y_i$ is pixels set belonging to the $i$th class, and $N$ is the number of classes.
Besides, $\mathit{TP}_i$, $\mathit{FP}_i$, $\mathit{TN}_i$, and $\mathit{FN}_i$ represent True/False Positives and True/False Negatives, respectively, for a given class $i$.
Note, these metrics are widely used in SS

We use clustering metrics to measure the latent space behavior.
Consequently, we utilize the Silhouette Coefficient, SSI <cit.> defined by

\begin{equation}
\label{eq:metric_ssi}
\text{SSI} = \sum_{i}^N \frac{b_i - a_i}{\max\{a_i, b_i\}},
\end{equation}

where $a_i$, $b_i$ are the mean intra-cluster and nearest-cluster distances from $i$, respectively.
The Calinski-Harabasz Index, CHI <cit.> is given by

\begin{equation}
\label{eq:metric_chi}
\text{CHI} = \frac{\text{SS}_M}{\text{SS}_W} \frac{N-k}{k-1},
\end{equation}

where $k$ is the number of clusters, and $N$ is the samples, $\text{SS}_W$ and $\text{SS}_M$ are the overall within-cluster and between-cluster variances, respectively.
Finally, the Davies-Bouldin Index, DBI <cit.> denoted by

\begin{equation}
\label{eq:metric_dbi}
\text{DBI} = \frac{1}{k}\sum_i^k \max_{j \ne i}\left(\frac{s_i + s_j}{d_{ij}}\right),
\end{equation}

where $s_i$ is the average distance between each point of cluster $i$ and its centroid, and $d_{ij}$ is the distance between cluster centroids $i$ and $j$.
Note, high values are better for SSI and CHI, while low ones are better for DBI.

§.§ Comparisons

6*-5.7[0pt][0pt][origin=c]90Cityscapes dataset

6*-5.7[0pt][0pt][origin=c]90Synthia dataset

Qualitative comparison of the predictions of the AdapNet++ <cit.>, OCNet <cit.>, DUNet <cit.>, HRNet <cit.>, HRNet+OCR <cit.>, and PHGMM models against the ground-truth.
The samples are chosen randomly from the validation set.
The prediction rows (output) show the outputs of the different models with their respective labels (colors) of the datasets.
The comparison row shows an overlap of the predictions (PHGMM vs. the model indicated in the column) against the ground truth.
The first column is PHGMM vs. AdapNet++; the second one is PHGMM vs. OCNet++, and so on.
In the comparison images, the green regions are correctly segmented, the red regions are erroneously segmented by original models (AdapNet++, OCNet, DUNet, HRNet, and HRNet+OCR) while the blue ones are errors made by PHGMM.
Regions incorrectly segmented by both predictions are purple.

Iteration $=10^2$
Iteration $=10^3$
Iteration $=10^4$
Iteration $=10^5$

SSI $=-0.092$
SSI $=-0.041$
SSI $=0.095$
SSI $=0.703$

CHI $=127.47$
CHI $=308.57$
CHI $=609.23$
CHI $=12074.30$

DBI $=6.951$
DBI $=5.144$
DBI $=3.050$
DBI $=0.454$

[Plot local latent space]We show latent space clustering behavior for $z$, using the Cityscape validation dataset, using $11$ classes and with crop size of $384 \times 768$. The classes are: cs-sky1 Sky, cs-build1 Building, cs-road1 Road, cs-sidewalk1 Sidewalk, cs-fence1 Fence, cs-vege1 Vegetation, cs-pole1 Pole, cs-car1 Car, cs-sign1 Sign, cs-person1 Person, cs-cyclist1 Cyclist.
It through the different iterations measured by Silhouette Coefficient, SSI (higher is better), Calinski-Harabasz Index, CHI (higher is better), and Davies-Bouldin Index, DBI (lower is better).

We present a set of experiments (quantitative and qualitative) to compare our PHGMM model with existing models in the literature.
For these, we employed Cityscape and Synthia datasets presented in the previous section.
We present the quantitative results in Tables <ref> and <ref>.
We use the IoU metric (higher is better) for each class on both datasets.
Note that to perform the comparison, we train all the models from scratch, with the same sample size and the same number of classes.
Due to our limited computational resources, we use the sample size (, $768 \times 384$) and the number of classes (, $11$ target classes) defined in previous works by Valada  <cit.>.

We present the qualitative results in Fig. <ref>.
The results are displayed on a set of random samples from the validation set.
These figures show the inferences of the models with the best segmentation performance (, IoU).
We present the predictions of the different models named in the columns.
These predictions show each dataset's respective color (label of classes) for Cityscape and Synthia
and are displayed in the odd rows named output.
The even rows named comparison exhibit the comparison (overlap) of our PHGMM model against the different models (AdapNet++, OCNet, DUNet, HRNet, and HRNet+OCR).
In the comparison image (, the first column is PHGMM vs. AdapNet++ second one is PHGMM vs. OCNet++), blue represents regions that the PHGMM model erroneously segments, and red represents those erroneously segmented by the several models (name of the columns).
The regions correctly and incorrectly segmented by both predictions are green and purple, respectively.
Finally, in the last comparison column, we present our model against itself, , PHGMM vs. PHGMM.
Here, we interpret the purple region as showing the incorrectly segmented regions of our PHGMM model.
Note that our results decrease spatial precision loss at the boundaries compared to the other models (quantitative improvement in Fig. <ref>).

Remember, we conducted this research to address the loss of spatial precision in segmentation.
We use Trimap <cit.>, which focuses on measuring the pixel-wise error on boundary regions of segmentation, with a distance to the boundaries given by trimap width (pixels).
The plot in Fig. <ref> (error curve comparison) demonstrates that our model has less loss of spatial precision compared to the literature (, HRNet + OCR).

In addition, we present an ablation study (Fig. <ref>) on the latent space $z$ (number of clusters $K$ used in the GMM) and latent space $g$ (including or excluding it from the architecture).
While it may be interesting to also evaluate the global features alone, we did not perform this experiment due to the low performance that these features may produce given their location on the decoder (Fig. <ref>).
However, from these plots, we can observe that our best results are produced when using the learning representation $z$ (local features) with the number of clusters $k$ equal to the number of classes $c$ in the dataset (, $ c = k = 12$).
Thus, we intuit that, in an unsupervised way, each cluster of $z$ (, GMM) is modeled to obtain the best representation of local features focused on each class.
Furthermore, our ablation study also shows that the latent space $g$ is crucial to improving the final segmentation.
The combination of features ($z+g$) produces the best predictions.

Finally, we visualize the behavior of latent space $z$ (Fig. <ref>).
A Mixture of Gaussian fits $z$ through the different iterations using the multidimensional projection method t-SNE <cit.>.
Additionally, we show the influence of $z$ in the prediction of the segmentation.
The visualization, obtained with t-SNE <cit.>, shows that the segmentation results are improved by providing clustering behavior to the latent space $z$.

label distance=.5,
label style/.append style=text width=.65cm, /kiviat/set anchor, anchor=, align=,
set anchor/.code=360/@kiv@radial*> 180 ? 1:0>0 westleft eastright,
[local bounding box=train,shift=(0,0)]

radial  style/.style =-,
lattice style/.style =black!30,
AdapNet++, OCNet, DUNet, HRNet, HRNet + OCR, PHGMM

/kiviatline/zero point=,

[unity=, label precision=0](0)

[local bounding box=test,shift=(32,0)]

radial  style/.style =-,
lattice style/.style =black!30,
AdapNet++, OCNet, DUNet, HRNet, HRNet + OCR, PHGMM

/kiviatline/zero point=,

[unity=, label precision=2, zero point=](0)

[local bounding box=train,shift=(0,27)]

radial  style/.style =-,
lattice style/.style =black!30,
AdapNet++, OCNet, DUNet, HRNet, HRNet + OCR, PHGMM

/kiviatline/zero point=,

[unity=, label precision=0](0)

[local bounding box=test,shift=(32,27)]

radial  style/.style =-,
lattice style/.style =black!30,
AdapNet++, OCNet, DUNet, HRNet, HRNet + OCR, PHGMM

/kiviatline/zero point=,

[unity=, label precision=2, zero point=](0)

[above] at (test.north) By Sample;
[above] at (train.north) By Dataset;

[above=8.5pt of $(train.north)!.5!(test.north)$, font=, ampersand replacement=&]
[plot1,scale=0.5,opacity=.8] plot coordinates (0,0.25)(1,0.25) – cycle; & [anchor=base]Trainig; &
[plot2,scale=0.5,opacity=.8] plot coordinates (0,0.25)(1,0.25) – cycle; & [anchor=base]Validation;

[Time execution]
Comparison of the execution time of semantic segmentation models for the entire dataset (left) and per sample (right), both in minutes. We use the Cityscapes dataset with 17500 and 500 samples for training and validation, respectively.
Note that the PostNet network is only used in the training phase.
Our PHGMM model in the testing phase presents an execution time comparable to the state-of-the-art.

Also, we present an efficiency comparison of the SS models (see Fig. <ref>) for the entire dataset and per sample.
We used the Cityscapes dataset with 17500 training samples and 500 validation samples (in our case, they are the validation samples) for these experiments.
The results show the execution time of one epoch (a forward-pass over the entire dataset) in a single GPU.
We executed the process five times and reported the averages of the entire training and validation dataset (left) and per sample (right), both in minutes.
Our plots on training (Fig. <ref>) show an increase in the time required to train the PHGMM model.
This increment is directly related to the use of the PostNet network and the optimization algorithm used to fit the weights in the training phase.
Note that the PostNet proved robust enough to fit a Gaussian mixture model structure in the latent space.
Remember, we do not use PostNet for the validation dataset, significantly reducing execution time.
Finally, the previous experiments have been conducted on four Nvidia GTX Titan X GPU 12 graphics cards,
1 of RAM, and 56 Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40 (multi-GPU).
Also, we use Python3, Numpy, Pillow, Tensorflow $1.8$, Docker, and Ubuntu operating system.

Our previous experiments show that the SS task can be improved by endowing/providing a structure to the latent space $z$ (, learning representation) in the form of a Gaussian mixture.
In order to take advantage of this rich feature extraction with structure and behavior clustering, we intelligently merge these features with encoder skip connections at the decoding stage.
We are achieving in this way to recover the geometric information of the segmented objects (, background and boundary of the objects).
In addition, it is necessary to add complementary information (latent space $t$) focused on the contour of the objects, to improve the boundary of the segmentation (see Fig. <ref>).
Finally, we address the loss of spatial precision due to the combination of three factors:
the PHGMM architecture (merge features in the decoder stage),
the extraction of specific features given by the latent space $t$, and
the internal structure (Gaussian mixture model) imposed on the latent space $z$.

§ FUTURE WORKS

Current models extract the context (generally global) of the objects in the images and delimit the object regions.
However, small regions on the edges of these objects demonstrate spatial difficulties and are lost (see Fig. <ref>).
This loss in the edges of objects can be caused by noise, blurring, smoothing, and image resolution low, among others.
This work addresses the loss of spatial precision problem in semantic segmentation.
As described in previous sections, the loss of information is commonly reflected in the edge of segmented objects.

Our PHGMM model works with local and global contexts presenting improved results at the edges of segmented objects.
This improvement in the results has a direct correlation with the provided (pushed) clustering behavior in the latent space, see Figs. <ref> and <ref>.
In addition, this correlation is influenced by the number of clusters used in the clustering algorithm in the
training stage, shown in Fig. <ref>.
The following research step would be to increase the number of clusters through hierarchical clustering.
In this way, we would group objects semantically at a higher level, and break them down into other sub-classes down in the hierarchy.
Also, to accomplish this model idea, we would need a larger number of classes.

§ CONCLUSION

In this work, we show that by endowing the latent spaces ($z$ and $g$) with clustering behavior and providing them with a structural representation (, GMM for $z$ and Normal distribution for $g$), we improved the results of the SS task.
This improvement is mainly due to the combination (produced in the decoding stage) of low-level features (from global-context information in $g$) and high-level ones (from local-context information in $z$).
The combination of features is performed in the encoder stage with the latent spaces $z$, $g$, and the multi-scale features from the encoder in order to recover the geometric information.
Furthermore, our comparative results show that more details can be extracted from PHGMM by setting the number of clusters equal to the number of classes.
The improvements are produced at the segmented objects' boundaries, thus addressing the loss of spatial precision problem.

[]Darwin Saire received his B.Eng. degree in Computer Engineering from Universidad de San Agustin (UNSA), Arequipa in 2013.  He completed his M.Sc. degree in Computer Engineering from University of Campinas, Brazil in 2017.  He is currently a Ph.D. candidate at the Institute of Computing, University of Campinas, Brazil.  His research interests are computer vision, pattern recognition, image processing, machine learning and deep learning.

[]Adín Ramírez Rivera (S'12, M'14, SM'21) received his B.Eng. degree in Computer Engineering from Universidad de San Carlos de Guatemala (USAC), Guatemala in 2009.  He completed his M.Sc. and Ph.D. degrees in Computer Engineering from Kyung Hee University, South Korea in 2013.  He is currently an Associate Professor at the Department of Informatics, University of Oslo, Norway.  His research interests are video understanding (including video classification, semantic segmentation, spatiotemporal feature modeling, and generation), and understanding and creating complex feature spaces.