11institutetext: Eindhoven University of Technology, Eindhoven, The
Netherlands 11email: {b.osman<EMAIL_ADDRESS>
# Voxlines: Streamline Transparency through Voxelization and View-Dependent
Line Orders

Besm Osman  Mestiez Pereira  Huub van de Wetering  Maxime Chamberland

###### Abstract

As tractography datasets continue to grow in size, there is a need for
improved visualization methods that can capture structural patterns occurring
in large tractography datasets. Transparency is an increasingly important
aspect of finding these patterns in large datasets but is inaccessible to
tractography due to performance limitations. In this paper, we propose a
rendering method that achieves performant rendering of transparent
streamlines, allowing for exploration of deeper brain structures
interactively. The method achieves this through a novel approximate order-
independent transparency method that utilizes voxelization and caching view-
dependent line orders per voxel. We compare our transparency method with
existing tractography visualization software in terms of performance and the
ability to capture deeper structures in the dataset.

###### Keywords:

Tractography Visualization Transparency Streamlines

##  1 Introduction

Tractography datasets are growing larger, posing challenges for their
visualization in terms of performance and usability. Various methods have been
developed to address performance issues, such as removing colinear points or
compressing the dataset[5]. Existing tractography visualization tools like
MRtrix3 [9] and TrackVis [11] provide filtering options for visualizing
different parts of the dataset. Recently, transparency-based methods have
emerged to improve usability, such as applying varying transparency to each
fiber based on orientation to highlight underlying tissue configurations [8].
Transparency can play an important role in tractography visualization by
enabling exploration of deeper brain structures. Transparency has been used to
better convey the spatial relationship between streamlines and surfaces that
illustrate their anatomical context. [7] However, existing tractography
visualization software, utilize subpar transparency methods due to performance
constraints [4], limiting the benefits gained from transparency.

###  1.1 Existing Transparency Methods

The basic approach to achieving transparency is by rendering the objects
furthest from the screen first, which requires sorting every translucent
object whenever the view changes. This method is suitable for visualization
contexts with a limited number of transparent objects or a fixed viewing
direction. However, this solution is not feasible for interactive tractography
visualization. It is not possible to determine a consistent sorting order for
entire streamlines, since different parts of a streamline can be at varying
distances from the view. Therefore, sorting needs to be done at the level of
individual line segments that make up the streamlines. The scale of the
tractography dataset makes the computational cost of real-time sorting of line
segments impractical. Hence, order-independent methods are required to achieve
tractography visualizations with transparency. These techniques eliminate the
need for ordering translucent objects. Various order-independent transparency
techniques have been researched for 3D line sets, including depth peeling [1],
multi-layer alpha blending [6], and raycast techniques [2]. In a recent
comprehensive study [3], different order-independent transparency techniques
for 3D linesets were compared, each showing different advantages and
drawbacks. Some techniques require multiple render passes, resulting in
decreased performance. Other techniques rely on complex render pipelines
supported only by newer hardware or have high preprocessing times.

##  2 Method Overview

We propose a novel method for voxel-based streamline rendering that achieves
fast, approximate order-independent transparency without relying on modern
render pipelines and minimal preprocessing time. The first part of this method
involves splitting the dataset into voxels, where each voxel stores streamline
segments that pass through it as described in sections 2.1 and 2.2. Next, a
mesh is generated for each voxel that connects the streamlines segments and
renders the streamlines explained in Section 2.3. This mesh generation is
performed once when reading the dataset. During rendering, we sort the voxels
from back to front every frame, solving the most noticeable transparency
issues. Furthermore, we extend this method by storing a set of precomputed
line segment render orders per voxel and selecting the closest ordering based
on the current view as described in Section 2.6 to improve the transparency
accuracy within a voxel.

###  2.1 Voxelization

Figure 1: This figure shows a 2D representation of the voxel meshes. Each
square in the figure corresponds to a voxel. The points and lines in the
figure are colored according to their respective mesh. Some points are used
twice to connect voxels, and to illustrate this these points are shown with
both colors.

The algorithm takes a set of streamlines as input, where each streamline
consists of a sequence of points. Our goal is to divide the dataset into
voxels so that each voxel contains all line segments passing through it. In
this paper, we define a voxel as a 3D cube with a fixed position and size. The
size of the voxels used in this method differs from voxel sizes used in MRI
scans or tractography algorithms. The voxels used in our method are an order
of magnitude larger than the voxel sizes used in MRI scans to encompass more
streamline points per voxel. To obtain the voxel coordinate of a streamline
point $p$, we first calculate its position relative to the dataset by
subtracting the minimum bound $\boldsymbol{B_{min}}$, which is defined as the
smallest coordinates contained in the dataset:
$\boldsymbol{p_{rel}}=\boldsymbol{p}-\boldsymbol{B_{min}}$. This ensures that
each voxel has non-negative coordinates. The voxel coordinates
$\boldsymbol{v}$ are then obtained by dividing $\boldsymbol{p_{rel}}$ by the
voxel size $\boldsymbol{s}$ and taking the floor of the resulting vector:
$\boldsymbol{v}=\lfloor\boldsymbol{p_{rel}}/\boldsymbol{s}\rfloor$.

###  2.2 Generating Voxlines

Intuitively, grouping points of a streamline based on which voxel the points
fall in, can be seen as dividing the streamline points into several parts when
a streamline passes through multiple voxels. This can be seen in Figure 1
where each streamline is split based on the voxel coordinate of each
streamline point. In the rest of the paper, we will refer to these parts as
”voxlines”. A voxline is defined as a sequence of consecutive points of a
streamline that is bounded by a voxel. In this context, consecutive means that
there are no missing points between the minimum and maximum points of a group
of streamline points. For example, points with streamline indices
$\\{6,7,8,9\\}$ would be considered consecutive, while $\\{3,4,12,13\\}$ would
not.

To generate voxlines making up the streamlines we can split each streamline in
voxlines based on the voxel each point falls within. However, using only the
points in each voxline would result in gaps between voxels when rendering the
voxlines. Consider a streamline that is separated into two voxlines. If we
render these voxlines as two distinct sets of lines, there will be a gap
between them because no line is drawn between the end of the first voxline and
the beginning of the second voxline. So defining the voxlines by using only
the points that fall within the voxel would cause lines that cross the voxel
borders to not be rendered. To address this issue, when generating the
voxlines we add an additional point for every voxline that does not contain
the final streamline point. This additional vertex corresponds to the next
point in the streamline that falls outside of the voxel. By doing this, we
fill all the gaps between the voxlines. This is illustrated in Figure 1 by
certain voxlines having an extra point outside of their voxel boundary and are
thus part of both voxel meshes.

Data: Input data consist of a sequence $streamlines$ where each element
consists of a sequence of points, bound minimum $B_{min}$ of the whole dataset
and voxel size $s$ blue.

$voxelset=\emptyset$ //Empty hashmap with voxel coordinates as keys and sets
of voxlines as values

forall  _streamlines $S=<p_{0},p_{1},...p_{n-1}>$_ do

$v_{p}=null,i=0,voxline=<>$

while  _$i <n$_ do

$v_{i}=\lfloor(p_{i}-B_{min})/s\rfloor$

if  _$v_{p}\neq v_{i}$_ then

if  _$i\neq n-1$_ then

$voxline\leftarrow voxline+p_{i}$

if  _ $voxelset\textup{{ contains key }}v_{p}$_ then

$voxelset[v_{p}]\leftarrow voxelset[v_{p}]\cup\\{voxline\\}$

else

$voxelset[v_{p}]\leftarrow\\{voxline\\}$

$v_{p}=v_{i},voxline=\emptyset$

$voxline\leftarrow voxline+p_{i}$

$i=i+1$

if  _$voxline\neq\emptyset$_ then

if  _ $voxelset\textup{{ contains key }}v_{p}$_ then

$voxelset[v_{p}]\leftarrow voxelset[v_{p}]\cup\\{voxline\\}$

else

$voxelset[v_{p}]\leftarrow\\{voxline\\}$

Result: Set of voxlines grouped by voxel coordinates.

Algorithm 1 Voxline algorithm. Generates voxlines grouped by voxel coordinates
from the input streamlines.

Algorithm 1 implements the voxelization and voxeline generation described in
Sections 2.1 and 2.2. It generates the set of voxlines and groups them based
on voxel coordinates by iterating over the streamlines. Each streamline is
split into voxlines whenever a point is found with a different voxel
coordinate than the previous point. Voxlines that do not contain the final
point have an additional point added to ensure proper connectivity between
voxlines.

###  2.3 Mesh generation

We generate a mesh per voxel containing all voxlines part of the voxel, which
can be rendered using the OpenGL ’lines’ primitive. A mesh consists of
vertices, each with a position and indices that define the render order of the
lines. The mesh vertices are simply defined by the points of each voxline that
fall within the voxel. We define the indices by concatenating each pair of
consecutive voxline points within a voxel. The order of these pairs in the
indices determines the rendering order of the line segments within a voxel.
For now, we define this order simply based on the dataset order. Later, we
will describe a more sophisticated view-dependent order in Section 2.6. Note
that this method focuses solely on vertex positions and indices to determine
the render order of the lines. Additional vertex information can be stored for
cosmetic purposes, such as line identifiers, relative point indices, line
tangents, and render flags. However, this additional data is not relevant for
the render method as we are only concerned with vertex positions and the
render order for improved transparency.

###  2.4 Render Order Accuracy

Transparency issues become most apparent when distant line segments are
rendered in the incorrect order, specifically when line segments closer to the
view are drawn before line segments further away. To address this, we divided
the dataset into voxels, with each voxel representing the line segments
passing through its bounds. Additional line segments were included to connect
different voxel meshes. Prior to rendering the voxel, we sort them from back
to front based on the current view. This sorting provides guarantees on the
render order between any two line segments when the distance between any two
consecutive streamline points is smaller than the voxel size. Line segments
that are fully contained within different voxel bounds will be rendered in the
correct back-to-front order. This holds true for the majority of line
segments. Line segments that cross voxel bounds may have inaccurate render
order relative to their immediate neighboring voxels. However, they are
guaranteed to be in the correct order relative to line segments in non-
neighboring voxels. Since we have not specifically ordered the lines within a
voxel, the render order of line segments within a voxel will be inaccurate.
Therefore, voxelization ensures that any two line segments have an accurate
render order unless they are within the same voxel or one voxel apart for line
segments crossing a voxel boundary, effectively resolving the most significant
render order issues.

###  2.5 Improved Transparency within Voxels

The primary issue remaining with the render order, and consequently
transparency, is the inaccurate rendering order within a voxel. We can take
advantage of the fact that voxelization divides the dataset into smaller
datasets with similar properties to the original streamline data. Since both
the original streamline dataset and each voxel consist of sets of lines
composed of consecutive points, we can apply existing rendering techniques
that approximate transparency for 3D line sets on a voxel level. Furthermore,
sorting within voxels becomes more feasible because each voxel contains a
smaller subset of the points. For this method, we have developed a solution
that stores different streamline orders per voxel to improve transparency.

###  2.6 View-Dependent Line Order per Axis

To achieve more accurate transparency within voxels, we can sort the line
segments based on the average position of the two endpoints and the current
viewing direction. However, performing this sorting process every time the
view changes would be computationally expensive, even after voxelization.
Instead, we precompute line render orders for a set of viewing directions.
When rendering, we select the precomputed ”closest” viewing direction for
improved transparency. The closest viewing direction is determined by
calculating the distance between each stored sorting direction and the current
viewing direction using the formula $\boldsymbol{s\cdot\frac{v-c}{||v-c||}}$,
where $\boldsymbol{s}$ represents the sorting direction, $\boldsymbol{v}$ is
the voxel position, and $\boldsymbol{c}$ is the camera position.

This method is similar to imposter rendering, a technique that precomputes
textures based on specific camera angles and then renders billboards instead
of meshes for improved performance. However, instead of precomputing textures,
we precompute line orders for a set of view directions. For the set of view
directions, we use the six vectors defined by positive and negative unit
vectors along each axis ($X$, $Y$, and $Z$). These six vectors are
particularly well-suited as they are faster to sort than others, requiring no
squared distance calculations. Additionally, these axis viewing directions
align with axial, sagittal, and coronal projections, commonly used in
tractography tools.

To sort the line segments for an arbitrary direction $\boldsymbol{d}$, we can
compare $\boldsymbol{\frac{(p_{0}+p_{1})}{2}\cdot d}$ for each line segment,
where $\boldsymbol{p_{0}}$ and $\boldsymbol{p_{1}}$ represent the two points
making up the line segment. Since we only use unit vectors for
$\boldsymbol{d}$, we can simplify the comparison to a single vector component,
comparing the non-zero component of $\boldsymbol{d}$. Additionally, we
optimize the sorting by using the sum of the two line positions instead of the
average position. For example, sorting line segments for the direction
$(0,1,0)$ would involve comparing the $Y$ component of
$\boldsymbol{p_{0}+p_{1}}$ for each line segment in the voxel. To find the
orders of the negative unit vectors, we simply reverse the orders found for
the positive unit vectors.

###  2.7 Evaluation

To evaluate our proposed approach, we generated a set of whole-brain
tractograms from a single participant sourced from the Human Connectome
Project [10]. A tractogram consisting of one million streamlines was
constructed using multi-shell multi-tissue constrained spherical deconvolution
in MRtrix.

Figure 2: This figure shows two different angles of a dataset consisting of
one million streamlines rendered using our method with voxelization and axis
view-dependent line orders. Each image is divided into two halves, with the
left half rendered with 0.5% streamline opacity, and the right half rendered
without transparency.

Subsequently, a smaller dataset was created using TractSeg [12], resulting in
a tractogram with 140k streamlines. We implemented our render method in
NeuroTrace (gitlab.com/Besm/NeuroTrace), a tractography visualization tool
developed by the authors. We compared the visual and performance results of
our method with two popular tractography visualization tools that support
transparency, namely MRtrix and TrackVis. We compare them to our method with
only voxelization (basic) and the extension with axis based view-dependent
line orders (axis). All results use voxel size $10$mm3, which through
experimentation was found to give a balance between performance and visual
quality for both individual bundles and whole brain tractograms.

##  3 Results

###  3.1 Qualitative

Figure 3 displays different transparency values for each method using a single
dataset. MRTrix handles transparency by blending two renders of the dataset,
one with a depth buffer and another without, based on the transparency value
of the dataset [4]. While this approach preserves line render order, it makes
it challenging to visualize the internal structures of the dataset. TrackVis
renders transparent streamlines based on the given dataset, causing noticeable
render order issues, particularly when streamline transparency is low.

In our method, by utilizing only voxelization (basic), we can observe more
information about the streamlines deeper in the brain with minimal sorting
inaccuracy. This is evident in the figure, for example, green superior
longitudinal fasciculus fibers can be seen behind the red, short superficial
fibers . Increasing transparency increases the visibility of the superior
longitudinal fasciculus fibers without distorting the fact that the short
superficial fibers are in front, while MRTrix and TrackVis have more
difficulty conveying this information.

When applying axis sorting, the render order issues are slightly reduced.
Although it may be challenging to discern in the image, axis method gives a
more accurate representation of the render order compared to basic method. For
instance comparing highlighted regions in Figure 3 shows axis sorting bringing
out certain red superficial fibers in front more clearly compared to the
method without axis sorting.

###  3.2 Quantitative

Table 1 presents the performance comparison of our method with MRTrix and
TrackVis on two tractography datasets: one with 140k streamlines and another
with 1 million streamlines. It is worth noting that some tractography
rendering methods include a preprocessing step that converts the dataset to a
different format, aiming to reduce loading time for subsequent dataset
loading. In our method, we intentionally omitted this preprocessing step to
ensure a fair time comparison with MRTrix and TrackVis. Our implementation
load the same TCK file format as MRTrix.

Figure 3: Figure 3 shows a comparison of transparency between MRTrix, TrackVis, and our method, with just voxelization (basic) and with view-dependent internal voxel order (axis). For each method, we display the entire dataset with $50\%$ transparency value from the sagittal plane and highlight a portion of the data with three opacity values: $50\%$, $5\%$, and $1\%$. Since MRTrix does not display opacity values, three comparable values were selected for the comparison. | Streamlines | Points | Method | Loading time (sec) | Render times (ms)
---|---|---|---|---|---
| 143,999 | 3,770,127 | MRTrix | 0.8 | 14
TrackVis | 19.8 (+2) | 30
our method (basic) | 3.6 | 26
our method (axis) | 6.5 | 34
| 1,000,000 | 56,240,953 | MRTrix | 2.6 | 67
TrackVis | 283 (+26) | 253
our method (basic) | 27 | 124
our method (axis) | 143 | 330
Table 1: Performance comparison between different methods with transparency.
Loading time is measured from selecting the dataset (TCK/TRK file) to first
render of the dataset. Performance is measured on a Windows Laptop with NVIDIA
GeForce GTX 1650 and Intel(R) Core(TM) i7-9750H 2.60GHz CPU. TrackVis has
additional loading time whenever modifying transparency value, which is shown
in parenthesis.

##  4 Discussion

Voxelization and view-dependent line orders have shown promising results. The
main benefits are from the voxelization step. View-dependent line order per
voxel improve visual results at a computational cost, allowing for increased
quality or larger voxel sizes with equivalent visual quality. Although we have
not achieved the same performance as MRtrix, both our rendering and loading
times are usable and faster than TrackVis. We believe that the decreased
performance is caused by our implementation and is unrelated to the method
outlined in this paper. In our implementation loading a dataset without
voxelization has similar performance to loading with voxelization, indicating
suboptimal performance in dataset loading independent of our method. The
tractography visualization tool we used to implement this method, NeuroTrace,
is primarily focused on functionality and experimentation rather than
performance optimization. We aim to improve performance in future work.

Currently, we manually determine the voxel size and whether to generate the
view-dependent render orders. We are exploring the possibility of utilizing
metadata of the dataset, such as step size and total point count, to
automatically determine the parameters of our method. We are experimenting
with varying transparency per streamline to highlight specific parts of the
dataset and utilizing the benefits gained from voxelization in novel ways. We
aim to explore these aspects in future work.

###  4.1 Different sorting orders

Figure 4: This figure illustrates the voxel line ordering accuracy for two
different camera angles. Each voxel is colored according to the absolute value
of the direction for the RGB values. This direction differs per column. The
second column displays the ’optimal sorting direction,’ which is defined as
the normalized direction from the camera to the voxel center. The third column
shows the nearest voxel order direction using axis orderings. The fourth
column shows the nearest voxel sorting direction using $64$ pseudo-random
sorting directions.

Regarding the set of precomputed view direction line orders, we have
investigated two approaches: per-axis sorting and pseudo-random sorting. The
per-axis method used in this work involves generating two orders per axis. For
each voxel, we store line render orders in both ascending and descending order
for each axis. The pseudo-random sorting approach involves sorting points
based on pseudo-random directions that differ for each voxel. Per-axis sorting
offers certain benefits mentioned previously, mainly performance, as axis
directions do not require (squared) distance calculations while sorting on any
other direction does. Additionally, these axis viewing directions align with
axial, sagittal, and coronal projections, which are common viewing angles in
tractography tools. However, we encountered an issue with voxelization and
per-axis method where moving the camera tends to update certain lines of
voxels simultaneously, causing visual jittering when rotating the viewing
direction while using high streamline opacity
(besm.gitlab.io/voxlines/videos). Using the pseudo-random direction mitigates
this issue but has less accurate results for the most common viewing direction
and is more computationally expensive.

In Figure 4, we demonstrate the difference between axis-based orders and
pseudo-random direction orders. We can observe that the axis-based order is
more accurate, when the dataset is viewed from one of the sides. Pseudo-random
sorting performs worse in these cases. However, when considering arbitrary
viewing directions, it outperforms the axis-based approach. Furthermore, when
moving the camera, the pseudo-random method exhibits less visual jittering.
However, the loading time for the pseudo-random method is worse. In future
work, we aim to explore different sorting orders to find a balance between
visual quality and performance.

## Conclusion

In this work, we have proposed and implemented a novel transparency method
that improves tractography visualizations, allowing the capturing of deeper
structures in tractograms. We demonstrate improved transparency and comparable
performance to existing tractography tools. Our method achieves this by
providing a novel approximate transparency, which enhances the visibility of
structures in the deeper regions of the brain compared to existing methods.

## References

  * [1] Everitt, C.: Nvidia corporation: Order-independent transparency (2001), https://developer.download.nvidia.com/assets/gamedev/docs/OrderIndependentTransparency.pdf
  * [2] Kanzler, M., Rautenhaus, M., Westermann, R.: A voxel-based rendering pipeline for large 3d line sets. IEEE Transactions on Visualization and Computer Graphics 25 (01 2018). https://doi.org/10.1109/TVCG.2018.2834372
  * [3] Kern, M., Neuhauser, C., Maack, T., Han, M., Usher, W., Westermann, R.: A comparison of rendering techniques for 3d line sets with transparency. IEEE Transactions on Visualization and Computer Graphics 27(8), 3361–3376 (2021). https://doi.org/10.1109/TVCG.2020.2975795
  * [4] MRtrix3: Add support for visualizing tractography data (Issue #177), {https://github.com/MRtrix3/mrtrix3/issues/177}
  * [5] Rheault, F., Houde, J.C., Descoteaux, M.: Visualization, interaction and tractometry: Dealing with millions of streamlines from diffusion mri tractography. Frontiers in Neuroinformatics 11 (06 2017). https://doi.org/10.3389/fninf.2017.00042
  * [6] Salvi, M., Vaidyanathan, K.: Multi-layer alpha blending. pp. 151–158 (03 2014). https://doi.org/10.1145/2556700.2556705
  * [7] Schultz, T., Sauber, N., Anwander, A., Theisel, H., Seidel, H.P.: Virtual klingler dissection: Putting fibers into context. Computer Graphics Forum 27(3) (2008). https://doi.org/10.1111/j.1467-8659.2008.01243.x
  * [8] Tax, C., Chamberland, M., van Stralen, M., Viergever, M., Whittingstall, K., Fortin, D., Descoteaux, M., Leemans, A.: Seeing more by showing less: Orientation-dependent transparency rendering for fiber tractography visualization. PloS one 10, e0139434 (10 2015). https://doi.org/10.1371/journal.pone.0139434
  * [9] Tournier, J.D., Smith, R., Raffelt, D., Tabbara, R., Dhollander, T., Pietsch, M., Christiaens, D., Jeurissen, B., Yeh, C.H., Connelly, A.: Mrtrix3: A fast, flexible and open software framework for medical image processing and visualisation. NeuroImage 202, 116137 (2019). https://doi.org/https://doi.org/10.1016/j.neuroimage.2019.116137, https://www.sciencedirect.com/science/article/pii/S1053811919307281
  * [10] Van Essen, D.C., Smith, S.M., Barch, D.M., Behrens, T.E., Yacoub, E., Ugurbil, K.: The wu-minn human connectome project: An overview. NeuroImage 80, 62–79 (2013). https://doi.org/https://doi.org/10.1016/j.neuroimage.2013.05.041, https://www.sciencedirect.com/science/article/pii/S1053811913005351, mapping the Connectome
  * [11] Wang, R., Benner, T., Sorensen, A., Wedeen, V.: Diffusion toolkit: A software package for diffusion imaging data processing and tractography. Proc Intl Soc Mag Reson Med 15 (01 2007)
  * [12] Wasserthal, J., Neher, P., Maier-Hein, K.H.: Tractseg - fast and accurate white matter tract segmentation. NeuroImage 183, 239–253 (2018). https://doi.org/https://doi.org/10.1016/j.neuroimage.2018.07.070, https://www.sciencedirect.com/science/article/pii/S1053811918306864


11institutetext: RIKEN Center for Life Science Technologies (CLST), RIKEN,
6-7-3 Minatojima-minamimachi, Chuo-ku, Kobe, Hyogo, 650-0047 Japan
<EMAIL_ADDRESS><EMAIL_ADDRESS>22institutetext: RIKEN CLST-JEOL
Collaboration Center, RIKEN
6-7-3 Minatojima-minamimachi, Chuo-ku, Kobe, Hyogo, 650-0047 Japan
<EMAIL_ADDRESS><EMAIL_ADDRESS>33institutetext: RIKEN Compass to
Healthy Life Research Complex Program, RIKEN Cluster for Science and
Technology Hub, RIKEN,
6-7-1 Minatojima-Minamimachi, Chuo-ku, Kobe, Hyogo, 650-0047 Japan
44institutetext: RIKEN BioResource Center (BRC),
3-1-1, Koyadai,Tsukuba, Ibaraki, 305-0074 Japan
<EMAIL_ADDRESS>55institutetext: Advanced Center for Computing and
Communication (ACCC), RIKEN,
2-1 Hirosawa, Wako, Saitama, 351-0198 Japan
<EMAIL_ADDRESS>
# Development of Semantic Web-based Imaging Database for Biological Morphome

Satoshi Kume 112233 Hiroshi Masuya 4455 Mitsuyo Maeda 22 Mitsuo Suga 22
Yosky Kataoka 112233 Norio Kobayashi 554422

###### Abstract

We introduce the RIKEN Microstructural Imaging Metadatabase, a semantic web-
based imaging database in which image metadata are described using the
Resource Description Framework (RDF) and detailed biological properties
observed in the images can be represented as Linked Open Data. The metadata
are used to develop a large-scale imaging viewer that provides a
straightforward graphical user interface to visualise a large microstructural
tiling image at the gigabyte level. We applied the database to accumulate
comprehensive microstructural imaging data produced by automated scanning
electron microscopy. As a result, we have successfully managed vast numbers of
images and their metadata, including the interpretation of morphological
phenotypes occurring in sub-cellular components and biosamples captured in the
images. We also discuss advanced utilisation of morphological imaging data
that can be promoted by this database.

###### Keywords:

Microscopy ontology, Open Microscopy Environment, Imaging Metadatabase,
Morphome, Morphological phenotype, Scanning electron microscopy

##  1 Introduction

Imaging data are an important fundamental aspect of contemporary life
sciences. They facilitate the understanding of detailed morphological changes
related to cellular functions and various diseases. We focus on
microstructural images obtained using electron microscopes (EM), which provide
detailed morphological information about tissues and cells at a nano-scale
level [1]. Typically, EM images have two significant problems. 1) Because the
target area of an EM is quite small (sub-cellular level: less than a
micrometre square), a wide-range overview (tissue level: millimetre square) is
required to screen phenotypes. 2) Interpreting an EM image requires an
advanced understanding of histology and histopathology.

To solve the first problem, we have developed a technology to obtain
comprehensive microstructural imaging data of biotissues using scanning
electron microscopy (SEM) with an autofocus function (Maeda, M. et al., under
preparation). This imaging technology provides wide-range and high-resolution
images of a sub-millimetre square area in biosamples. This type of
comprehensive imaging is referred to as the morphome analysis, especially
micro-morphomics in order to identify the totality of the microstructural
morphological features and is considered to be an omic research. While such
imaging data are accumulating rapidly as big data, a common procedure for
sharing, integrating and analysing such data has not yet been developed. In
addition, specialised experience is required to interpret morphological
phenotypes in the images. Recently, Williams et al. launched the Image Data
Resource (IDR), a prototype public database and repository for imaging data
[2]. The IDR is the first general repository for image data in life sciences.
Thus, the potential impact of accessing, sharing and referencing imaging data
in experimental results and published research has increased. However, issues
associated with metadata, image formats and vocabulary associated with imaging
experiments remain unresolved. To ensure reproducibility of experimental data
and results, including expert image analysis, a common data publication
framework and a multi-faceted ontology that describes various imaging
experiments and experimental conditions are desired. Moreover, to complement
specialised histology and histopathology experience and realise further
automatic imaging data analysis, such as machine learning, metadata that
describe related experimental conditions and phenotypes should be machine
readable.

Our goal is to generate standardised machine-readable metadata that include
related information, such as experimental conditions and phenotype data, for
life sciences research. Currently, we are constructing an imaging metadata
system for optical and electron microscopy that employs the Resource
Description Framework (RDF). We are also developing an ontology to describe
RDF metadata. Relative to the development of an ontology to describe imaging
metadata, we have previously proposed general microscopy ontology concepts
that involved translating XML-based Open Microscopy Environment (OME) metadata
and the extension of incomplete vocabularies [3]. The RDF ontology provides
vocabularies and semantics to describe metadata, including information about
electron microscopy imaging conditions, biosamples and sample preparation. In
this paper, we describe the proposed RDF-based RIKEN Microstructural Imaging
Metadatabase, which provides imaging metadata integrated with biosample and
phenotype metadata and an imaging viewer that takes advantage of such
metadata.

##  2 Methods

###  2.1 Extension of Microscopy Ontology for Morphomics Data

The OME is an open-source interoperable toolset for biological imaging data to
manage multi-dimensional and heterogeneous optical microscopy imaging data
[4]. Previously, using the latest version of the OME data model, we translated
the XML based OME data model into a base concept for an RDF schema [3]. At
this point, we identified missing concepts in the OME data model and extended
the vocabularies to describe imaging metadata integrated with biosample and
phenotype metadata.

To improve the ability to describe experimental situations, we further
expanded vocabularies in the microscopy ontology. Note that phenotypic
annotations often differ between observation of microscopic images and direct
observation of biosamples because their magniﬁcation levels are different
(i.e. sub-cellular and individual levels). Therefore, in the microscopy
ontology, we enabled separate descriptions of phenotype data obtained from
imaging analysis (i.e. sub-cellular phenotypes) and direct observation of
biosamples (i.e. phenotype data of biosamples obtained from an individual).

In comprehensive microstructural imaging at the sub-tissue level, it is
necessary to manage image data continuously photographed in units of several
thousands of images or huge image data at several gigabyte levels using a
tiling process. Recently, such huge image data have been converted and
visualised using a pyramidal image format called the Deep Zoom Image (DZI)
format. Therefore, we added new vocabularies (image directory, image overlap,
tiling method, tiling map, etc.) to describe the metadata for tiling image
data and DZI visualisation. In addition, the class of the DZI dataset is
related to rdfs:subClassOf of the dataset class, and it has a structure that
can link individual image data.

###  2.2 Microstructural Bioimaging and Image Processing

To prepare a biosample, we initially prepared wide-range ultra-thin (70
$\mathrm{nm}$) tissue sections obtained from a rat liver (600 $\mathrm{\mu m}$
x 500 $\mathrm{\mu m}$). Following the electron staining of the liver
sections, we performed large-scale microstructural imaging at high spatial
resolution (7.87 $\mathrm{nm/pixel}$) covering the entire area of the section
using SEM. As a result, 288 16-bit images (5120 x 3840 pixels) were obtained,
and the amount of data was approximately 10 GB.

To construct a tiling image, overlaps of resized images were computed using
the Grid/Collection stitching plug-in in ImageJ/Fiji [5] and stitched to a
single tiling image. Then, the tiling image was converted to the DZI format
using the Python-based code deepzoom.py (initially developed by Kapil
Thangavelu) for visualisation by the electron microscopy viewer.

###  2.3 Development of Electron Microscopy Viewer and the Annotation of
Phenotype Data

To display many microstructural images as a single large image, we developed
the JavaScript-based RIKEN CLST Electron Microscopy Viewer, in which an
interactive multi-resolution visualisation function was implemented using
OpenSeadragon (https://openseadragon.github.io). In addition, overview,
metadata linkage, snapshot, annotation, measurement and importing and
exporting annotation file functions were implemented. The metadata of the
viewer are a part of metadata based on the microscopy ontology and are
contained in the taxon of National Center for Biotechnology Information
(NCBI), animal strain, derives from, imaging method and staining method used
to describe tiled images. We then annotated the phenotype data of the images,
such as the binuclear cell phenotype (CMPO:0000213) [6] in the liver, using
the electron microscopy viewer.

###  2.4 Construction of Metadatabase for Microstructural Imaging Data of
Biotissue

RIKEN has been making efforts to publish metadata which is suitable for the
cutting-edge research communities in life sciences by discussions with experts
in various fields such as ontology, informatics and biology [7]. Furthermore,
RIKEN has developed a database platform called the RIKEN MetaDatabase
(http://metadb.riken.jp) [7] with the help of two virtual machines on the
RIKENs private cloud platform using OpenLink Virtuoso
[https://virtuoso.openlinksw.com/], and have already started conducting pilot
operation as a metadata publishing service for biologists. In this study, we
introduced an RDF-based metadatabase for large-scale tissue microstructural
images. Then we generated the metadata and published the images and their
metadata in the Imaging Metadatabase
(http://metadb.riken.jp/metadb/db/clstMultimodalMicrostruct).

##  3 Results and Discussion

###  3.1 Vocabulary Extension for Experimental Description and Morphome Data
Description

We developed an ontology to define the upper-level concepts in microscope
imaging [3] as descriptions of ‘Project’, ‘Experimenter’, ‘Biosample’,
‘Image’, ‘Instrument’ and ‘Screening experiments’. This ontology is described
in OWL (OWL 2 DL) and has 324 classes and 225 properties. The representative
classes are shown in Table 1. For example, in the image descriptions, the
classes, such as dataset and image, were translated directly from the OME data
model. In addition, we added new classes, including a DZI dataset, phenotype
data of biosample and image, bioformat and tiling image (Table 1).

We defined a high-level concept so as to realise the common description of the
description vocabulary of the microscopy imaging and the difference among
various types of microscopies at the same time, by examining the concept in
the microscopy ontology. As a specific example, in the microscopy imaging
conditions, one of the difference between the optical microscope and the
electron microscope is the apparatus and the initial parameters. Although they
are individualised as subclasses, those that can be integrated using the
higher concept ‘ImagingCondition’.

Additionally, we focused on the description of the phenotype data because
phenotypic observations relating to the whole cell, cellular components and
cellular processes are a noteworthy research topic in life sciences [6]. Then,
we performed discrimination of the phenotype data relative to what was imaged
(subject) and what was captured (image) as Phenotype data of sample and
Phenotype data of image classes under the upper Phenotype data concept.

Further, we integrated such concepts which are having 15 super classes
described using rdfs:subClassOf and 27 classes which defines choices using
rdf:type. In this way, reusing data has progressed by handling such super
classes and choice classes in a large scale, which was not organised by the
XML-based OME data schema. Resultantly, imaging metadata for the rat liver
tissue, such as imaging method, SEM parameters, experimental conditions,
biological sample (a rat) and phenotype data, obtained from observations of
the biosample and images were successfully described, as shown in Figure 1.

Figure 1: Example graph structure showing that an abnormal binuclear phenotype
of a liver cell is detected from the imaging analysis of a Slc:SD rat with a
normal phenotype. The rounded rectangles are instances of classes, while ovals
are classes. An EM image (Image 001 of liver sample 001) shows a liver sample
(Liver sample 001) derived from a Slc:SD rat (Individual of Slc:SD rat 1).
Referencing Image 001 of liver sample 001, phenotype data (Phenotype data of
image 001) were produced. The graph structure can link bioresource information
in RIKEN to an external database from RS:0001400.

On the other hand, we designed the semantics of the DZI dataset such that they
can be handled properly by the image viewer application. We defined a DZI
dataset to link to a DZI. By referring to the DZI URL, this viewer provides a
graphical user interface to visualise the tiling image combined with hundreds
of microstructural biotissue images (Figure 2A). The viewer also functions
successfully for the metadata card view for the tiling image (Figure 2B) and
generates snapshots and a direct URI for a part of an image (Figure 2C).
Furthermore, using the viewer’s annotation function, users can define a region
of interest (ROI) and insert phenotype annotation of the cells or cellular
components (i.e. binuclear cell phenotype (CMPO:0000213) of liver cells)
(Figure 3). Users can search the ROIs for a specific phenotype in the search
mode by designating a phenotype term via a text search. Thus, the electron
microscopy viewer is a powerful and a useful tool to accumulate phenotype
information in morphome data. Also, the CMPO provided the description of the
general phenotypic observations in various kinds of biological samples and
utilised for phenotype annotation in several image repositories including the
IDR, allowing for future cross-species integration of phenotypic data [6].

Figure 2: Interactive visualisation of large tiling image in the RIKEN CLST
Electron Microscopy Viewer. (A) DZI of the tiling image visualised using the
viewer and images around the interlobular vein of the liver. (B) The viewer
contains metadata such as NCBI taxon, animal strain and derives from. (C) The
viewer includes a snapshot function to capture a certain part of the image.
These viewer data are published on the RIKEN web site
(http://clst.multimodal.riken.jp/CLST_ViewerData/EMV_025_161114SEM_RatLiverNormal/)

###  3.2 Imaging Metadatabase and Comprehensive Morphome Analysis on the
Semantic Web

Practically, a microscopy imaging database is required to provide various
imaging metadata corresponding to various experimental results and conditions
such as the disease phenotype of biotissues. Such metadata are a great benefit
when users search images as linked data for research and education purposes.
In this study, we conducted a series of histological experiments from the
sample preparation of biotissue (rat liver) to imaging experiments by SEM. In
addition, we described RDF-based metadata of these experimental processes
using the microscopy ontology. We also verified that images, experimental
conditions and phenotype data are linked and can be implemented as a practical
metadatabase. Using the microscopy ontology, we have described metadata for
approximately 300 images of rat liver tissue and have published the RIKEN
Microstructural Imaging Metadatabase. As a result, we have successfully
managed a vast number of images and their metadata, including the
interpretation of morphological phenotypes occurring in sub-cellular
components and biosamples captured in the images. Also, the accumulation of
machine-readable knowledge in the morphome, e.g. metadata that include
morphological phenotype data in diseases, can be utilised as a higher
educational system for specialised experience as well as an advanced research
in life sciences. Also by introducing such a semantic foundation to common
medical imaging data such as CT, MRI and PET, which are done in the clinical
field, the management and sharing of these imaging data will be considered to
be beneficial.

Figure 3: Metadata of a DZI based on the microscopy ontology to control the
image viewer. The deep zooming function of the image viewer accesses the DZI
dataset, which links the class of deep zoom image and other metadata
components. The DZI dataset can have an ROI that can be linked to image
phenotype annotations.

##  4 Conclusions

We have developed a semantic web-based imaging database, i.e. the RIKEN
Microstructural Imaging Metadatabase. As far as we know, a combination
application of the ontology-based imaging metadatabase and its viewer is the
first attempt in the world, and it has also made data integration with
bioresources and other data in the life science possible. In this paper, we
have shown an imaging dataset in the metadatabase that consists of
approximately 300 images of rat liver tissue. To describe the imaging-related
information, such as the experimental conditions of the EM and biosamples, we
extended the original OME vocabularies for optical microscopy and used them to
describe our EM imaging metadata. Moreover, we have developed an interactive
EM image viewer that can visualise a tiling image in the DZI format by
referring to the image metadata. We believe that these results will lead to
the evolution of morphomics and emerging bioimaging research fields.
Additionally, these results will be particularly useful for practical
biomedical research and education.

#### Acknowledgements

This paper is dedicated to RIKEN’s centennial. This work was supported by the
Management Expenses Grant for RIKEN CLST-JEOL Collaboration Center, the RIKEN
Ageing project, and the RIKEN engineering network project.

## References

  * [1] Kasthuri, N., et al: Saturated Reconstruction of a Volume of Neocortex. Cell, 162(3): 648–661. (2015)
  * [2] Williams, E. et al: Image Data Resource: a bioimage data integration and publication platform. Nature Methods, 14: 775–-781. (2017)
  * [3] Kume, S., et al: Development of an Ontology for an Integrated Image Analysis Platform to enable Global Sharing of Microscopy Imaging Data. Proceedings of the ISWC 2016 Posters & Demonstrations Track (2016)
  * [4] Goldberg, I.G., et al: The Open Microscopy Environment (OME) Data Model and XML file: open tools for informatics and quantitative analysis in biological imaging. Genome Biology, 6:R47 (2005)
  * [5] Preibisch, S. et al.: Globally optimal stitching of tiled 3D microscopic image acquisitions. Bioinformatics, 1; 25(11): 1463–1465. (2009)
  * [6] Jupp, S., et al: The cellular microscopy phenotype ontology. Journal of Biomedical Semantics, 7–28. (2016)
  * [7] Kobayashi, N. et al: RIKEN MetaDatabase: A Database Platform as a Microcosm of Linked Open Data Cloud in the Life Sciences. JIST 2016: Semantic Technology, pp. 99–115. (2016)


# A General Architecture for Client-Agnostic Hybrid Model Editors as a Service

Liam Walsh<EMAIL_ADDRESS>Queen’s UniversityCanada ,  Juergen Dingel
<EMAIL_ADDRESS>Queen’s UniversityCanada and  Karim Jahed
<EMAIL_ADDRESS>Queen’s UniversityCanada

(2022)

###### Abstract.

In this paper, we propose a general architecture for designing language
servers for hybrid modeling languages, that is, modeling languages that
contain both textual and graphical representations. The architecture consists
of a textual language server, a graphical language server, and a client that
communicates with the two servers. The servers are implemented using the
Language Server Protocol (LSP) and the Graphical Language Server Protocol
(GLSP) and are based on a shared abstract syntax of the hybrid language. This
means that only static resources need to be common between the graphical and
textual language servers. The servers’ separation allows each to be developed
and maintained independently, while also enabling forward-compatibility with
their respective dependencies.

We describe a prototype implementation of our architecture in the form of a
hybrid editor for the UML-RT language. The evaluation of the architecture via
this prototype gives us useful insight into further generalization of the
architecture and the way it is used. We then sketch a suitable extension of
the architecture to enable support for multiple diagram types and, thus,
multiple graphical views.

Hybrid, LSP, UML-RT, Graphical Modeling, Language Server

††ccs: Software and its engineering Software design engineering††ccs: Software
and its engineering Domain specific languages††journalyear: 2022††copyright:
rightsretained††conference: ACM/IEEE 25th International Conference on Model
Driven Engineering Languages and Systems; October 23–28, 2022; Montreal, QC,
Canada††booktitle: ACM/IEEE 25th International Conference on Model Driven
Engineering Languages and Systems (MODELS ’22 Companion), October 23–28, 2022,
Montreal, QC, Canada††price: 15.00††doi: 10.1145/3550356.3563131††isbn:
978-1-4503-9467-3/22/10

##  1\. Introduction

With the increasing amount of complexity in software systems, it is important
to be able to express them clearly and consistently. This is a must to ensure
effective communication among possibly diverse sets of stakeholders such as
designers, developers, and users. Abstracting away implementation-level
details is one way to provide clarity to a user. Consistency can be achieved
by a definition of rules and patterns. This type of approach to expressing
systems allows them to be easily understood by way of familiarity with their
domain, rather than programmatic knowledge.

The high-level motivation of our work is to enable better ways to express
models of software as used in Model-Driven Development (MDD). The ultimate
goal is to facilitate the development of hybrid editors for modeling
languages.

We propose a methodology of creating two separate services to equip Integrated
Development Environments (IDEs) with the capabilities of hybrid editing. This
involves the implementation of two concrete syntaxes, one textual and the
other graphical, each providing suitable representations (or views) of the
same underlying model and its abstract syntax. While this methodology seems
straightforward conceptually, it is challenging to implement in a way that
supports cohesion, maintainability and evolvability.

##  2\. Background

###  2.1. Hybrid Modeling Languages

Hybrid modeling languages are characterized by their use of more than one
concrete syntax paradigm. In this paper, we will use the term to refer
specifically to a modeling language which has both a textual concrete syntax,
and a graphical concrete syntax.

###  2.2. Language Servers

The classical method of implementing support for a language in an IDE usually
results in high coupling to the IDE, even though most modern IDEs are
functionally similar. Language servers take advantage of these similarities by
treating IDEs as clients to be serviced and whose behaviour is dictated by the
language’s semantics.

####  2.2.1. Language Server Protocol

The Language Server Protocol allows a client IDE to trade information and
instructions with a language server. The language server can run as a
background process and be queried by the IDE in order to give the user
advanced editing features such as “auto complete”, “go to
definition/declaration”, “find all references”, among others (Sourcegraph,
2022). LSP allows for the client IDE to be completely language-agnostic.

####  2.2.2. Graphical Language Server Protocol (GLSP)

The Graphical Language Server Protocol (Eclipsesource, 2022; Forstner, 2022),
is an open source framework originally built to facilitate displaying SVG
graphs and providing editing tools for graphical languages. This is all done
in such a way that one implementation would be able to service multiple IDEs
compliant with GLSP, hence the similarity in names between it and LSP.

GLSP functions by visiting a model, similar to how any other modeling tool
would parse it, and systematically creating graph objects that reflect each
different object in the original model specified by the developer. GLSP also
allows the language developer to choose where to persist any elements of the
concrete syntax that do not impact the execution semantics of the model (i.e.
graph X/Y positioning, colors, shapes, etc.).

###  2.3. UML for Real-Time

The Unified Modeling Language for Real-Time (UML-RT) is a modeling language
that is specified as a UML Profile. The language was created with the
properties necessary to be able to describe Real-Time systems (Posse and
Dingel, 2016; Selic, 2002). UML-RT’s established graphical specifications made
it a good candidate for our prototype hybrid editor.

The two main components of a system’s architecture are its structural and
behavioral aspects. UML-RT is able to create graphical models of systems that
are capable of specifying both these structural and behavioral aspects. Some
tools for specifying UML-RT models, such as the open-source eclipse-based
Papyrus-RT (Guindon, 2015), are able to automatically generate functional code
from a given model.

RTPoet is a toolset that we have created for the purposes of working with UML-
RT models of different forms (Queen’s University MASE Lab, 2022a). The
original use case was to bridge the gap between different UML-RT
implementations (specifically Papyrus-RT (Guindon, 2015) and RTIST/RSARTE
(Mohlin, 2018, 2020)) by model transformation. We have already specified a
textual implementation of UML-RT under the RTPoet umbrella. Once the UML-RT
hybrid language server prototype is sufficiently mature, we intend to
integrate it into the RTPoet suite.

##  3\. Proposed Architecture

In this section, we identify some key requirements that the architecture must
satisfy. Then, the structure of the architecture is described together with
sequence diagrams showing the interaction between components to realize
aspects of these requirements.

The use of any language in an IDE must include the support of the standard
CRUD (create, read, update, delete) operations. However, these are not a
necessary point of focus for our proof-of-concept implementation, as these
operations are mostly realized by the language definitions contained in the
servers themselves. We are more interested in the design of a high-level
architecture.

Current best practices in GUI development include the use of the Model-View-
Controller (MVC) design pattern. GLSP’s design is informed by this pattern, as
graphs naturally suggest the use of GUIs. We have chosen to model the textual
language server in a similar way. While MVC is not normally applied to text
editors, its properties allow us to maintain synchronization between views. In
the context of language servers, most of the responsibilities of the
“controller” and “view” are carried out by the client, language-specific tasks
are delegated to the server.

###  3.1. Four Core Interface Requirements

Figure 1. High-Level Relationship Between Graphical and Textual Views

Figure 1 shows our proposed general approach to creating the architecture for
hybrid language services. There are four main interfaces that must be
implemented in order to fulfill the CRUD use cases in each of the textual and
graphical views while keeping them synchronized. These four interfaces are
denoted by the edges in the figure, and each represent a transformation to, or
from, the views and model. Essentially, the model is the abstract syntax,
while the graphical and textual views are two different types of concrete
syntaxes used to specify and represent it. The four different interfaces will
be discussed in more detail below.

####  3.1.1. Generation

We will use “generation” or “regeneration” to refer to the transformation of
the textual view into the model. This is the intended usage of Xtext (Eysholdt
and Behrens, 2010), and will not be any different than the first step of
implementing a textual DSL (Domain-Specific Language) using the tool. The
realization of this interface requires the specification of a grammar for the
language. The textual language server uses the grammar to instruct the
language client how to perform this transformation.

####  3.1.2. Serialization

The model that the textual view generates retains enough information about the
original specification that it is still possible to perform a transformation
from the model itself back to the textual view. This process is the inverse to
the previously explained one, and the client is advised on how to do so by the
textual language server.

####  3.1.3. Rendering

The “rendering” of the graphical view should be realized by specifying a
mapping between model objects and graph element objects (predominantly
different types of nodes and edges). More granular parts of the language may
be left out of this mapping, if they can not be appropriately graphed and can
be more effectively handled by the textual view (e.g., action code in state
machines).

####  3.1.4. Updating

Graphical views typically only display a subset of the elements of the model
they represent. Lower-level properties may be simplified or excluded
altogether, meaning we cannot reliably retrieve these by transforming a
graphical view into a model. We can, however, manipulate the model directly by
keeping track of the graph elements’ source mappings. GLSP allows for this to
be done natively. We will refer to this as “updating” the model, as we are
applying an update directly to the model objects by proxy of the graphical
elements. Models can be kept complete and correct by restricting the
operations available in the graphical view (often by using “palettes of
operations”). Each time an update is applied, the graphical view must then be
rendered anew using the updated model.

###  3.2. Architecture Diagram

Figure 2 depicts our proposed general architecture for hybrid language
servers. The diagram shows the communication between two language clients,
encapsulated together in the “hybrid language client”, and two language
servers, together in the “hybrid language server”. The two pairs of clients
and servers communicate on different ports as, despite their similarities,
they make use of different protocols.

The previously described “generation” and “serialization” occur between the
“textual client” and the “model” shown in figure 2. Though, the methods that
the client uses to do so are given to it by the textual language server. The
same is true with respect to the graphical side: “rendering” and “updating”
occur between the “graphical client” and “model”, and the information on how
to carry out these steps is held by the graphical language server.

Figure 2. Architecture of Hybrid Language Server

###  3.3. Sequence Diagrams

In order to better illustrate the interaction between the modules in the
proposed general architecture, some sequence diagrams depicting different
behaviours have been included below.

Figure 3. Sequence Diagram: Textual Edit Action

The sequence diagram in figure 3 shows how a change to the textual view would
be propagated to the graphical view.

Figure 4. Sequence Diagram: Graphical Edit Action

The sequence diagram in figure 4 shows how a change to the graphical view
would affect the textual view. By simply observing the shape of this sequence
compared to the previous one, it is clear that they are very similar.

##  4\. Prototype Implementation

In order to evaluate our proposed architecture, we have created a prototype
hybrid language server, which we will refer to as the “RTPoet language”, based
on UML-RT. Below is a description of our implementation of the four core
interfaces described in section 3.1. This is followed by a discussion of our
findings.

###  4.1. Generation

To implement the generation interface, a textual grammar for the RTPoet
language has been specified using Xtext.

While most Xtext-based languages often define code generation instructions in
addition to a grammar, the code stubs included when creating a new Xtext
project are sufficient to allow model files to be generated from the textual
view. This is possible because the model file is purely structural.

We have made our Xtext implementation of UML-RT (i.e., the RTPoet language’s
textual syntax) available as part of the RTPoet tool suite, available on
GitHub (Queen’s University MASE Lab, 2022b).

###  4.2. Serialization

In order to be able to serialize a model back into the textual view, specific
mappings to code templates must be specified for each of the model objects.
Whereas the aforementioned generation process would typically yield “markup
language” string representations of objects as their target (such as EMF/XML
models), this type of transformation would need to yield complete and correct
“code”. The templates can consist of predefined code blocks with different
types of values and identifiers substituted into the appropriate places. A
traversal of the model’s abstract syntax tree allows the string templates to
be correctly ordered. This is similar to how an Xtext language might normally
be equipped to generate code, but it easier to implement considering we
already have access to the target language’s grammar and metamodel.

###  4.3. Rendering

The metamodel resulting from the specification of the Xtext grammar was used
to automatically create a full Java class model for the RTPoet language. The
structure of these classes exactly reflects the scope of models specifiable in
the textual syntax. This collection of Java classes was used as a basis for
the implementation of the graphical language server. The collection’s format
is identical to the metamodel. The next step was to map each metamodel type to
an appropriate GLSP graphical type. GLSP contains its own classes for graph
elements, and assigns them SVG properties at runtime.

Next, a model visitor was created, based on one of GLSP’s extension points,
that converts any given model into a collection of graphical objects based on
the mappings. The graph resulting from our rendering operation is held
directly in memory. The source mappings are kept track of by GLSP. Since we
have opted to not store any additional graphical properties, we chose to
implement automatic arrangement of the elements.

###  4.4. Updating

Some basic update instructions were added to the graphical language server,
such as the ability to add a new simple state to a state machine. However, the
implementation of the prototype was halted here, as there were architectural
incompatibilities that needed to be addressed. This will be discussed in
detail below.

##  5\. New Proposed Architecture

We found that, given our current approach, the functionality of our
implementation of UML-RT was not up to par with existing editors. In order to
implement hybrid languages in a more convincing, user-friendly way, single
views are insufficient. In particular, single views offer poor support for the
following modeling concepts and activities.

###  5.1. Containment

A concept that is present in many modeling languages is that of object
containment. If an object owns a collection of other objects, it may make
sense to render the collection of objects within the boundaries of their
owner. However, as object ownership depth increases, this becomes harder to
manage. A better solution is to render only objects with a direct containment
relation. This would mean that in order to interact with elements of greater
depth, we would need the ability to drill down into elements using new views.

###  5.2. Projection

Models often contain many elements, complicating their user-friendly graphical
display. Projection, that is, the selective display of only specific subsets
of model elements, can mitigate this problem.

These projections can be based on language concepts, such as types or other
fixed properties, and they can be further influenced by user input. For
instance, some elements in a UML-RT model represent structure, while some
other represent behaviour (as seen in Figure 5). While displaying different
projections simultaneously in a single view can be useful, it can also cause
confusion, and support for different views seems preferable in general.

###  5.3. Static Analysis

A static analysis often involves using the model as an input and, after some
additional processing, returning an output whose syntax and semantics may well
be outside the scope of the modeling language itself. For example, we might
want to see a full traversal of a state machine. This would include duplicate
states, something not possible using only the vanilla model view. Therefore,
rendering graphs for these types of operations is intrinsically incompatible
with a single model visualization.

Figure 5. Structure vs Behaviour in UML-RT Model

In summary, an architecture for hybrid language servers should enable support
for different views. Only then can different parts of the model, as well as
results of different projections and analyses, be displayed most
appropriately.

By using multiple different graphical views, we may represent languages’
structure more accurately while also providing an extension point for
supplemental graph functionality.

###  5.4. Modified Architecture

Our architecture can be modified to support multiple views. This modification
is minimally invasive and, from the point of view of the client, it is the
same as before. The new proposed architecture can be seen in figure 6.

Figure 6. New Architecture of Hybrid Language Server

####  5.4.1. Switching Views

A short sequence diagram depicting the switching of graphical views can be
seen in figure 7. The initial action could come from simply clicking on an
element. In this case, the client would send the click with a reference to the
clicked element along with necessary metadata. We can also see that the latter
half of the sequence is the same as the previous graph rendering sequence in
figure 3. Unless we explore the local scope of the Graphical Language Server,
the process will not appear much different.

Figure 7. Sequence: Graphical View Switching

##  6\. Related Work

##### Passing Graphical Information Using LSP

Papers (Rodriguez-Echeverria et al., 2018) and (Rodríguez-Echeverría et al.,
2018) discuss the proposal and implementation of a custom graphical editing
solution for EMF languages, using LSP without any extensions. Their solution
involves converting the EMF model into a JSON-based format in order to
exchange it with the server across LSP. The JSON-based equivalent of the whole
EMF model is what is transmitted, including pure graph properties to render
the graph as an SVG (e.g. width:100, height:50, shape:square). The usage of
JSON-based messages makes this effectively an implementation of a textual
language server. They also define actions that the front-end client is able to
perform and manually map them to specific LSP messages, such as creating or
removing edges or nodes. In the earlier of the their two papers, they propose
the solution as completely compliant with LSP as a prototype. In the later
paper, they discuss the possibility of extend LSP or defining a brand new
protocol, but ultimately decide to remain compliant with LSP. Since our work
separates the two editing mediums’ implementations, we will not need to
evaluate problems such as protocol extensions.

##### UML Profile Hybrid Editors

Addazi et al. (Addazi et al., 2017) propose a methodology of implementing
hybrid languages based on UML by extending it (i.e. creating UML profiles). In
concept, their approach is similar to ours, as it makes use of a shared
abstract syntax between the two different views of a given language. Similar
to our proposed architecture, the source of their shared metamodel is based on
Xtext’s grammar specifications. The authors use UML as a basis for their
specification of Xtext grammars, allowing them to leverage its existing
graphing libraries. Our approach does not rely on UML, but it could be
extended in the same way (i.e. used as a basis for typing in the Xtext grammar
specification). However, reusing any of the graphical editor implementation
designs in this work is not possible in the context of language servers. The
UML graphing libraries’ dependence on IDE-specific features render them
incompatible with our proposed architecture both in concept and in practice.
That being said, the stability of UML could motivate the creation of similar
UML-based graphing libraries that operate within the boundaries of graphical
language servers.

##### bigER Modeling Language

Glaser et al. have created the “bigER” tool (Glaser and Bork, 2021) which, as
of yet, seems to be the most closely related to our own work. The bigER tool
is an example of a functional hybrid language server that also makes use of
Xtext. The approach to the design of the bigER tool mirrors one of our prior
proposed architectures for hybrid language servers (Walsh et al., 2020; Walsh,
2020). At a high level: graphical operations are mapped to code snippets which
are injected directly into to the textual model. This is similar to the code
templating required for defining the “serialization” interface, but controlled
by a palette of operations similar to that required by the “updating”
interface (as seen in figure 1). The textual model can then regenerate the
graphical model, displaying the changes graphically. We did not pursue this
style of architecture for several reasons. Most notably, this solution to
synchronization of the two editors is more difficult to specify and maintain
as the size of the language increases. For more complex languages, static
analysis might need to be done on the code in order properly contextualize the
code snippets. Our current proposed architecture uses the language’s structure
to guarantee completeness and correctness of graphical edits. Language
specification errors in our implementation can more easily be detected at
compile time, whereas errors using this style of implementation may only be
clear at runtime.

##### Graphical Viewspoints

In (Andrés et al., 2007), domain-specific modeling languages with multiple
“viewpoints” are discussed. The work describes the nuances of the relationship
between model and graphs. The authors present a methodology for designing
multi-view DSVLs (Domain-Specific Visual Languages) in which they note that
models can be projected into submodels in different “viewpoints” (which are
analogous to the “views” described in this work). The concept of viewpoints
closely mirrors our proposed implementation of views. While our work was not
inspired by this paper, it corroborates our findings well. The authors’ work
predates LSP, meaning that it evidently does not make use of it. This is a key
differentiating factor relative to our work.

##  7\. Conclusion and Future Work

We have proposed and evaluated a methodology for designing and implementing
hybrid language servers, in the form of a general architecture, that offers a
cohesive approach, simplifies maintainability, and facilitates language
evolution.

We anticipate that there is much work to be done in the study of hybrid
language design and usability. This may be inspired by existing research on
best practices for textual and graphical mediums alike (Moody, 2009; Tufte et
al., 1998). We hope that our findings may be foundational to the accessibility
of hybrid modeling languages, such that we may facilitate future work in
studying them as a whole, and not only relative to language servers.

## References

  * (1)
  * Addazi et al. (2017) Lorenzo Addazi, Federico Ciccozzi, Philip Langer, and Ernesto Posse. 2017.  Towards Seamless Hybrid Graphical–Textual Modelling for UML and Profiles. 20–33.  https://doi.org/10.1007/978-3-319-61482-3_2
  * Andrés et al. (2007) Francisco Andrés, Juan Lara, and Esther Guerra. 2007\.  Domain Specific Languages with Graphical and Textual Views, Vol. 5088. 82–97.  https://doi.org/10.1007/978-3-540-89020-1_7
  * Eclipsesource (2022) Eclipsesource. 2022\.  Graphical Language Server Protocol.  https://github.com/eclipse-glsp/glsp
  * Eysholdt and Behrens (2010) Moritz Eysholdt and Heiko Behrens. 2010.  Xtext: Implement Your Language Faster Than the Quick and Dirty Way. In _Proceedings of the ACM International Conference Companion on Object Oriented Programming Systems Languages and Applications Companion_ (Reno/Tahoe, Nevada, USA) _(OOPSLA ’10)_. ACM, New York, NY, USA, 307–309.  https://doi.org/10.1145/1869542.1869625
  * Forstner (2022) Luca Forstner. 2022\.  Integrating GLSP based Tooling into Visual Studio Code.  Bachelor Thesis.
  * Glaser and Bork (2021) Philipp-Lorenz Glaser and Dominik Bork. 2021.  The bigER Tool - Hybrid Textual and Graphical Modeling of Entity Relationships in VS Code. In _2021 IEEE 25th International Enterprise Distributed Object Computing Workshop (EDOCW)_. 337–340.  https://doi.org/10.1109/EDOCW52865.2021.00066
  * Guindon (2015) Christopher Guindon. 2015\.  Papyrus-RT - Overview: The Eclipse Foundation.  https://www.eclipse.org/papyrus-rt/content/overview.php
  * Mohlin (2018) Mattias Mohlin. 2018\.  Modeling Real-Time Applications in RTist.  https://rtist.hcldoc.com/help/topic/com.ibm.xtools.rsarte.webdoc/pdf/RTist%20Concepts.pdf
  * Mohlin (2020) Mattias Mohlin. 2020\.  Modeling Real-Time Applications in RSARTE.  https://rsarte.hcldoc.com/help/topic/com.ibm.xtools.rsarte.webdoc/pdf/RSARTE%20Concepts.pdf
  * Moody (2009) Daniel Moody. 2009\.  The “Physics” of Notations: Toward a Scientific Basis for Constructing Visual Notations in Software Engineering.  _IEEE Transactions on Software Engineering_ 35, 6 (2009), 756–779.  https://doi.org/10.1109/TSE.2009.67
  * Posse and Dingel (2016) Ernesto Posse and Juergen Dingel. 2016.  An executable formal semantics for UML-RT.  _Software & Systems Modeling_ 15, 1 (2016), 179–217.
  * Queen’s University MASE Lab (2022a) Queen’s University MASE Lab. 2022a.  RTPoet.  https://github.com/qumase/rtpoet
  * Queen’s University MASE Lab (2022b) Queen’s University MASE Lab. 2022b.  RTPoet DSL.  https://github.com/qumase/rtpoet-dsl
  * Rodriguez-Echeverria et al. (2018) Roberto Rodriguez-Echeverria, Javier Luis Cánovas Izquierdo, Manuel Wimmer, and Jordi Cabot. 2018\.  An LSP infrastructure to build EMF language servers for web-deployable model editors. In _MODELS Workshops_.
  * Rodríguez-Echeverría et al. (2018) Roberto Rodríguez-Echeverría, Javier Luis Cánovas Izquierdo, Manuel Wimmer, and Jordi Cabot. 2018\.  Towards a Language Server Protocol Infrastructure for Graphical Modeling. In _Proceedings of the 21th ACM/IEEE International Conference on Model Driven Engineering Languages and Systems_ (Copenhagen, Denmark) _(MODELS ’18)_. ACM, New York, NY, USA, 370–380.  https://doi.org/10.1145/3239372.3239383
  * Selic (2002) B. Selic. 2002\.  The real-time UML standard: definition and application. In _Proceedings 2002 Design, Automation and Test in Europe Conference and Exhibition_. 770–772.  https://doi.org/10.1109/DATE.2002.998385
  * Sourcegraph (2022) Sourcegraph. 2022\.  Langserver.org A community-driven source of knowledge for Language Server Protocol implementations.  https://langserver.org/
  * Tufte et al. (1998) Edward R Tufte, Susan R McKay, Wolfgang Christian, and James R Matey. 1998.  Visual explanations: Images and quantities, evidence and narrative.
  * Walsh (2020) Liam Walsh. 2020\.  Slide Deck: Toward Client-Agnostic Hybrid Model Editor Tools as a Service.  https://msdl.uantwerpen.be/conferences/MPM4CPS/2020/wp-content/uploads/2020/11/Liam_Walsh_MPM4CPS_2020.pdf
  * Walsh et al. (2020) Liam Walsh, Juergen Dingel, and Karim Jahed. 2020\.  Toward client-agnostic hybrid model editor tools as a service. In _Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings_. 1–1.  https://doi.org/10.1145/3417990.3421440