figs

# HetTree: Heterogeneous Tree Graph Neural Network

Mingyu Guan  Georgia Institute of TechnologyUSA<EMAIL_ADDRESS>,  Jack
W. Stokes  Microsoft ResearchUSA<EMAIL_ADDRESS>,  Qinlong Luo
MicrosoftUSA<EMAIL_ADDRESS>,  Fuchen Liu  MicrosoftUSA
<EMAIL_ADDRESS>,  Purvanshi Mehta  Lica World IncUSA
<EMAIL_ADDRESS>,  Elnaz Nouri  Microsoft ReasearchUSA<EMAIL_ADDRESS>and  Taesoo Kim  Georgia Institute of TechnologyUSA<EMAIL_ADDRESS>
###### Abstract.

The recent past has seen an increasing interest in Heterogeneous Graph Neural
Networks (HGNNs) since many real-world graphs are heterogeneous in nature,
from citation graphs to email graphs. However, existing methods ignore a tree
hierarchy among metapaths, which is naturally constituted by different node
types and relation types. In this paper, we present HetTree, a novel
heterogeneous tree graph neural network that models both the graph structure
and heterogeneous aspects in a scalable and effective manner. Specifically,
HetTree builds a semantic tree data structure to capture the hierarchy among
metapaths. Existing tree encoding techniques aggregate children nodes by
weighting the contribution of children nodes based on similarity to the parent
node. However, we find that this tree encoding fails to capture the entire
parent-children hierarchy by only considering the parent node. Hence, HetTree
uses a novel subtree attention mechanism to emphasize metapaths that are more
helpful in encoding parent-children relationships. Moreover, instead of
separating feature learning from label learning or treating features and
labels equally by projecting them to the same latent space, HetTree proposes
to match them carefully based on corresponding metapaths, which provides more
accurate and richer information between node features and labels. Our
evaluation of HetTree on a variety of real-world datasets demonstrates that it
outperforms all existing baselines on open benchmarks and efficiently scales
to large real-world graphs with millions of nodes and edges.

##  1\. Introduction

Graph neural networks (GNNs) have been widely explored in a variety of domains
from social networks to molecular properties (PinnerSage2020, ; Park2019, ;
sun2022does, ), where graphs are usually modeled as homogeneous graphs.
However, real-world graphs are often heterogeneous in nature (OGB, ; HGB, ).
For example, as shown in Figure 1(a), a heterogeneous email graph can involve
multiple types of nodes, including Domain, Sender, Recipient, Message and IP
Address, and the relations among them. Moreover, multiple relations can exist
between two entities in complex heterogeneous graphs. For example, a _Sender_
node can be a P1 sender and/or P2 sender of a _Message_ , where the P1 sender
denotes the entity that actually sent the message while an email application
displays the P2 sender as the “From” address. If Bob sends an email on behalf
of Alice, the email appears to come from Alice and Alice is the P2 sender
while Bob is the P1 sender in this scenario. As a result, there are two
relations between Sender and Message in Figure 1(a), p1_sends and p2_sends.

Figure 1. (a) Relational scheme of a heterogeneous email graph (b) An example
of the email graph.

To better understand real-world heterogeneous graphs, various heterogeneous
graph neural networks (HGNNs) have been proposed. The most well-known
approaches are the metapath-based methods (RGCN, ; HAN, ; MAGNN, ), which
first aggregate neighbor representations along each metapath at the node level
and then aggregate these representations across metapaths at the metapath
(semantic) level. However, metapath-based approaches often involve manual
effort to select a subset of meaningful metapaths, because the node-level
aggregation along each metapath for every node is computationally expensive
(MAGNN, ; HAN, ). Other models that do not apply the metapath method, such as
HetGNN (HetGNN, ) and HGT (HGT, ), carefully encode representations for
different node types and/or relation types in heterogeneous graphs. These
fine-grained embedding methods often utilize multi-layer message passing
approaches in traditional GNNs thus facing scalability issues. To efficiently
model real-world web-scale graphs, researchers and practitioners have explored
various ways to scale HGNNs. Sampling-based methods (HGT, ; HetGNN, ) sample
sub-graphs with different strategies, while others (NARS, ; SeHGNN, ) use
model simplification to execute feature propagation as a pre-processing stage
before training.

However, a tree hierarchy among the metapaths is ignored by existing HGNNs.
Considering the example in Figure 1, intuitively, metapath
$Sender\xrightarrow{p1\\_sends}Message$ $\xrightarrow{is\\_sent\\_from}IP$ is
more related to metapath $Sender\xrightarrow{p1\\_sends}Message$ than metapath
$Sender\xrightarrow{s\\_has\\_domain\\_of}Domain$, because of the overlap of
more node types and relations. Existing tree-based HGNNs (SHGNN, ; HetGTCN, ;
T-GNN, ) use tree structure to model the local topology of nodes, where a
semantic aggregation using attention is followed to combine the semantic
information for metapaths. However, simple semantic attention cannot fully
capture the semantic tree hierarchy among metapaths, where the parent-child
relationships between metapaths are ignored.

Moreover, to augment the data usage, label utilization is widely used in GNNs
(SLE, ; UniMP, ; GAMLP, ; LEGNN, ), which leverages ground truth in the
training set and propagates them through the graph structure as inputs to the
model. These methods either completely separate feature learning from label
learning and only combine them at the end, or simply treat the features and
label vectors the same by projecting them to the same latent space. They may
be feasible in homogeneous graphs but not in the case of heterogeneous graphs,
as the features and labels are related by the corresponding metapaths.

In this paper, we present HetTree, a scalable HGNN that extracts a unified
tree representation on metapaths, semantic tree, from the input heterogeneous
graph and proposes a novel tree aggregation with subtree attention to encode
the resulting semantic tree, in which propagated labels are carefully matched
with corresponding features based on metapaths.

To scale efficiently on web-scale graphs, HetTree follows the model
simplification approach that simplifies heterogeneous feature aggregation as a
pre-processing stage. Meanwhile, label aggregation of the target node types
for each metapath is also executed. HetTree then builds a semantic tree to
capture the hierarchy among metapaths. Instead of separating feature learning
from label learning (SLE, ; GAMLP, ) or treating features and labels equally
by projecting them to the same latent space (SeHGNN, ), HetTree proposes to
match them carefully on corresponding metapaths, which provides more accurate
and richer information between node features and labels.

To encode the resulting semantic tree, HetTree uses a novel subtree attention
mechanism to emphasize children nodes that are more helpful in encoding
parent-children relationships. Existing tree encoding techniques (SHGNN, ;
HetGTCN, ; T-GNN, ) aggregate children nodes by weighting the contribution of
children nodes based on similarity to the parent node. However, in the
semantic tree, this tree encoding fails to capture the entire parent-children
hierarchy by only considering the parent node. Hence, subtree attention in
HetTree models a broader parent-children structure while enhancing
correlations, bringing a better representation for each metapath.

In summary, we make the following contributions in this paper.

  * •

We observe that existing HGNNs use simple semantic attention to aggregate
metapath representations, which ignore the hierarchy of the metapath features.
HetTree takes a radically new approach to encoding metapath hierarchy by
building a semantic tree for both pre-computed features and labels. To the
best of our knowledge, HetTree is the first to explore the tree hierarchy
among metapaths constructed from a heterogeneous graph.

  * •

HetTree proposes a novel tree aggregation with subtree attention to encode the
proposed semantic tree structure, which emphasizes metapaths that are more
helpful in encoding parent-children relationships. For better label usage,
HetTree matches pre-computed features and labels correspondingly, which
constitutes a complete representation of a metapath.

  * •

We conduct extensive experiments on five open graph datasets from Open Graph
Benchmark (OGB, ) and HGB Benchmark (HGB, ), as well as a real-world
commercial email dataset. The results demonstrate that HetTree is able to
outperform the state-of-the-art architectures on all datasets with low
computation and memory overhead incurred.

##  2\. Related Work

###  2.1. Graph Neural Networks

Graph Neural Networks (GNNs) are neural networks that take input structured as
graphs. The fundamental task in a GNN is to generate the representation of
graph entities, such as nodes and edges, in a $d$-dimensional space, referred
to as the embedding of the entity. To generate the embedding, GNNs usually use
a multi-layer feature propagation followed by a neural network to combine
structural information from the graph structure and the input features.
Various GNN architectures exist today, but they differ in how the information
is aggregated and transformed (Hamilton2017, ; GCN2016, ; GAT2018, ;
GraphSage2017, ). Nevertheless, a main problem with these classic GNNs is that
they are hard to scale due to the feature propagation performed at each layer
of the neural networks. A number of sampling methods (GraphSage2017, ;
FastGCN2018, ; LADIES2019, ; zou2019layer, ) have been proposed to reduce both
computation and memory complexity by using only a subset of nodes and edges.
Besides sampling, other methods (SGC, ; SIGN, ; S2GC, ) simplify models by
making feature propagation an offline pre-processing stage, so that this
computation-intensive process only needs to be executed once and is not
involved during the training process.

###  2.2. Heterogeneous Graph Neural Networks

While GNNs achieve excellent performance on homogeneous graphs, many real-
world graphs tend to be heterogeneous and applying homogeneous GNN approaches
to them is not trivial. Consequently, various excellent works have
investigated heterogeneous GNN architectures (HGNNs). The most general
approach in HGNNs is the so-called metapath-based method (MAGNN, ; HAN, ),
where the feature propagation is performed based on semantic patterns and
attention mechanism is usually applied at both the node level for each
metapath and the semantic level across metapaths. Other models (RGCN, ;
HetGNN, ; HGT, ; HGB, ) encode the heterogeneous aspect of the graph at the
fine-grained level using the multi-layer, message-passing framework common in
GNNs, where different weights are learned for distinct node and edge types.

However, HGNNs also inherit the scalability problem from traditional
homogeneous GNNs. Hence, sampling (HGT, ) and model simplification (NARS, )
have also been explored in the heterogeneous graph learning domain. NARS
(NARS, ) first applies the scaling approach proposed by SIGN (SIGN, ) on
heterogeneous graphs, which samples multiple relational subgraphs using
different sets of relations and then treats them as homogeneous graphs. Though
solving the problem of scalability, this approach can result in losing
semantic information by treating heterogeneous graphs as homogeneous ones
thereby confusing model learning by mixing different metapath semantics in
relational subgraphs. SeHGNN (SeHGNN, ) computes averaged metapath features
separately and applies Transformer-like attention to learn metapath features.
However, simply applying the Transformer ignores the hierarchy among metapath
features and thus results in sub-optimal results. Additionally, existing HGNNs
represent matapaths using node types, which fails to differentiate metapaths
that have different relation compositions but the same node types along the
metapaths.

###  2.3. Tree-based HGNNs

A few HGNNs have explored tree structure based on the local topology of nodes.
T-GNN (T-GNN, ) and SHGNN (SHGNN, ) construct hierarchical tree structures at
the node level, where each hierarchical tree represents a metapath and each
level of the tree contains nodes of a certain type. Similarly, HetGTCN
(HetGTCN, ) also constructs the tree hierarchy based on the local graph
structure of each node, where tree nodes at $k^{th}$ level are $k-hop$
neighbors of the root node. To encode tree-structured data, i.e. parent-child
relationships among tree nodes, they either use a weighted sum aggregator
(T-GNN, ; HetGTCN, ) or compute weights using attention mechanism for each
parent-child pair (SHGNN, ; HetGTCN, ). Moreover, these methods utilize the
tree structure to aggregate neighbor information for each metapath first and
then use semantic attention (HAN, ) to aggregate metapath representations to
obtain the final node representation. However, the tree hierarchy among
metapaths is ignored by existing methods. To the best of our knowledge,
HetTree is the first to explore the heterogeneous aspect of graphs based on a
tree-structured hierarchy among metapaths, which is naturally constituted by
different node types and relation types.

###  2.4. Label Utilization

Label utilization has been commonly applied in graph representation learning.
Label propagation (LabelPropagation, ) uses partially observed labels and
propagates them through the network structure. Label usage (LabelUsage, ) uses
a simple but effective method that concatenates node features and one-hot
labels. SLE (SLE, ) utilizes label propagation, separates feature and label
learning with two MLPs and then adds them together to generate the final
representations. However, label utilization naturally incurs two problems:
first, using labels in training can easily cause label leakage; second, high
training accuracy and low test accuracy can occur since only the training data
has labels. UniMP (UniMP, ) randomly masks the training nodes for each epoch.
GAMLP (GAMLP, ) modifies label propagation with residual connections to each
hop results to alleviate the label leakage issue.

However, these methods either completely separate feature learning and label
learning and only combine them at the end, or they simply add the features and
label vectors together as propagation information, which may result in good
performance in homogeneous graphs but not in the case of heterogeneous graphs,
since the features and labels are related by the corresponding metapaths.

##  3\. Preliminary

In this section, we provide formal definitions of important terminologies
related to HetTree. We also summarize notations frequently used in this paper
in Table 1.

Notations | Definitions
---|---
$\mathcal{G}$ | A heterogeneous graph $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{O},\mathcal{R})$.
$\mathcal{V}$ | The set of nodes.
$\mathcal{E}$ | The set of edges.
$\mathcal{O}$ | The set of node types.
$\mathcal{R}$ | The set of relation types.
$\mathcal{P}^{k}$ | The set of metapaths up to hop $k$.
$P$ | A metapath.
$\mathcal{P}_{O}$ | The set of metapaths ending with node
| type $O$ excluding $P=O^{init}$.
$\mathcal{N}^{v}_{P}$ | The set of metapath-$P$-based neighbors of node $v$.
$x^{v}$ | Initial feature of node $v$.
$y^{v}$ | Ground truth label of node $v$.
$X_{P}$ | Aggregated feature along metapath $P$.
$\hat{Y}_{P}$ | Aggregated label along metapath $P$.
$T_{O}$ | Semantic tree for node type $O$.
$C_{P}$ | A node in the semantic tree for metapath $P$.
$\mathcal{M}$ | The set of Metapath features.
$M_{P}$ | Metapath feature of metapath $P$.
$Z_{P}$ | Tree-encoded metapath feature of metapath $P$.
Table 1. Notations frequently used in this paper.

###### Definition 3.0.

Heterogeneous Graph. A heterogeneous graph is denoted as
$\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{O},\mathcal{R})$, where each
node $v\in\mathcal{V}$ and edge $e\in\mathcal{E}$ are associated with a node
mapping function $\tau(v):\mathcal{V}\rightarrow\mathcal{O}$ from node set
$\mathcal{V}$ to node type set $\mathcal{O}$, and a edge mapping function
$\phi(e):\mathcal{E}\rightarrow\mathcal{R}$ from edge set $\mathcal{E}$ to
relation set $\mathcal{R}$, respectively.

###### Example 3.0 0.

Figure 1(a) shows the relational scheme of a heterogeneous email graph while
Figure 1(b) shows an illustrative example. It is composed of five types of
nodes: $Domain$,$Sender$, $Message$, $Recipient$, $IP$, and six types of
relations: s_has_domain_of(H),
r_has_domain_of(D), p1_sends(O), p2_sends (T), receives(R),
is_sent_from(F).

Figure 2. (a) The offline process of feature aggregation. The center node is
the target node with $Sender$ node type and features are aggregated for all
metapaths $\mathcal{P}^{k}$ up to hop $k$, where $k=2$ in this example. For
instance, to compute aggregated feature $X_{HH}$, features of three
metapath-$HH$-based neighbors are aggregated, including the target node
itself. (b) The offline process of label aggregation. Unlike feature
aggregation, labels are aggregated only for metapaths
$\mathcal{P}^{k}_{Sender}$ that end with node type $Sender$ up to hop $k$. For
example, to compute aggregated label $\hat{Y}_{HH}$, features of two
metapath-$HH$-based neighbors are aggregated, excluding the target node
itself. (c) An illustration of the semantic tree with height of $k$ for Sender
nodes in heterogeneous email graph. A tree node $C_{P}$ represents metapath
$P$, where $P\in\mathcal{P}^{k}$.

###### Definition 3.0.

Metapath. A metapath $P$ is a path that describes a composite relation
$R=R_{1}\circ R_{2}\circ\cdots\circ R_{l}$ between node types $O_{1}$ and
$O_{l+1}$, where $\circ$ denotes the composition operator on relations. $P$ is
denoted as
$O_{1}\xrightarrow{\text{$R_{1}$}}O_{2}\xrightarrow{\text{$R_{2}$}}\cdots\xrightarrow{\text{$R_{l}$}}O_{l+1}$,
which is abbreviated as $R_{1}R_{2}\cdots R_{l}$. A special metapath where no
relation is present but includes only a node type $O$ is simply denoted as
$O^{init}$ (abbreviated as $init$ when $O$ is specified). The set of metapaths
ending with node type $O$ excluding $P=O^{init}$ is denoted as
$\mathcal{P}_{O}$. The set of metapaths up to hop $k$ is denoted as
$\mathcal{P}^{k}$, where $l\leq k,\forall P=R_{1}R_{2}\cdots
R_{l}\in\mathcal{P}^{k}$. Metapath-based neighbors $\mathcal{N}^{v}_{P}$ of
node $v$ along metapath $P$ are the set of nodes that are connected with node
$v$ via metapath $P$. Note that when $\tau(v)=O$ and $P$ ends with $O$,
$\mathcal{N}^{v}_{P}$ includes $v$ itself.

###### Remark 3.0 0.

Most of metapath-based HGNNs denote a metapath
$P=O_{1}\xrightarrow{\text{$R_{1}$}}O_{2}\xrightarrow{\text{$R_{2}$}}\cdots\xrightarrow{\text{$R_{l}$}}O_{l+1}$
as $O_{1}O_{2}\cdots O_{l+1}$ for short. We note that this notation fails to
differentiate metapaths that have different relation compositions but the same
node types along the metapaths, as multiple relations can be present between
two node types in a heterogeneous graph.

###### Example 3.0 0.

In Figure 1, a sender can be connected to a recipient through three 2-hop
metapaths: $Sender\xrightarrow{O}Message\xrightarrow{R}Recipient$ ($OR$),
$Sender\xrightarrow{T}Message\xrightarrow{R}Recipient$ ($TR$),
$Sender\xrightarrow{H}Domain\xrightarrow{D}Recipient$ ($HD$). Moreover, let
$S=\\{OR,TR,HD\\}$, then we have $S\subset\mathcal{P}^{k}$, for any $k\geq 2$.

###### Definition 3.0.

Semantic Tree. A semantic tree $T_{O}$ with depth of $k$, for node type $O$,
is composed of tree nodes $\mathcal{C}={C_{P},\forall P\in\mathcal{P}^{k}}$
and relation edges $\mathcal{R}$. The root node represents the metapath
$P=O^{init}$, denoted as $C_{O^{init}}$ (abbreviated as $C_{init}$ when $O$ is
specified). A non-root node $C_{R_{1}R_{2}\cdots R_{l}}$ represents the
metapath from the root node $C_{init}$ to them via relation edges
$R_{1}R_{2}\cdots R_{l}$. The root node $C_{init}$ is the parent of all nodes
$C_{R_{1}}$ at depth $1$ of the semantic tree $T_{O}$. A node
$C_{R_{1}R_{2}\cdots R_{l}}$ with depth $\geq 2$ has a parent node
$C_{R_{1}R_{2}\cdots R_{l-1}}$, and they are connected by edge $R_{l}$.

###### Example 3.0 0.

A semantic tree with depth of $2$ for Sender nodes in the heterogeneous email
graph is shown in Figure 2, where the root node $C_{init}$ represents metapath
init and other nodes represent the metapath from the root node to them. For
example, node $C_{OF}$ is connected with the root node $C_{init}$ by relation
edges $O$ and $F$ in order.

##  4\. Methodology

In this section, we describe a novel heterogeneous tree graph neural network,
HetTree, for scalable and effective heterogeneous graph learning. HetTree
consists of three major components: offline feature aggregation and semantic
tree construction (Section 4.1), metapath feature transformation (Section
4.2), and semantic tree aggregation (Section 4.3). We conduct complexity
analysis on HetTree and compare it with representative tree-based and scalable
HGNNs (Section 4.4). The overall process of HetTree is shown in Algorithm 1.

###  4.1. Offline Aggregation and Semantic Tree Construction

As a pre-processing stage, node features and labels are aggregated to prepare
for training. In node classification tasks on heterogeneous graphs, initial
node features are normally associated with every node (if not, we can use
external graph embedding algorithms like ComplEx (ComplEx, ) to generate
them), while labels are often only associated with nodes of target node types
that need to be classified. For example, in the email dataset we collect, the
task is to classify Sender nodes as a compromised email account or not, and
only Sender nodes are associated with labels. Moreover, as mentioned in
Section 2.4, only labels of training nodes can be used as part of the input
features, and the problem of label leakage in label utilization should be well
addressed. We also construct a novel semantic tree structure to organize the
aggregation results and capture the hierarchy of metapaths, which can be
leveraged in semantic tree aggregation (Section 4.3). We next describe the
offline feature and label aggregation as well as semantic tree construction in
detail.

Algorithm 1 The overall process of HetTree

0: Heterogeneous graph
$\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{O},\mathcal{R})$, Node features
$\mathcal{X}$, Node labels $\mathcal{Y}$, Metapaths $\mathcal{P}^{k}$.

0: The predicted labels $Y_{pred}$.

1: /* Offline Feature and Label Aggregation */

2: for $P\in\mathcal{P}^{k}$ do

3: for $v\in\mathcal{V}$ do

4: Feature aggr. $X_{P}^{v}=aggregate(\\{x_{u},\forall
u\in\mathcal{N}_{v}^{P}\\})$

5: if $P$ ends with target node type $O_{tgt}$ then

6: Label aggr. $\hat{Y}_{P}^{v}=aggregate(\\{y_{u},\forall
u\in\mathcal{N}_{v}^{P}\setminus\\{v\\}\\})$

7: end if

8: end for

9: end for

10: /* Metapath Feature Transformation */

11: for $P\in\mathcal{P}^{k}$ do

12: $M_{P}=\begin{array}[]{lcl}MLP(X_{P}\parallel\hat{Y}_{P}),&\mbox{if
}P\in\mathcal{P}_{O_{tgt}}^{k}\\\ MLP(X_{P}),&\mbox{otherwise}\end{array}$

13: end for

14: /* Semantic Tree Aggregation */

15: for $P\in\mathcal{P}^{k}$ do

16: Subtree reference
$S_{P}=MLP(M_{P}\parallel\sum_{Q\in\mathcal{P}^{child}_{P}}M_{Q})$

17: $\alpha_{Q}=\frac{exp(\delta(W_{P}\cdot[S_{P}\parallel
Z_{Q}]))}{\sum_{B\in\mathcal{P}^{child}_{P}}exp(\delta(W_{P}\cdot[S_{P}\parallel
Z_{B}]))}$

18: $Z_{P}=M_{P}+\delta(\sum_{Q\in\mathcal{P}^{child}_{P}}\alpha_{Q}\cdot
Z_{Q})$

19: end for

20:
$Y_{pred}=\begin{array}[]{lc}MLP(\sum_{P\in\mathcal{P}^{k}}Z_{P})+MLP(X_{init})\\\
+MLP(aggregate(\\{\hat{Y}_{P},\forall
P\in\mathcal{P}_{O_{tgt}}^{k}\\}))\end{array}$

####  4.1.1. Feature Aggregation

Figure 2(a) shows the process of offline feature aggregation. Unlike existing
metapath-based methods (HAN, ; MAGNN, ), where feature aggregation is involved
with model learning such as projection and attention, HetTree separates
feature aggregation as a pre-processing stage. Existing methods manually
select metapaths with domain knowledge. The choice saves computation
complexity but also results in information loss. As the feature aggregation
happens offline and involves no parameter learning, which is much less
expensive than existing approaches, we use all metapaths up to hop $k$, where
$k$ is a user-defined parameter. For example, when $k=2$ as shown in Figure
2(a), feature aggregation is conducted on 14 metapaths
($init,O,T,H,OO,OT,OR,OF,TT,TO,TR,TF,HH,HD$). Possible aggregators include but
are not limited to $mean$, $sum$, $max$, and $min$, and we use the $mean$
aggregator in this paper for both feature and label aggregation. For each node
$v$, we compute a set of aggregated features $\mathcal{X}^{v}$:

(1) $\mathcal{X}^{v}=\\{X^{v}_{P}=aggregate(\\{x^{u},\forall
u\in\mathcal{N}^{v}_{P}\\}),\forall P\in\mathcal{P}^{k}\\}$

where $X_{P}^{v}$ represents the aggregated feature for node $v$ along
metapath $P$, and $aggregate$ is the aggregation function ($mean$ by default).

####  4.1.2. Label Aggregation

Figure 2(b) shows the process of offline label aggregation. The label
aggregation process is very similar to the feature aggregation process
described in Section 4.1.1, except for two differences: first, since only
labels from nodes with target node type $O_{tgt}$ (node type to be classified)
in the training set can be used, the label aggregation is only conducted for
metapaths $\mathcal{P}_{O_{tgt}}^{k}$ where they end at node type $O_{tgt}$;
second, feature aggregation applies to all nodes in $\mathcal{N}^{v}_{P}$
including $v$ itself, while $v$ is excluded during label aggregation to avoid
label leakage. For example, when $k=2$ as shown in Figure 2(b), label
aggregation is conducted on 3 metapaths ($OT,TO,HH$) excluding the center
target node, respectively. Note that only labels from nodes in the training
set are used for label aggregation and zero vectors are used for nodes in the
non-training set. Specifically, for each node $v$, we compute a set of
aggregated features $\hat{\mathcal{Y}}^{v}$:

(2) $\begin{split}\hat{y^{v}}&=\left\\{\begin{array}[]{lll}y^{v},&\mbox{if
}v\in training\\_set\\\ \textbf{0},&\mbox{otherwise}\end{array}\right.\\\
\hat{\mathcal{Y}}^{v}&=\\{\hat{Y}^{v}_{P}=aggregate(\\{\hat{y}^{u},\forall
u\in\mathcal{N}_{v}^{P}\setminus\\{v\\}\\}),\forall
P\in\mathcal{P}_{O_{tgt}}^{k}\\}\end{split}$

where $y_{v}$ is the ground truth label for node $v$, $\hat{y}_{v}$ is the
label used in label aggregation for node $v$, and $\hat{Y}_{P}^{v}$ is the
aggregated label along metapath $P$ for node $v$.

####  4.1.3. Semantic Tree Construction

We can construct a semantic tree $T_{O}$ for node type $O$ with tree nodes
$\\{C_{P},\forall P\in\mathcal{P}^{k}\\}$. $C_{init}$ is the root node and a
non-root node $C_{R_{1}R_{2}\cdots R_{l}}$ represents the metapath from the
root node $C_{init}$ to itself via relation edges $R_{1}R_{2}\cdots R_{l}$.
The parent of all 1-hop tree nodes $C_{R_{1}}$ is $C_{init}$, which are at
depth 1 of $T_{O}$. Starting from depth 2, $C_{R_{1}R_{2}\cdots R_{l}}$ is
connected with its parent node $C_{R_{1}R_{2}\cdots R_{l-1}}$ via edge
$R_{l}$. By constructing the semantic tree, the hierarchy among metapaths can
be captured, which provides the model the structural information of the
metapaths, compared to approaches that treat the relationships between
metapaths equally. Moreover, the semantic tree is also used as the underlying
data structure for semantic tree aggregation discussed in Section 4.3, where
the metapath features are aggregated following the tree structure in a bottom-
up way. Figure 2(c) shows an illustration of the semantic tree for the Sender
node type.

###  4.2. Metapath Feature Transformation

After obtaining aggregated features and labels for metapaths, we transform
them to the same latent space. This is due to the aggregated features for
metapaths being generated from row features of metapath-based neighbors with
different node types, which may have different initial spaces. Instead of
separating the transformation of features and labels as in existing methods
(SLE, ; GAMLP, ), HetTree automatically matches and concatenates the
aggregated features and labels of the same metapath $P$ for
$P\in\mathcal{P}_{O_{tgt}}$. This gives the model more accurate label
information of its metapath-based neighbors by designating the aggregated
labels with corresponding metapaths. Specifically, we compute the metapath
features $\mathcal{M}$ as

(3)
$\mathcal{M}=\\{M_{P}=\left\\{\begin{array}[]{lcl}MLP(X_{P}\parallel\hat{Y}_{P}),&\mbox{if
}P\in\mathcal{P}_{O_{tgt}}^{k}\\\
MLP(X_{P}),&\mbox{otherwise}\end{array}\right.,\forall
P\in\mathcal{P}^{k}\\}.$

###  4.3. Semantic Tree Aggregation

As discussed in Section 4.1.3, we construct a semantic tree $T$, and each tree
node $C_{P}$ represents a metapath $P$, where $P\in\mathcal{P}^{k}$. Since we
also obtain metapath features $\mathcal{M}$, we can associate each tree node
$C_{P}$ with $M_{P}$ correspondingly. Note that the semantic tree structure is
the same for all nodes with the same node type in a heterogeneous graph, so
the target nodes (to be classified) can easily be batched. We now have tree-
structured metapath features, and the hierarchical relationship between
metapath features needs to be well modeled when aggregating them.

The tree aggregation in HetTree is conducted in a bottom-up fashion. As it
gets closer and closer to the target node as the process proceeds, the
semantic tree aggregation can gradually emphasize those metapaths that
contribute more to the local subtree structure, i.e., the parent-children
relationship. To calculate the encoded representation $Z_{P}$ for each
metapath node in the semantic tree, HetTree applies a novel subtree attention
mechanism to aggregate the children nodes thus encoding the local subtree
structure. Unlike existing tree encoding methods (TreeLSTM, ; T-GNN, ; SHGNN,
; HetGTCN, ) that use either a simple weighted-sum aggregator or attention
mechanism to emphasize parent tree nodes, HetTree proposes the subtree
attention to encode both the parent and children representation and uses it to
emphasize the hierarchical correlation between metapaths. Specifically, let
$\mathcal{P}^{child}_{P}$ be the set of metapaths that
$\\{C_{Q},Q\in\mathcal{P}^{child}_{P}\\}$ are the set of children nodes of
$C_{P}$ in the semantic tree, HetTree computes a subtree reference as
$S_{P}=MLP(M_{P}\parallel\sum_{Q\in\mathcal{P}^{child}_{P}}M_{Q})$, for each
metapath $P$ in the semantic tree. Then, the weight coefficient $a_{Q}$ of
each children node $C_{Q}$ can be calculated as:

(4) $\alpha_{Q}=\frac{exp(\delta(W_{P}\cdot[S_{P}\parallel
Z_{Q}]))}{\sum_{B\in\mathcal{P}^{child}_{P}}exp(\delta(W_{P}\cdot[S_{P}\parallel
Z_{B}]))}.$

where $\delta$ is the activation function, $W_{P}$ is a learnable projection
vector for matapath $P$ and $\parallel$ stands for concatenation. Then we can
finally compute the encoded representation $Z_{P}$ for parent node $P$ by
aggregating encoded representation of children nodes as:

(5) $Z_{P}=M_{P}+\delta(\sum_{Q\in\mathcal{P}^{child}_{P}}\alpha_{Q}\cdot
Z_{Q}).$

|  | Offline
---
Preprocessing
| Feature
---
Transformation
| Neighbor
---
Aggregation
| Semantic
---
Aggregation
| Total Training
---
Complexity
HAN | - | $\mathcal{O}(nd^{2})$ | $\mathcal{O}(nmpd)$ | $\mathcal{O}(nmd^{2})$ | $\mathcal{O}(nd(mp+md))$
SHGNN | - | $\mathcal{O}(nd^{2})$ | $\mathcal{O}(nmqd)$ | $\mathcal{O}(nmd^{2})$ | $\mathcal{O}(nd(mq+md))$
HetGTCN | - | $\mathcal{O}(nd^{2})$ | $\mathcal{O}(ed)$ | $\mathcal{O}(nmd^{2})$ | $\mathcal{O}(ed+nmd^{2})$
SeHGNN | $\mathcal{O}(hef)$ | $\mathcal{O}(nmd^{2})$ | - | $\mathcal{O}(nm^{2}d^{2})$ | $\mathcal{O}(nm^{2}d^{2})$
HetTree | $\mathcal{O}(hef)$ | $\mathcal{O}(nmd^{2})$ | - | $\mathcal{O}(nmbd^{2})$ | $\mathcal{O}(nmbd^{2})$
Table 2. Time complexity comparison among HAN, SeHGNN and HetTree. $n$, $e$,
$m$, $h$ and $N$ are the number of nodes, edges, metapaths, hops, and target
nodes respectively. $f$ and $d$ are input dimensions and hidden dimensions,
respectively. $p$ is the average number of metapath neighbor nodes and $q$ is
the average total number of nodes along a metapath. $b$ represents the average
number of children nodes in the semantic tree.

After the semantic tree aggregation has finished from bottom to top, the sum
of metapath representation will be used as the final representation of the
semantic tree. Moreover, we add a feature residual and a label residual to
further emphasize the initial features and labels aggregated from the
metapath-based neighbors. Specifically,

(6) $\begin{split}Y_{pred}\quad=&\quad
MLP(\sum_{P\in\mathcal{P}^{k}}Z_{P})+MLP(X_{init})\\\
&\quad+MLP(aggregate(\\{\hat{Y}_{P},\forall
P\in\mathcal{P}_{O_{tgt}}^{k}\\})).\end{split}$

where $aggregate$ is an aggregation function, which can be $mean$, $sum$,
$max$, $min$, etc. An illustration of the semantic tree aggregation process is
shown in Figure 3, following the same example in Figure 2.

###  4.4. Complexity Analysis

Table 2 provides a theoretical analysis of the time complexity for each
component of HetTree, compared with HAN (HAN, ), SHGNN (SHGNN, ), HetGTCN
(HetGTCN, ), SeHGNN (SeHGNN, ), where $n$, $e$, $m$, $h$, and $N$ are the
number of nodes, edges, metapaths, hops, and target nodes, respectively. $f$
and $d$ are input dimensions and hidden dimensions, respectively. Besides, $p$
is the average number of neighbor nodes for each metapath and $q$ is the
average total number of nodes along a metapath. $b$ is the average number of
children nodes for a parent node in the semantic tree.

Note that HAN, SHGNN, and HetGTCN perform multi-layer neighbor aggregation,
while SeHGNN and HetTree simplify neighbor aggregation as an offline
preprocessing process. SHGNN, HetGTCN, and HetTree utilize tree-based data
structure but the former two apply the tree structure on neighborhood
aggregation, i.e., at node level, while the semantic structure in HetTree is
for semantic aggregation, i.e., at metapath level. For each node, SHGNN
constructs a tree structure for each metapath based on the node topology, and
thus all nodes along the metapath take part in neighbor aggregation. HetGTCN
performs the neighbor aggregation for not only the target nodes but also nodes
of other types, where all edges are involved in neighbor aggregation. Since
SeHGNN is a transformer-based model, the self-attention mechanism results in
quadratic complexity ($m^{2}$) with respect to the number of metapaths.
HetTree utilizes the semantic tree structure to perform semantic aggregation,
where the proposed subtree attention mechanism gives $mb$ complexity. In
general, we have $e\geq nmq$ and $q>p\gg m>b$. Hence, HetTree has the lowest
theoretical complexity compared with the baselines.

Figure 3. The overall process of semantic tree aggregation. We aggregate
metapath representations based on our semantic tree structure in a bottom-up
fashion, which gradually emphasizes those metapaths that contribute more to
the local subtree structure, i.e., the parent-children relationship. A novel
subtree attention mechanism is used to compute weight coefficients of children
nodes, and then a weighted sum of them and the parent representation is used
to represent the encoded representation $Z_{P}$ for metapath $P$.

##  5\. Experiments

We conduct extensive experiments on six heterogeneous graphs to answer the
following questions.

Q1.:

How does HetTree compare to the state-of-the-art overall on open benchmarks?

Q2.:

How does HetTree perform in a practical compromised account detection task on
a noisy real-world email graph?

Q3.:

How does each component of HetTree contribute to the performance gain?

Q4.:

Is HetTree practical w.r.t. running time and memory usage?

Experimental Setup. All of the experiments were conducted on a computer with
dual 12-core Intel Xeon Gold 6226 CPU, 384 GB of RAM, and one NVIDIA Tesla
A100 80GB GPU. The server runs 64-bit Red Hat Enterprise Linux 7.6 with CUDA
library v11.8, PyTorch v1.12.0, and DGL v0.9.

| DBLP | IMDB | ACM | Freebase
---|---|---|---|---
| Macro-F1 | Micro-F1 | Macro-F1 | Micro-F1 | Macro-F1 | Micro-F1 | Macro-F1 | Micro-F1
RGCN |  91.52 $\pm$ 0.50 |  92.07 $\pm$ 0.50 |  58.85$\pm$0.26 |  62.05$\pm$0.15 |  91.55$\pm$0.74 |  91.41$\pm$0.75 |  46.78$\pm$0.77 |  58.33$\pm$1.57
HAN |  91.67$\pm$0.49 |  92.05$\pm$0.62 |  57.74$\pm$0.96 |  64.63$\pm$0.58 |  90.89$\pm$0.43 |  90.79$\pm$0.43 |  21.31$\pm$1.68 |  54.77$\pm$1.40
HetGNN |  91.76$\pm$0.43 |  92.33$\pm$0.41 |  48.25$\pm$0.67 |  51.16$\pm$0.65 |  85.91$\pm$0.25 |  86.05$\pm$0.25 | - | -
MAGNN |  93.28$\pm$0.51 |  93.76$\pm$0.45 |  56.49$\pm$3.20 |  64.67$\pm$1.67 |  90.88$\pm$0.64 |  90.77$\pm$0.65 | - | -
HGT |  93.01$\pm$0.23 |  93.49$\pm$0.25 |  63.00$\pm$1.19 |  67.20$\pm$0.57 |  91.12$\pm$0.76 |  91.00$\pm$0.76 |  29.28$\pm$2.52 |  60.51$\pm$1.16
HGB |  94.01$\pm$0.24 |  94.46$\pm$0.22 |  63.53$\pm$1.36 |  67.36$\pm$0.57 |  93.42$\pm$0.44 |  93.35$\pm$0.45 |  47.72$\pm$1.48 |  66.29$\pm$0.45
SeHGNN |  95.06$\pm$0.17 |  95.42$\pm$0.17 |  67.11$\pm$0.25 |  69.17$\pm$0.43 |  94.05$\pm$0.35 |  93.98$\pm$0.36 |  51.87$\pm$0.86 |  65.08$\pm$0.66
HetTree | 95.34$\pm$0.17 | 95.64$\pm$0.15 | 68.43$\pm$0.31 | 70.92$\pm$0.29 | 94.26$\pm$0.20 | 94.19$\pm$0.20 | 52.35$\pm$0.96 | 66.39$\pm$0.40
Table 3. Experimental Results of HetTree and baselines over four graphs in the
HGB benchmark. "-" means that the model runs out of memory on the
corresponding graph.

Datasets. We evaluate HetTree on four graphs from the HGB (HGB, ) benchmark:
DBLP, IMDB, ACM, and Freebase, a citation graph ogbn-mag from the OGB
benchmark (OGB, ) and a real-world email dataset collected from a commercial
email platform. We summarize the six graphs in Table 4.

Graph | #Nodes | #Edges |  | #Node
---
Types
| #Relation
---
Types
#Classes
DBLP | 26,128 | 239,566 | 4 | 6 | 4
IMDB | 21,420 | 86,642 | 4 | 6 | 5
ACM | 10,942 | 547,872 | 4 | 8 | 3
Freebase | 180,098 | 1,057,688 | 8 | 36 | 7
ogbn-mag | 1,939,743 | 21,111,007 | 4 | 5 | 349
Email | 7,864,303 | 34,943,782 | 5 | 6 | 2
Table 4. Statistics of datasets.

Baselines. For the four graphs from the HGB benchmark, we compare the HetTree
results to the results reported in the HGB paper (HGB, ) as well as a state-
of-the-art work SeHGNN (SeHGNN, ). For ogbn-mag, we compare the HetTree with
top-performing methods from either the baseline paper or the leaderboard of
OGB (OGB, ). For the email dataset, we compare HetTree with the best-
performing baseline SeHGNN. All experimental results reported are averaged
over five random seeds.

Ethics and Broader Impacts. This work was reviewed and approved by independent
experts in Ethics, Privacy, and Security. For the email dataset, all users’
identities were anonymized twice, and the map from the second anonymized user
IDs to the first anonymized user IDs was deleted. Furthermore, the data was
handled according to GDPR regulations. In addition, the data was secured to
protect against data leakage and the data flow was approved by a privacy
manager. Since the goal of the model is to help protect email users from
attacks, we believe there are no potential negative, societal impacts.

###  5.1. Experiments on Open Benchmarks

To answer Q1, we compare the performance of the proposed HetTree to state-of-
the-art models on five heterogeneous graphs from two open benchmarks - HGB
(HGB, ) and OGB (OGB, ).
Performance on HGB Benchmark. Table 3 shows results that compare HetTree with
best-performing baselines on four datasets from the HGB benchmark. HetTree
outperforms the baselines on all graphs in terms of both Macro-F1 and Micro-F1
scores. We notice that HetTree benefits more from its semantic tree
aggregation, which encodes the hierarchy among metapaths, on more complex
tasks with more classes. As shown in Table 3, HetTree has more performance
gain on IMDB and Freebase with 5 and 7 classes, respectively, compared with
DBLP and ACM with 3 and 4 classes, respectively. This can be attributed to
HetTree’s semantic tree aggregation that learns more information, i.e., the
hierarchy among metapaths, which is ignored by the other baselines.

Performance on Ogbn-Mag Dataset. We also evaluate HetTree on a large-scale
citation graph, ogbn-mag (OGB, ), with millions of nodes in Table 5. We report
results without self-enhanced techniques like multi-stage training
(li2018deeper, ; sun2020multi, ; LEGNN, ), which are orthogonal to HetTree and
can be incorporated for additional benefits. The results show that HetTree
maintains its benefits with large graphs and outperforms all baselines.

Methods | Validation Accuracy | Test Accuracy
---|---|---
RGCN | 48.35 $\pm$ 0.36 | 47.37 $\pm$ 0.48
HGT | 51.24 $\pm$ 0.46 | 49.82 $\pm$ 0.13
NARS | 53.72 $\pm$ 0.09 | 52.40 $\pm$ 0.16
LEGNN | 54.43 $\pm$ 0.09 | 52.76 ± 0.14
GAMLP | 55.48 $\pm$ 0.08 | 53.96 $\pm$ 0.18
SeHGNN | 56.56 $\pm$ 0.07 | 54.78 $\pm$ 0.17
HetTree | 57.31 $\pm$ 0.15 | 55.54 $\pm$ 0.17
Table 5. Detection accuracy of the HetTree model and other baselines for the
ogbn-mag dataset.

###  5.2. Experiments on Commercial Email Graph

Besides the open benchmark datasets, we also collect a large email dataset
from a commercial email platform to answer Q2. In this experiment, a subsample
of real-world email data is used, which contains five types of entities -
senders (1,169,011 nodes), recipients (1,020,982 nodes), domains (686,787
nodes), IP addresses (296,939 nodes), and messages (4,690,584 nodes). The task
is to predict if the sender is legitimate or compromised, given its domain,
messages, recipients of the message, recipients’ domains, and message IP
addresses, where a compromised account may send several types of malicious
emails including phishing emails, malware attachments, and spam.

Since the email dataset contains binary labels (i.e., legitimate and
compromised), we can construct a Receiver Operating Characteristic (ROC) curve
for the models. The ROC curve for the detection of compromised emails is
presented in Figure 4 for the email dataset, and the accuracies are shown in
Table 6. Again, HetTree significantly outperforms the best-performing baseline
SeHGNN (SeHGNN, ).

Figure 4. ROC Curves for the email dataset. The error bars for HetTree are
tiny.

Methods | Val Accuracy | Test Accuracy
---|---|---
SeHGNN | 97.22 $\pm$0.02 | 97.26 $\pm$ 0.02
HetTree | 98.48$\pm$0.01 | 98.48 $\pm$ 0.02

Table 6. Detection accuracy for the email dataset.

###  5.3. Ablation Study

We here evaluate whether adding the subtree attention component and using
labels from the training set really help or not to answer Q3. The test F1
scores or accuracy of HetTree is evaluated on IMDB, ACM, and Ogbn-Mag compared
with its three variants: "weighted-sum", "parent-att" and "no-label". Variant
weighted-sum removes the proposed subtree attention but uses a weighted child-
sum like TreeLSTM (TreeLSTM, ). Variant parent-att removes the proposed
subtree attention but computes weights of children nodes using attention on
the parent node, which is the tree-encoding method used in SHGNN (SHGNN, ).
Variant no-label does not use labels as extra inputs.

The results in Table 7 show that each component is effective for HetTree. We
notice that datasets exhibit distinct sensitivities to individual components.
The subtree attention results in more performance gain on IMDB, which could be
attributed to the sparsity of the graph. It demonstrates that subtree
attention can capture the metapath hierarchy, compared to other tree encoding
mechanisms. Ogbn-Mag is more sensitive to label utilization, which could be
attributed to the large number of classes of labels that provide richer
information through propagation.

|  IMDB |  ACM |  Ogbn-Mag
---|---|---|---
|  Micro-F1 |  Micro-F1 |  Accuracy
HetTree |  70.92$\pm$0.29 |  94.19$\pm$0.20 |  55.54$\pm$0.17
weighted-sum |  69.70$\pm$0.24 |  93.54$\pm$0.23 |  55.38$\pm$0.21
parent-att |  69.69$\pm$0.41 |  93.83$\pm$0.18 |  54.87$\pm$0.26
no-label |  70.11$\pm$0.25 |  93.41$\pm$0.23 |  52.93$\pm$0.12

Table 7. Effectiveness of each component of HetTree.

0.0$\times$0.5$\times$1.0$\times$1.5$\times$2.0$\times$2.5$\times$3.0$\times$3.5$\times$4.0$\times$4.5$\times$DBLPIMDBACMEpoch
TimeDBLPIMDBACM0G2G4G6G8G10GMemory Usage

Normalized Epoch Time

Memory (GB)

HANHGBSeHGNNHetTree

Figure 5. Average epoch time and memory usage on HGB datasets.

###  5.4. Computation Cost Comparison

We next investigate the computational cost of HetTree in terms of epoch time
and memory footprint to answer Q4. We select three performant models - HAN
(HAN, ), HGB (HGB, ), and SeHGNN (SeHGNN, ) \- to compare with HetTree on four
graphs in the HGB benchmark. For fair comparison, we use a 2-layer structure
for HAN and HGN, and 2-hop feature propagation for SeHGNN and HetTree. The
result in Figure 5 shows that HetTree incurs the lowest computational cost in
terms of both running time and memory usage across three datasets from the HGB
benchmark.

##  6\. Conclusion and Discussion

In this paper, we present HetTree, a novel Graph Neural Network for
heterogeneous graphs. HetTree is based on the observation that existing HGNNs
ignore a tree hierarchy among metapaths, which is naturally constituted by
different node types and relation types. HetTree builds a semantic tree
structure to capture the hierarchy among metapaths and proposes a novel
subtree attention mechanism to encode the resulting semantic tree. Compared
with existing tree-encoding techniques that weight the contribution of
children nodes based on similarity to the parent node, subtree attention in
HetTree can model the broader local structure of parent nodes and children
nodes. The evaluation shows that HetTree is able to outperform state-of-the-
art baselines on open benchmarks and efficiently scale to large real-world
graphs with millions of nodes and edges. A future direction is to generalize
the semantic tree structure to not only scalable HGNNs but also HGNNs with
multi-layer aggregation, which performs a more fine-grained encoding on
semantics and the semantic tree structure could bring additional benefits.

## References

  * [1] Jie Chen, Tengfei Ma, and Cao Xiao.  FastGCN: Fast learning with graph convolutional networks via importance sampling.  In International Conference on Learning Representations, 2018.
  * [2] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst.  Convolutional neural networks on graphs with fast localized spectral filtering.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, pages 3844–3852. Curran Associates, Inc., 2016.
  * [3] Fabrizio Frasca, Emanuele Rossi, Davide Eynard, Ben Chamberlain, Michael Bronstein, and Federico Monti.  Sign: Scalable inception graph neural networks, 2020.
  * [4] Xinyu Fu, Jiani Zhang, Ziqiao Meng, and Irwin King.  Magnn: Metapath aggregated graph neural network for heterogeneous graph embedding.  In Proceedings of The Web Conference 2020, WWW ’20, page 2331–2341, New York, NY, USA, 2020. Association for Computing Machinery.
  * [5] Will Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 1024–1034. Curran Associates, Inc., 2017.
  * [6] William L. Hamilton, Rex Ying, and Jure Leskovec.  Representation Learning on Graphs: Methods and Applications.  IEEE Data Engineering Bulletin, page arXiv:1709.05584, September 2017.
  * [7] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec.  Open graph benchmark: Datasets for machine learning on graphs.  Advances in neural information processing systems, 33:22118–22133, 2020.
  * [8] Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun.  Heterogeneous graph transformer.  In Proceedings of The Web Conference 2020, WWW ’20, page 2704–2710, New York, NY, USA, 2020. Association for Computing Machinery.
  * [9] Qimai Li, Zhichao Han, and Xiao-Ming Wu.  Deeper insights into graph convolutional networks for semi-supervised learning.  In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
  * [10] Qingsong Lv, Ming Ding, Qiang Liu, Yuxiang Chen, Wenzheng Feng, Siming He, Chang Zhou, Jianguo Jiang, Yuxiao Dong, and Jie Tang.  Are we really making much progress? revisiting, benchmarking and refining heterogeneous graph neural networks.  In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1150–1160, 2021.
  * [11] Aditya Pal, Chantat Eksombatchai, Yitong Zhou, Bo Zhao, Charles Rosenberg, and Jure Leskovec.  Pinnersage: Multi-modal user embedding framework for recommendations at pinterest.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, pages 2311–2320, New York, NY, USA, 2020. Association for Computing Machinery.
  * [12] Namyong Park, Andrey Kan, Xin Luna Dong, Tong Zhao, and Christos Faloutsos.  Estimating node importance in knowledge graphs using graph neural networks.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19, pages 596–606, New York, NY, USA, 2019. Association for Computing Machinery.
  * [13] Ziyue Qiao, Pengyang Wang, Yanjie Fu, Yi Du, Pengfei Wang, and Yuanchun Zhou.  Tree structure-aware graph representation learning via integrated hierarchical aggregation and relational metric learning.  In 2020 IEEE International Conference on Data Mining (ICDM), pages 432–441, 2020.
  * [14] Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling.  Modeling relational data with graph convolutional networks, 2017.
  * [15] Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun.  Masked label prediction: Unified message passing model for semi-supervised classification, 2020.
  * [16] Chuxiong Sun, Hongming Gu, and Jie Hu.  Scalable and adaptive graph neural networks with self-label-enhanced training, 2021.
  * [17] Ke Sun, Zhouchen Lin, and Zhanxing Zhu.  Multi-stage self-supervised learning for graph convolutional networks on graphs with few labeled nodes.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 5892–5899, 2020.
  * [18] Ruoxi Sun, Hanjun Dai, and Adams Wei Yu.  Does GNN pretraining help molecular representation?  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
  * [19] Kai Sheng Tai, Richard Socher, and Christopher D Manning.  Improved semantic representations from tree-structured long short-term memory networks.  arXiv preprint arXiv:1503.00075, 2015.
  * [20] Théo Trouillon, Christopher R. Dance, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard.  Knowledge graph completion via complex tensor factorization, 2017.
  * [21] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.  Graph attention networks.  In International Conference on Learning Representations, 2018.
  * [22] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu.  Heterogeneous graph attention network.  In The World Wide Web Conference, WWW ’19, page 2022–2032, New York, NY, USA, 2019. Association for Computing Machinery.
  * [23] Yangkun Wang, Jiarui Jin, Weinan Zhang, Yong Yu, Zheng Zhang, and David Wipf.  Bag of tricks for node classification with graph neural networks, 2021.
  * [24] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger.  Simplifying graph convolutional networks.  In International conference on machine learning, pages 6861–6871. PMLR, 2019.
  * [25] Nan Wu and Chaofan Wang.  Heterogeneous graph tree networks, 2022.
  * [26] Wentao Xu, Yingce Xia, Weiqing Liu, Jiang Bian, Jian Yin, and Tie-Yan Liu.  Shgnn: Structure-aware heterogeneous graph neural network, 2021.
  * [27] Xiaocheng Yang, Mingyu Yan, Shirui Pan, Xiaochun Ye, and Dongrui Fan.  Simple and efficient heterogeneous graph neural network.  In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’23/IAAI’23/EAAI’23. AAAI Press, 2023.
  * [28] Le Yu, Leilei Sun, Bowen Du, Tongyu Zhu, and Weifeng Lv.  Label-enhanced graph neural network for semi-supervised node classification, 2022.
  * [29] Lingfan Yu, Jiajun Shen, Jinyang Li, and Adam Lerer.  Scalable graph neural networks for heterogeneous graphs.  CoRR, abs/2011.09679, 2020.
  * [30] Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V. Chawla.  Heterogeneous graph neural network.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &; Data Mining, KDD ’19, page 793–803, New York, NY, USA, 2019. Association for Computing Machinery.
  * [31] Wentao Zhang, Ziqi Yin, Zeang Sheng, Yang Li, Wen Ouyang, Xiaosen Li, Yangyu Tao, Zhi Yang, and Bin Cui.  Graph attention multi-layer perceptron.  In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, page 4560–4570, New York, NY, USA, 2022. Association for Computing Machinery.
  * [32] Hao Zhu and Piotr Koniusz.  Simple spectral graph convolution.  In International Conference on Learning Representations, 2020.
  * [33] Xiaojin Zhu Ѓ and Zoubin GhahramaniЃн.  Learning from labeled and unlabeled data with label propagation.  2002\.
  * [34] Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu.  Layer-dependent importance sampling for training deep and large graph convolutional networks.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, pages 11249–11259. Curran Associates, Inc., 2019.
  * [35] Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu.  Layer-dependent importance sampling for training deep and large graph convolutional networks.  Advances in neural information processing systems, 32, 2019.


June-WooKim∗1,2 MiikaToikkanen2 YeraChoi3 Seoung-EunMoon3† Ho-YoungJung1†

# BTS: Bridging Text and Sound Modalities for Metadata-Aided Respiratory Sound
Classification

###### Abstract

Respiratory sound classification (RSC) is challenging due to varied acoustic
signatures, primarily influenced by patient demographics and recording
environments. To address this issue, we introduce a text-audio multimodal
model that utilizes metadata of respiratory sounds, which provides useful
complementary information for RSC. Specifically, we fine-tune a pretrained
text-audio multimodal model using free-text descriptions derived from the
sound samples' metadata which includes the gender and age of patients, type of
recording devices, and recording location on the patient's body. Our method
achieves state-of-the-art performance on the ICBHI dataset, surpassing the
previous best result by a notable margin of 1.17%. This result validates the
effectiveness of leveraging metadata and respiratory sound samples in
enhancing RSC performance. Additionally, we investigate the model performance
in the case where metadata is partially unavailable, which may occur in real-
world clinical setting.

###### keywords:

Respiratory Sound Classification, Pretrained Language-Audio Model, ICBHI,
Metadata

##  1 Introduction

Identifying abnormal respiratory sounds is pivotal for diagnosing and
providing timely interventions for respiratory conditions. Automated detection
of abnormal respiratory sounds has great potential to improve health and
quality of life for those affected by respiratory diseases by identifying
risks early and expediting first aid for potentially life-threatening
conditions, such as pneumonia or chronic obstructive pulmonary disease.
Machine learning approaches have been regarded as a promising way for
automated detection of abnormal respiratory sounds. Recently, a number of
studies [1, 2, 3, 4, 5, 6, 7, 8, 9] have tackled the respiratory sound
classification (RSC) task and notably increased the performance by utilizing
models that have been pretrained on large non-medical datasets [10, 11], and
then fine-tuned on a respiratory sound dataset [12].

Nevertheless, the inherent heterogeneity of respiratory sound data presents an
obstacle to further performance improvement in RSC. The heterogeneity arises
from differences in patient demographics, recording devices, and environmental
conditions, which can significantly impact the acoustic properties of
respiratory sounds [1]. This may lead to poor generalization on unseen data,
particularly in cases underrepresented by the training data. ICBHI [12], one
of the widely adopted respiratory sound datasets, provides metadata that
associates the recorded audio with attributes of patients and recording
environments. Such metadata may be useful for addressing difficulties caused
by heterogeneity.

Some previous work has adapted the metadata associated with respiratory sound
for RSC to mitigate the heterogeneity issue. For instance, incorporating
demographic information of patients such as age and gender into the
pretraining process provides better representations of respiratory audio
samples [5]. Moreover, metadata concerning the recording environment (i.e.,
stethoscope) also provides useful information. SG-SCL [1] employed domain-
transfer techniques to reduce the effect of heterogeneity by regarding
different types of recording devices as distinct domains. Despite the
potential benefits of leveraging the metadata, these previous works did not
fully incorporate it as text data into the model inputs.

Recent developments in multimodal models, exemplified by Contrastive Language-
Image Pretraining (CLIP) [13] for text and image data and Contrastive
Language-Audio Pretraining (CLAP) [14, 15] for text and audio data, offer a
flexible framework for integrating text data with non-textual data. Several
studies [16, 17, 18, 19] have demonstrated the effectiveness of language-EEG
multimodal models for the sentiment classification and EEG-to-text decoding
tasks. Recognizing the success of multimodal models and the demonstrated
benefits of multimodal data in healthcare tasks, it is compelling to consider
them for RSC, where such method has not yet been explored.

In this paper, we take a step into a new direction and fully make use of the
respiratory audio metadata by adapting a text-audio multimodal model, aiming
not only to leverage the metadata as an additional learning signal, but to
benefit from the further context during the inference stage. Building on the
foundation of contrastive language-audio pretrained models, our work
incorporates the respiratory audio metadata alongside the sound recordings. To
this end, we format the patient's metadata into descriptions derived from key
attributes including age, gender, recording device, and recording location on
the body, and encode them with respiratory sound data into shared feature
representation by the pretrained encoders. With these joint representations,
we train a classification head for the RSC task.

Figure 1: An overall illustration of the proposed BTS architecture. The
pretrained text and audio encoders extract feature representations of text
description derived from metadata and respiratory sound samples, respectively.
After the projection, the representations are integrated by a concatenation
operation and used for RSC.

Our approach, which we name the _BTS_ (_B_ ridging the _T_ ext and _S_ ound
modalities), a method that leverages multimodal text-audio model to fully
exploit the potential of respiratory audio metadata, achieves the state-of-
the-art (SOTA) result on the ICBHI dataset, outperforming upon the previous
best [4] by 1.17%. Our results reveal the capability of contrastive language-
audio pretraining to improve RSC both in audio-only and multimodal settings.
Moreover, we demonstrate that our method retains its performance gains in the
absence of metadata during the inference. This result suggests that our
approach can be adopted for practical clinical settings where additional
information other than audio signals may be unavailable. Our main
contributions are as follows:

  * •

We show that leveraging metadata of respiratory sounds improves the RSC
performance. Our approach sets the new SOTA performance on the ICBHI dataset.

  * •

We thoroughly explore ways to utilize the metadata considering a real clinical
setting where the type of metadata differs from the expectation, or the
metadata is partially or totally unavailable. We demonstrate that our method
robustly performs RSC in such scenarios.

  * •

We analyze how the different types of metadata affect the performance of our
model. Our result shows that the information about the recording environment,
such as the type of recording stethoscopes and recording locations of the
human body are particularly helpful in minimizing the effect of heterogeneity
in respiratory sounds.

##  2 Method

We introduce _B_ ridging _T_ ext and _S_ ound modalities (BTS), an approach
that leverages multimodal text-audio model to fully exploit the potential of
respiratory audio metadata. To mitigate the heterogeneity of respiratory
sounds, we propose to explicitly utilize the metadata, which we expect to
capture the significant sources of acoustic variability. By integrating this
metadata, we aim to reduce the heterogeneity issue and improve the RSC
performance. Toward this goal, we propose the adoption of a multimodal text-
audio model for RSC, as depicted in Figure 1.

###  2.1 CLAP Model

While the metadata of respiratory sounds can be employed for RSC in several
different ways, a free text format is flexible and easily applicable to human-
produced data such as medical records. For instance, the metadata can be
described by a vector of numeric values where each element indicates different
metadata attributes. However, this approach is usually vulnerable to changes
in input sources, such as missing data and unseen data types. In contrast,
encoders for free text data are trained to understand the given input, which
makes approaches utilizing input data in a free text format robust to the
changes. For this reason, we use CLAP (Contrastive Language-Audio Pretraining)
[15] as our starting point. The CLAP model includes both text and audio
encoders, which are trained on the large-scale LAION-Audio-630K [15] dataset
including diverse audio data.

Given the text and audio data denoted as $X_{i}^{t}$ and $X_{i}^{a}$ where
$i\in[1,N]$ indicates the data index within a batch of size $N$, CLAP
processes the text and audio data independently through dedicated encoders
$f_{t}(\cdot)$ and $f_{a}(\cdot)$ for each modality. The embedding vectors
produced by the encoders are projected onto a $d$-dimensional shared embedding
space through projection layers $h_{t}(\cdot)$ and $h_{a}(\cdot)$.

$\displaystyle z_{t}$ $\displaystyle=h_{t}(f_{t}(X_{i}^{t})),$ $\displaystyle
z_{a}$ $\displaystyle=h_{a}(f_{a}(X_{i}^{a})).$ (1)

The CLAP model is trained to maximize the similarity between the text and
audio embeddings by contrasting them with negative samples (i.e., mismatched
text or audio embeddings obtained from $X_{j\in[1,N];j\neq i}^{t}$ or
$X_{j\in[1,N];j\neq i}^{a}$).

Table 1: Examples of generated text descriptions derived from metadata. `All'
is the case that includes all attributes: age, sex, recording location, and
recording device.

Metadata | Generated text descriptions
---|---
Age | This patient is an adult patient.
Sex | This patient is a male patient.
Loc | This sound was recorded from the left anterior chest.
Dev | This sound was recorded with a Meditron stethoscope.
Age-Loc-Dev | This sound was recorded from the left anterior chest
| of an adult patient, using a Meditron stethoscope.
……… | ………
All | This sound was recorded from the left anterior chest
| of an adult male patient, using a Meditron stethoscope.

###  2.2 Text Description Generation for Metadata

Among the metadata available in the ICBHI [12] dataset, we choose four types
of data as follows: age (adult or pediatric) and gender (male or female) of
patients, recording location on the chest of the patients (trachea, anterior
left, anterior right, posterior left, posterior right, lateral left, or
lateral right), and type of recording devices (Meditron, LittC2SE, Litt3200,
or AKGC417L). Using the attributes, we construct simple text descriptions. A
generated description can include any combination of the attributes, totaling
644 unique texts. Table 1 illustrates a few examples with different
combinations of metadata.

###  2.3 Bridging Text and Sound Modalities

As shown in Figure 1, we train the text and audio encoders of CLAP for RSC by
using the respiratory sound samples and generated text descriptions. For
classification, we concatenate text and audio representations $z_{t}$ and
$z_{a}$ from text and audio pipelines as described in Figure 1. Consequently,
we can obtain the multimodal combined representations $z=concat(z_{t},z_{a})$
where $z\in\mathbb{R}^{N\times 2d}$. We then simply add a 4-dimensional linear
layer for classifier $g(\cdot)$ followed by softmax function and train it with
the Cross-Entropy loss $\mathcal{L}_{\text{CE}}$ (division by $N$ is omitted):

$\displaystyle\mathcal{L}_{\text{CE}}=-\sum_{i=1}^{n}\\!\,y_{i}\\!\,\log\,\\!(\hat{y_{i}}),$
(2)

where $n$ is number of samples, $y$ is the respiratory sound label
$\in\\{$normal, crackle, wheeze, both$\\}$, and $\hat{y}$ is the predicted
probabilities obtained by the classifier.

##  3 Experimental Setup

###  3.1 Dataset

We utilized the ICBHI Respiratory dataset [12]. The dataset contains a total
of approximately 5.5 hours of respiratory sound recordings with pre-defined
and balanced splits for training (60%) and test (40%) without patient overlap.
There are 4,142 training and 2,756 testing respiratory cycles across four
classes. Table 2 illustrates the details of the ICBHI dataset. We binarize the
age as the adult (over 18 years old) or pediatric (18 years old or under) for
simplicity. Other than the age, we follow the metadata information of the
official ICBHI records. Body mass index (BMI) data, which was provided only
for adult patients, are solely employed for further analysis. For non-adults,
we calculated it using their weight and height data.

###  3.2 Training Details

Following the data pre-processing described in [1, 3, 4, 9], we extracted the
respiratory cycles from the waveform samples and standardized them to have a
duration of 8 seconds. We then conducted resampling to 48kHz to match the
pretraining data of CLAP. We employed the CLAP [15] model pretrained on the
LAION-Audio-630K [15] dataset for all experiments. The maximum length of the
text descriptions is limited to 64 tokens, which was sufficient for avoiding
truncation of text. We fine-tuned the models using the Adam optimizer [20]
with an initial learning rate of 5e–5. The learning rate was adjusted by
cosine scheduling through a total of 50 epochs of training with a batch size
of 8. To reduce the impact of random initialization, we conducted the
experiments with five different random seeds.

###  3.3 Metrics

We adapt the _Specificity_ ($S_{p}$), _Sensitivity_ ($S_{e}$), and their
average ($Score$) as performance metrics for RSC, following the definitions in
[12]. All reported values of $S_{p}$, $S_{e}$, and $Score$ are the mean and
variance from the five runs with different seeds.

###  3.4 Baselines

We compare the proposed method with previous studies including the current
SOTA method [4], which uses Audio Spectrogram Transformer (AST) [21] as a
backbone model. We also consider the result based solely on the audio
embedding of CLAP ($z_{a}$ in Equation (1)) as an additional baseline, which
we denote as Audio-CLAP.

Table 2: Details of the ICBHI dataset including the number of audio samples
for each class and the types of metadata. L/R stands for left or right.

| Label | Train | Test | Sum
---|---|---|---|---
Lung Sound | Normal | 2,063 | 1,579 | 3,642
Crackle | 1,215 | 649 | 1,864
Wheeze | 501 | 385 | 886
Both | 363 | 143 | 506
| Type | Metadata Label
Metadata | Age | Adult, Pediatric
Sex | Male, Female
Location | Trachea, L/R Anterior, L/R Posterior, L/R Lateral
Stethoscope | Meditron, LittC2SE, Litt3200, AKGC417L
| Others | BMI (Adult only), Weight/Height (Pediatric only)

##  4 Results

###  4.1 Main Results

Table 3: The RSC performance on the ICBHI dataset with the official 60–40%
train–test split. Here, in the Pretraining Data column, IN, AS, and LA refer
to ImageNet [10], AudioSet [11], and LAION-Audio-630K [15], respectively. $*$
denotes the previous state-of-the-art ICBHI Score. The Best and second best
results are highlighted by the bold characters and underlines.

Method | Backbone | Pretraining Data | Venue |  $S_{p}$ (%) |  $S_{e}$ (%) |  Score (%)
---|---|---|---|---|---|---
SE+SA [22] | ResNet18 | - | INTERSPEECH`20 | 81.25 | 17.84 | 49.55
LungRN+NL [23] | ResNet-NL | - | INTERSPEECH`20 | 63.20 | 41.32 | 52.26
RespireNet [9] (CBA+BRC+FT) | ResNet34 | IN | EMBC`21 | 72.30 | 40.10 | 56.20
Chang et al. [24] | CNN8-dilated | - | INTERSPEECH`22 | 69.92 | 35.85 | 52.89
Ren et al. [25] | CNN8-Pt | - | ICASSP`22 | 72.96 | 27.78 | 50.37
Wang et al. [8] (Splice) | ResNeSt | IN | ICASSP`22 | 70.40 | 40.20 | 55.30
Late-Fusion [7] | Inc-03 + VGG14 | IN | EMBC`22 | 85.60 | 30.00 | 57.30
Nguyen et al. [6] (StochNorm) | ResNet50 | IN | TBME`22 | 78.86 | 36.40 | 57.63
Nguyen et al. [6] (CoTuning) | ResNet50 | IN | TBME`22 | 79.34 | 37.24 | 58.29
Moummad et al. [5] | CNN6 | AS | WASPAA`23 | 70.09 | 40.39 | 55.24
Moummad et al. [5] (SCL) | CNN6 | AS | WASPAA`23 | 75.95 | 39.15 | 57.55
Bae et al. [4] (Fine-tuning) | AST | IN + AS | INTERSPEECH`23 | 77.14 | 41.97 | 59.55
Bae et al. [4] (Patch-Mix CL) | AST | IN + AS | INTERSPEECH`23 | 81.66 | 43.07 | $\text{62.37}^{\textbf{*}}$
Kim et al. [3] (AFT on Mixed-500) | AST | IN + AS | NeurIPSW`23 | 80.72 | 42.86 | 61.79
Kim et al. [1] (SG-SCL) | AST | IN + AS | ICASSP`24 | 79.87 | 43.55 | 61.71
Kim et al. [2] (RepAugment) | AST | IN + AS | EMBC`24 | 82.47 | 40.55 | 61.51
Audio-CLAP [ours] | CLAP | LA | INTERSPEECH`24 | $\text{{80.85}}_{\pm 3.33}$ |<EMAIL_ADDRESS>3.77}$ |<EMAIL_ADDRESS>0.37}$
BTS [ours] | CLAP | LA | INTERSPEECH`24 | $\text{81.40}_{\pm 2.57}$ | $\text{{45.67}}_{\pm 2.66}$ | $\text{{63.54}}_{\pm 0.80}$

Table 4: A comparison of the ICBHI Scores between the Audio-CLAP baseline and
BTS. The results are shown depending on the metadata classes. Note that there
is no sample of the LittC2SE in the test set. The bold characters and
underlines indicate the best and second best Score improvement.

Metadata |  Method |  Score
---|---|---
Type |  Class |  Ratio (%) |  BTS |  Audio-CLAP |  Difference
Age |  Adult |  85.70 |  64.53 |  61.67 |  2.86
Pediatric |  14.30 |  64.53 |  61.99 |  2.54
Sex |  Male |  78.74 |  64.53 |  62.00 |  2.53
Female |  21.26 |  64.46 |  61.92 |  2.54
Loc |  Trachea |  11.97 |  64.46 |  61.92 |  2.54
Left Anterior |  21.99 |  64.52 |  61.66 |  2.86
Right Anterior |  9.51 |  64.78 |  61.58 |  3.20
Left Posterior |  22.57 |  64.54 |  62.00 |  2.54
Right Posterior |  15.64 |  65.31 |  61.45 |  3.86
Left Lateral |  9.43 |  64.41 |  61.83 |  2.58
Right Lateral |  8.89 |  60.21 |  54.44 |  5.77
Dev |  Meditron |  16.65 |  64.54 |  62.00 |  2.54
LittC2SE |  0.0 |  - |  - |  -
Litt3200 |  16.73 |  64.52 |  61.65 |  2.87
AKGC417L |  66.62 |  64.78 |  61.53 |  3.25

Table 3 presents comprehensive ICBHI results including our method. Our method
achieves a new SOTA by 1.17% improvement from the previous best without using
any additional training techniques that other methods rely on, such as
stethoscope-specific fine-tuning [9], co-tuning [6], Patch-Mix augmentation
[4], or domain adaptation techniques [1, 3]. Particularly, it is noteworthy
that our method has a considerably higher sensitivity ($S_{e}$) than the
previous best model while maintaining a similar specificity ($S_{p}$). We
consider that the additional context from textual descriptions enhances the
model's ability to correctly identify positive cases, without increasing the
false positive ratio. Additionally, the improvement of Audio-CLAP over the
previous SOTA highlights the effectiveness of the contrastively language-audio
pretrained encoder as a strong baseline for audio-only related tasks, where
the encoder is pretrained using text descriptions as opposed to previous
backbone models for RSC that employed categorical audio labels for
pretraining.

Table 4 compares the performance of BTS and Audio-CLAP for each metadata
category within the ICBHI test set. Although the dataset is notably imbalanced
for the metadata categories, our model consistently surpasses the Audio-CLAP
baseline across all classes. It is also noteworthy that a notable enhancement
is observed in minority classes. Specifically, the result of the test samples,
of which _Loc_ is the right lateral chest, yields the Score increase of 5.77%.
This underscores the value of the metadata in not only the overall performance
improvement but also the effectiveness of accounting for underrepresented
categories.

###  4.2 Influence of Metadata on Classification Performance

Table 5: Results of the ablation study with different combinations of the
metadata. The best result is indicated by the bold.

Method | Setting | Metadata |  $S_{p}$ (%) |  $S_{e}$ (%) |  Score (%)
---|---|---|---|---|---
BTS | (1) | All | $\text{81.40}_{\pm 2.57}$ | $\text{45.67}_{\pm 2.66}$ | $\text{{63.54}}_{\pm 0.80}$
(2) | Age-Sex-Loc | $\text{81.71}_{\pm 4.12}$ | $\text{43.63}_{\pm 3.48}$ | $\text{62.66}_{\pm 0.35}$
(2) | Age-Sex-Dev | $\text{79.49}_{\pm 3.66}$ | $\text{46.04}_{\pm 2.39}$ | $\text{62.76}_{\pm 1.09}$
(2) | Age-Loc-Dev | $\text{82.28}_{\pm 5.27}$ | $\text{43.48}_{\pm 4.38}$ | $\text{62.88}_{\pm 0.83}$
(2) | Sex-Loc-Dev | $\text{84.66}_{\pm 3.63}$ | $\text{41.09}_{\pm 3.37}$ | $\text{62.88}_{\pm 0.54}$
Audio-CLAP | (3) | - | $\text{80.85}_{\pm 3.33}$ | $\text{44.67}_{\pm 3.77}$ | $\text{62.56}_{\pm 0.37}$

We analyze the impact of metadata on classification performance by comparing
the results of three distinct experiment settings: (1) the full set of
metadata with BTS, (2) subset with exclusion of a single attribute with BTS,
and (3) audio-encoder only in the case of Audio-CLAP. The results are
summarized in Table 5, which shows that more textual context results in higher
performance. Specifically, using all metadata in (1) yields the highest Score,
while audio-encoder only (3) scored the lowest. The results of all metadata
subsets (2) fall in between them.

Furthermore, the results demonstrate that the measurement location (_Loc_) and
recording device type (_Dev_) have a larger effect than the demographic
attributes, i.e., age and gender of patients. The absence of _Loc_ and _Dev_
leads _Score_ drop of 0.78% and 0.88%, respectively, compared to the all
metadata case. This suggests that the type of recording device and the
measurement location significantly influence the acoustic properties of
respiratory sounds. Therefore, the combined use of these metadata provides the
meaningful context to understand the respiratory sounds.

###  4.3 Unknown Metadata Scenario

To probe how well the model generalizes to unseen text descriptions, we
examined the model performance for unseen test data, which additionally
includes a new metadata attribute that is not used for training. Specifically,
we added a sentence to describe the BMI of patients to the test data. The
additional sentence is written in the same style as the training descriptions,
e.g., ``The BMI of the patient was 20.50.''. The evaluation result is denoted
as BTS[BMI] in Table 6. Adding unknown metadata to text descriptions at test
time shows only minor performance degradation, which suggests that the model
performs reliably even with the unexpected metadata.

###  4.4 Missing Metadata Scenarios

Table 6: Results with variations on the metadata. The variations include the
additional BMI attribute, partial metadata, and no metadata in text
description. The Best result.

Method |  $S_{p}$ (%) |  $S_{e}$ (%) |  Score (%)
---|---|---|---
BTS | $\text{81.40}_{\pm 2.57}$ | $\text{45.67}_{\pm 2.66}$ | $\text{{63.54}}_{\pm 0.80}$
BTS[BMI] | $\text{81.40}_{\pm 2.57}$ | $\text{45.66}_{\pm 2.65}$ | $\text{63.53}_{\pm 0.80}$
BTS[Partial Metadata] | $\text{81.29}_{\pm 2.55}$ | $\text{45.54}_{\pm 2.61}$ | $\text{63.41}_{\pm 0.78}$
BTS[No Metadata] | $\text{80.82}_{\pm 3.54}$ | $\text{45.59}_{\pm 2.59}$ | $\text{63.21}_{\pm 1.04}$
Audio-CLAP | $\text{80.85}_{\pm 3.33}$ | $\text{44.67}_{\pm 3.77}$ | $\text{62.56}_{\pm 0.37}$

To understand how metadata that is partially or entirely missing effect the
model, we conducted two experiments. We first partially removed the metadata
(BTS[Partial Metadata]), by randomly eliminating one of the metadata
attributes from test samples and substituting ``Unknown'' with 10%
probability. Then, we entirely removed the metadata (BTS[No Metadata]) and
replace the whole description by ``No description.'' Table 6 describes the
experiment results. As expected, BTS[Partial Metadata] shows a slightly
degraded Score compared to BTS, while BTS[No Metadata] results in a relatively
large performance reduction. Nevertheless, the results with missing metadata
maintain an edge over Audio-CLAP. The results show that the BTS model is
robust to missing metadata. We conjecture that the model learns to infer
certain metadata characteristics directly from the audio, thereby preserving
its strong performance even in the absence of metadata during inference.

##  5 Conclusion

In this work, we proposed to directly utilize the metadata to improve the
performance of RSC. Our experiments demonstrated that including the metadata
as additional context for the classification leads to a considerable
performance increase, which results in the new SOTA for RSC. In particular,
the experiment results showed that our method helps minimize the performance
degradation due to the acoustic variations induced by the inherent factors
relating to the demographics and recording environment. Moreover, our method
works reliably even when the metadata is with unexpected information,
partially unavailable, or even completely unavailable. Besides, we showed that
the CLAP model provides a strong baseline for audio-only tasks.

##  6 Acknowledgement

This research was supported by the MSIT (Ministry of Science and ICT), Korea,
under the ITRC (Information Technology Research Center) support program
(IITP-2024-2020-0-01808) supervised by the IITP (Institute of Information &
Communications Technology Planning & Evaluation) and by Brian Impact
Foundation, a non-profit organization dedicated to the advancement of science
and technology for all.

## References

  * [1] J.-W. Kim, S. Bae, W.-Y. Cho, B. Lee, and H.-Y. Jung, ``Stethoscope-guided supervised contrastive learning for cross-domain adaptation on respiratory sound classification,'' in _ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2024, pp. 1431–1435.
  * [2] J.-W. Kim, M. Toikkanen, S. Bae, M. Kim, and H.-Y. Jung, ``Repaugment: Input-agnostic representation-level augmentation for respiratory sound classification,'' _arXiv preprint arXiv:2405.02996_ , 2024.
  * [3] J.-W. Kim, C. Yoon, M. Toikkanen, S. Bae, and H.-Y. Jung, ``Adversarial fine-tuning using generated respiratory sound to address class imbalance,'' _arXiv preprint arXiv:2311.06480_ , 2023.
  * [4] S. Bae, J.-W. Kim, W.-Y. Cho, H. Baek, S. Son, B. Lee, C. Ha, K. Tae, S. Kim, and S.-Y. Yun, ``Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification,'' in _Proc. INTERSPEECH 2023_ , 2023, pp. 5436–5440.
  * [5] I. Moummad and N. Farrugia, ``Pretraining respiratory sound representations using metadata and contrastive learning,'' in _2023 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)_. IEEE, 2023, pp. 1–5.
  * [6] T. Nguyen and F. Pernkopf, ``Lung sound classification using co-tuning and stochastic normalization,'' _IEEE Transactions on Biomedical Engineering_ , vol. 69, no. 9, pp. 2872–2882, 2022.
  * [7] L. Pham, D. Ngo, K. Tran, T. Hoang, A. Schindler, and I. McLoughlin, ``An ensemble of deep learning frameworks for predicting respiratory anomalies,'' in _2022 44th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_ , 2022, pp. 4595–4598.
  * [8] Z. Wang and Z. Wang, ``A domain transfer based data augmentation method for automated respiratory classification,'' in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 9017–9021.
  * [9] S. Gairola, F. Tom, N. Kwatra, and M. Jain, ``Respirenet: A deep neural network for accurately detecting abnormal lung sounds in limited data setting,'' _2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_, 2021.
  * [10] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, ``Imagenet: A large-scale hierarchical image database,'' in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [11] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, ``Audio set: An ontology and human-labeled dataset for audio events,'' in _Proc. IEEE ICASSP 2017_ , New Orleans, LA, 2017.
  * [12] B. Rocha, D. Filos, L. Mendes, I. Vogiatzis, E. Perantoni, E. Kaimakamis, P. Natsiavas, A. Oliveira, C. Jácome, A. Marques _et al._ , ``A respiratory sound database for the development of automated classification,'' in _Precision Medicine Powered by pHealth and Connected Health: ICBHI 2017, Thessaloniki, Greece, 18-21 November 2017_. Springer, 2018, pp. 33–37.
  * [13] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , ``Learning transferable visual models from natural language supervision,'' in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [14] B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, ``Clap learning audio concepts from natural language supervision,'' in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [15] Y. Wu, K. Chen, T. Zhang, Y. Hui, T. Berg-Kirkpatrick, and S. Dubnov, ``Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,'' in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [16] N. Hollenstein, C. Renggli, B. Glaus, M. Barrett, M. Troendle, N. Langer, and C. Zhang, ``Decoding eeg brain activity for multi-modal natural language processing,'' _Frontiers in Human Neuroscience_ , p. 378, 2021.
  * [17] Z. Wang and H. Ji, ``Open vocabulary electroencephalography-to-text decoding and zero-shot sentiment classification,'' in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 5, 2022, pp. 5350–5358.
  * [18] Y. Duan, C. Chau, Z. Wang, Y.-K. Wang, and C.-t. Lin, ``Dewave: Discrete encoding of eeg waves for eeg to text translation,'' in _Advances in Neural Information Processing Systems_ , A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36, 2023, pp. 9907–9918.
  * [19] J. Qiu, W. Han, J. Zhu, M. Xu, D. J. Weber, B. Li, and D. Zhao, ``Can brain signals reveal inner alignment with human languages?'' in _The 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023.
  * [20] D. P. Kingma and J. Ba, ``Adam: A method for stochastic optimization,'' _arXiv preprint arXiv:1412.6980_ , 2014.
  * [21] Y. Gong, Y.-A. Chung, and J. Glass, ``AST: Audio Spectrogram Transformer,'' in _Proc. Interspeech 2021_ , 2021, pp. 571–575.
  * [22] Z. Yang, S. Liu, M. Song, E. Parada-Cabaleiro, and B. W. Schuller, ``Adventitious respiratory classification using attentive residual neural networks,'' in _Interspeech_ , 2020.
  * [23] Y. Ma, X. Xu, and Y. Li, ``Lungrn+ nl: An improved adventitious lung sound classification using non-local block resnet neural network with mixup data augmentation.'' in _Interspeech_ , 2020, pp. 2902–2906.
  * [24] Y. Chang, Z. Ren, T. T. Nguyen, W. Nejdl, and B. W. Schuller, ``Example-based Explanations with Adversarial Attacks for Respiratory Sound Analysis,'' in _Proc. Interspeech 2022_ , 2022, pp. 4003–4007.
  * [25] Z. Ren, T. T. Nguyen, and W. Nejdl, ``Prototype learning for interpretable respiratory sound analysis,'' in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 9087–9091.