# Navigating Conflicting Views: Harnessing Trust for Learning

Jueqing Lu
Monash University
<EMAIL_ADDRESS>&Lan Du
Monash University
<EMAIL_ADDRESS>
&Wray Buntine
VinUniversity
<EMAIL_ADDRESS>
&Myong Chol Jung
Monash University
<EMAIL_ADDRESS>
&Joanna Dipnall
Monash University
<EMAIL_ADDRESS>
&Belinda Gabbe
Monash University
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Resolving conflicts is essential to make the decisions of multi-view
classification more reliable. Much research has been conducted on learning
consistent informative representations among different views, assuming that
all views are identically important and strictly aligned. However, real-world
multi-view data may not always conform to these assumptions, as some views may
express distinct information. To address this issue, we develop a
computational trust-based discounting method to enhance the existing
trustworthy framework in scenarios where conflicts between different views may
arise. Its belief fusion process considers the trustworthiness of predictions
made by individual views via an instance-wise probability-sensitive trust
discounting mechanism. We evaluate our method on six real-world datasets,
using Top-1 Accuracy, AUC-ROC for Uncertainty-Aware Prediction, Fleiss’ Kappa,
and a new metric called Multi-View Agreement with Ground Truth that takes into
consideration the ground truth labels. The experimental results show that
computational trust can effectively resolve conflicts, paving the way for more
reliable multi-view classification models in real-world applications. 111We
will release our codes once our paper is accepted.

##  1 Introduction

Multi-View Classification (MVC) plays a critical role in deep learning by
greatly enhancing the ability to make accurate decisions through integrating
the multi-source information. Its effectiveness has been verified with the
successful application in many domains such as autonomous driving [37] and AI-
assisted medical diagnostic systems [21]. Most of the existing studies on MVC
rely on the assumption that data from different views are consistently of good
quality and informative with regard to the ground truth [24, 39, 36].
Nevertheless, this assumption may not always be valid in real-world scenarios.
Substantial variations in the quality of data from different views can produce
contradictory results, thereby undermining the reliability of the model’s
predictions.

A possible solution for resolving conflicts is to project data from different
views into a latent common space [10, 33, 2, 11], and then draw a combined
representation from the latent space for the classification. This is achieved
by integrating essential features via weighting schemes, such as attention
mechanisms [41] and weighted fusion [1, 38]. These methods typically assign
higher weights to more informative views or features, thus reducing the impact
of potential conflicting or noisy information. Although these methods have
achieved promising results in MVC, their focus on the combined joint
representation can be a limitation. Solely relying on the joint representation
hinders the capacity to thoroughly grasp information provided by different
views. In contexts such as medical diagnosis, characterized by heterogeneous
data sources from various views of medical imaging (e.g., lateral, frontal and
scaphoid views in X-rays (Figure 1) ), it’s crucial to thoroughly analyze and
comprehend each view before establishing a comprehensive diagnosis, as each
view often offers unique information [31].

Figure 1:  Conflict in Medical Imaging. Fracture (red box) is evident in one
view, but not evident in another two. Adapted from a case study by [27].

Existing approaches to resolve conflicts build neural networks to generate
view-specific predictions and then combine view-specific predictions together.
As a prime example, the evidence-based framework is emerging as a promising
approach, offering a reliable means for the final fusion stage. Within this
framework, evidence acts as a metric of endorsement for the associated
predicted label, and the evidence is collected through view-specific neural
networks. Subsequently, evidence from diverse viewpoints is fused, considering
their respective epistemic uncertainties. However, there may exist cases where
the view-specific information is not well aligned with the ground truth,
resulting in misleading predictions with high confidence (low uncertainty).
For example, as shown in Figure 1, Fracture is evident in the scaphoid view
but not evident in others. A pioneering work [36] based on the evidence
framework tried to resolve conflicts by minimizing the degree of conflict
loss. However, this approach may inadvertently converge towards incorrect
predictions, potentially leading to unstable performance.

In this work, we take a significant step further: leveraging the evidence-
based framework, we propose a new computational trust based opinion fusion
method to resolve potential conflicts in MVC. Specifically, the computational
trust is modelled through an evidence network that operates on a view-specific
and sample-wise basis. Drawing upon the principle of trust discounting in
subjective logic, it evaluates the trustworthiness of view-specific
predictions generated by existing evidence-based frameworks, such as
Evidential Deep Learning (EDL) [29]. Within the proposed method, each view-
specific evidence is transformed into a degree of trust using the Binomial
opinion theory [15]. These degrees of trust are then utilized to establish
uncertainty and a trust-aware opinion, ultimately facilitating the generation
of reliable predictions. In summary, the contributions of this paper include:

  1. 1.

We present a novel learnable trust-discounting mechanism to extend the widely-
used evidence-based trustworthy MVC framework, enhancing its conflict
resolution capabilities. Drawing from the Binomial opinion theory within
subjective logic, it operates on a view-specific and instance-wise basis,
adeptly resolving conflicts among views through a probability-sensitive trust
discounting rule;

  2. 2.

We develop a stage-wise training strategy to optimize the parameters of the
proposed mechanism, which works robustly on different datasets;

  3. 3.

We conduct extensive experiments on six real-world datasets, showing that our
method outperforms the existing trusted MVC methods based on evidential deep
learning, particularly on the datasets exhibiting large discrepancy among
view-specific predictions. In addition, our method can also enhance the
consistency among opinions derived from different views.

##  2 Trust-discounted Fusion for Conflict Resolving

Given training data
$\mathcal{D}={\\{\\{\bm{x}^{v}_{i}\\}_{v=1}^{V},y_{i}\\}}_{i=1}^{N}$ where $N$
is the number of training data, each instance $\bm{x}_{i}$ has $V$ views,
ground truth label $y_{i}$ and an one-hot encoded label $\mathbf{y}_{i}$
(i.e., for a $K$-class classification problem, $\mathbf{y}_{i,k}$ is 1 if $k$
is the index of ground truth label for $i$-th instance, otherwise it is 0).
The task of MVC is to learn a function $f$ that maps
$\\{\bm{x}_{i}^{v}\\}_{v=1}^{V}$ to $\mathbf{y}_{i}$.

###  2.1 Preliminaries on Evidence Theory for MVC

The evidence-based framework applies Subjective Logic (SL) to the $K$-class
classification problem by assigning belief masses to individual class labels
and computing epistemic uncertainty for the generated belief masses. The
formulation links the evidence collected from instance view-specific
observation to the concentration parameter of the Dirichlet Distribution. Let
$f_{\theta}^{v}(\cdot)$ denote the view-specific neural network for evidence
generation, where the view-specific evidence for an instance is
$\bm{e}^{v}=f_{\theta}^{v}(\bm{x}^{v})$, the association between the evidence
and the Dirichlet parameters is simply $\alpha_{k}=e_{k}+1$ [29, 8] . The
belief mass on class label $k$, denoted as $b_{k}$, and uncertainty $u$ are
subject to the additive requirement, i.e., $u+\sum_{k=1}^{K}b_{k}=1$. With
respect to MVC, the view-specific belief mass $b_{k}^{v}$ and uncertainty
$u^{v}$ can then be computed as

$S^{v}=\sum_{k=1}^{K}\alpha^{v}_{k},\quad
b_{k}^{v}=\frac{e_{k}^{v}}{S^{v}}=\frac{\alpha_{k}^{v}-1}{S^{v}},\quad
u^{v}=1-\sum_{k=1}^{K}b_{k}^{v}=\frac{K}{S^{v}}$ (1)

To generate the final prediction, SL models the view-specific predictions as
multinomial opinions, denoted as $\omega^{v}=[\bm{b}^{v},u^{v},\bm{a}^{v}]$,
with $\bm{a}^{v}$ being the base rate (i.e., a prior probability distribution
over classes, generally a discrete uniform distribution), and then combine
them together with an appropriate belief fusion rules based on the context
[17]. The Belief Constraint Fusion (BCF) [17], an extension of Dempher-Shafer
combination rule [30], was first adopted by [8] in trusted MVC. Other fusion
rules, such as Aleatory Cumulative Belief Fusion (A-CBF) [25] and Averaging
Belief Fusion (ABF) [36] have also been explored. We stay with BCF in our
experiments based on the empirical results. For fusing different opinions,
i.e., $\omega=\omega^{1}\oplus\omega^{2}\oplus\cdots\omega^{V}$ from $V$
views, the fusion rule of BCF among two views can be formulated as follows:

$b_{k}=\frac{1}{1-C}(b^{1}_{k}b^{2}_{k}+b^{1}_{k}u^{2}+b^{2}_{k}u^{1}),\quad
u=\frac{1}{1-C}u^{1}u^{2}$ (2)

where $C=\sum_{i\neq j}b^{1}_{i}b^{2}_{j}$ is the normalization factor.
Applying Eq. (2) iteratively will derive the final fused opinion from the $V$
views, and the order does not affect the combined result [15]. For the fused
opinion $\omega$, we can derive the parameters of the Dirichlet $\alpha_{k}$
by reversing the computation of Eq. (1).

An alternative representation for BCF is based on combining the evidence, from
which the opinions $[\bm{b},u,\bm{a}]$ can be derived:

$e_{k}=e^{1}_{k}+e^{2}_{k}+\frac{e^{1}_{k}e^{2}_{k}}{K}$ (3)

Note that A-CBF corresponds to the case where the third term is removed, and
thus for the Dirichlet likelihood can be interpreted as Bayesian updating
assuming independent evidence.

Table 1: Illustration of fusing conflicting opinions with BCF, A-CBF and ABF | Source Opinions | Fused Opinions | Trust Fusion
---|---|---|---
|  $\acute{\omega}^{1}$ |  $\acute{\omega}^{2}$ |  BCF |  A-CBF |  ABF |  $\ddot{\omega}^{1}$ |  $\ddot{\omega}^{2}$ |  TD+BCF
$b_{1}$ |  0.67 |  0.1 |  0.387 |  0.391 |  0.385 |  0.9 |  0.0 |  0.56
$b_{2}$ |  0.2 |  0.2 |  0.222 |  0.203 |  0.200 |  0.0 |  0.9 |  0.19
$b_{3}$ |  0.1 |  0.67 |  0.387 |  0.391 |  0.385 |  NA |  NA |  0.16
u |  0.03 |  0.03 |  0.004 |  0.015 |  0.030 |  0.1 |  0.1 |  0.09

###  2.2 Conflict Resolving by Trust Discounting

We realize conflicts can happen when view-specific opinions express
conflicting preferences with similar level of confidence, leading to
inaccuracies in the combined opinion. Based upon this, we define the conflict
problem as follows:

######  Definition 1 (Conflicts within Multi-view Learning).

For a $K$-class multi-class classification problem, the view-specific models
give confident yet opposite opinions for a given data instance.

Table 1 presents a scenario with a two-view instance for 3-class
classification, illustrating the concept of conflict views. Conflict arises
for such a data instance $\\{\bm{x}^{v}\\}_{v=1}^{2}$ with the ground label
$y=1$, when the opinions are $\bm{b}^{1}=[0.67,0.2,0.1]$ and $u^{1}=0.03$,
alongside $\bm{b}^{2}=[0.1,0.2,0.67]$ and $u^{2}=0.03$. Existing fusion
approaches, as depicted in Table 1, struggle to provide a clear and accurate
final prediction, as they all generate the same belief mass for labels 1 and 3
due to the confidence of both opinions with low uncertainty.

Instead, we utilize the principle of Trust Fusion (TF) by Trust Discounting
(TD) [18] to resolve conflicts. The basic idea of TD is to discount
information provided by an individual view as a function of trust in that
view. It can be used to weigh the current view-specific opinion according to
the degree of trust, thus guiding the fusion process to generate more reliable
prediction. Here we present a Probability-sensitive Trust Discounting rule, as
show in Eq. (4), and use it in an instance-wise manner in our experiments,
This rule computes the trust degree by the expect value of projected Beta
distribution $\beta(\alpha_{1},\alpha_{2})$ (i.e.,
$p_{t}=\frac{\alpha_{2}}{\alpha_{1}+\alpha_{2}}$) for each individual instance
on each view, following Binomial opinion theory. It is noteworthy that other
choices to compute the trust degree are also possible, such as Uncertainty
Favouring Discounting and Opposite Belief Favouring Discounting [16]. They can
be easily integrated into our framework.

To distinguish different types of opinions, we modify notations used in the
previous section. Concretely, the opinions before applying trust fusion, is
often referred to as Functional (Trust) Opinion, denoted as $\acute{\omega}$,
in the form of multinomial opinion discussed in Section 2.1. Meanwhile, the
opinion used for discounting is known as the Referral (Trust) Opinion, denoted
as a Binomial opinion $\ddot{\omega}$, and $\breve{\omega}$ is used to
indicate the trust-discounted opinion. From [15], the discounting rule is
defined as:

######  Definition 2 (Instance-wise Probability-Sensitive Trust Discounting).

For each view of each individual instance, the trust-discounted opinion is
defined as

$\breve{\omega}=\ddot{\omega}\otimes\acute{\omega}=\left\\{\begin{array}[]{lr}\breve{\bm{b}}=\ddot{p}_{t}*\acute{\bm{b}},\\\
\breve{u}=1-\ddot{p}_{t}*\left(\sum_{k=1}^{K}\acute{b}_{k}\right),\\\
\breve{\bm{a}}=\acute{\bm{a}}.\end{array}\right.$ (4)

where $\otimes$ indicates the trust discounting operator, and the scalar
probability $\ddot{p}_{t}$ denotes the degree of trust, representing how much
we are confident with the information given by the view-specific model. In
contrast,
$\ddot{p}_{d}=\frac{\ddot{\alpha}_{1}}{\ddot{\alpha}_{1}+\ddot{\alpha}_{2}}$
denotes the degree of distrust, and is subject to the constraint
$\ddot{p}_{t}+\ddot{p}_{d}=1$.

Note this corresponds to updating the Dirichlet evidence by
$\breve{e}_{k}=\frac{\ddot{p}_{t}\acute{u}}{1-\ddot{p}_{t}+\ddot{p}_{t}\acute{u}}\,\acute{e}_{k}$.
Given Eq. (4), we fuse the trust-discounted opinions from $V$ views of $i$-th
instance with BCF by:

$\bar{\omega}_{i}=\breve{\omega}^{1}_{i}\oplus\breve{\omega}^{2}_{i}\oplus\cdots\oplus\breve{\omega}^{V}_{i}=\left(\ddot{\omega}^{1}_{i}\otimes\acute{\omega}^{1}_{i}\right)\oplus\left(\ddot{\omega}^{2}_{i}\otimes\acute{\omega}^{2}_{i}\right)\oplus\cdots\oplus\left(\ddot{\omega}^{V}_{i}\otimes\acute{\omega}^{V}_{i}\right)$
(5)

In the presence of conflicting opinions, the estimated uncertainty is expected
to increase, even if the uncertainty associated with the prediction of each
individual view is small. However, the uncertainty of fused opinion given by
the fusion rules mentioned above is always under-estimated, thus inconsistent
with the generated beliefs, as shown by the example in Table 1. With the
trust-discounting rule, conflicts can be effectively solved within the
evidential learning framework, ensuring theoretically-guaranteed better
fusion, as demonstrated by Proposition 1 and Proposition 2 together with the
example in Table 1. Specifically, Proposition 1 illustrates that our TD rule
consistently enhance classification accuracy, while Proposition 2 establishes
that TD corrects and adjusts the under-estimated uncertainty. The following
propositions provide theoretical analysis of the proposed TD rule, and their
detailed proof can be found in Appendix A.4.

######  Proposition 1.

Probability Sensitive Trust-Discounting maximizes the belief mass of the
Ground truth label after BCF, under the assumption that at least one view’s
prediction is correct.

######  Proposition 2.

The combined opinion generated by BCF with trust discounting for conflicting
views, will exhibit greater uncertainty than obtained through fusion with non-
discounted functional opinions.

###  2.3 Learning to Form Opinions

Figure 2:  The Trust Discounting Mechanism based Multi-view Evidential
Framework

Similar to the generation process of a functional opinion, we construct
another view-specific evidence network parameterized by $\ddot{\theta}$, for
outputs referral evidence $\ddot{\bm{e}}$, i.e.,
$\ddot{\bm{e}}^{v}_{i}=f_{\ddot{\theta}}^{v}([\bm{x}^{v}_{i},\acute{\bm{b}}^{v}_{i}])$222We
used Bi-Linear layer instead of Dense/Linear Layer in our experiments.. Figure
2 depicts our TF rule along with entire evidence-based learning framework.

For optimizing network parameters, we propose a stage-wise training algorithm
(Algo. 1), drawing inspiration from [7], to avoid any potential numerical
instability problems during training. In the proposed training algorithm, we
follow [29, 8, 9, 36], and use the following loss term $L(\bm{\alpha})$ 333The
closed form of $ace$ loss and KL-divergence loss will be shown in Appendix
A.5. to optimize the parameters of the Dirichlet distribution for where it
applies,

$L(\bm{\alpha})=L_{ace}(\bm{\alpha})+\lambda_{o}\text{D}_{KL}[\text{Dir}(\mathbf{p}|\bm{\tilde{\alpha}})||\text{Dir}(\mathbf{p}|\bm{1})]\,$
(6)

where $\tilde{\bm{\alpha}}=\mathbf{y}+(1-\mathbf{y})\odot\bm{\alpha}$ is the
Dirichlet parameters after removing non-misleading evidence from predicted
parameters $\bm{\alpha}$ , and $\mathbf{p}$ is the projected probability,
i.e., $\mathbf{p}=\frac{\bm{\alpha}}{S}$. $\odot$ is the element-wise product
and $\lambda_{o}$ is the annealing factor. Specifically, in the proposed
algorithm, we apply Eq. (6) to the Dirichlet parameters with regard to all
functional opinions (i.e., $\\{\acute{\bm{\alpha}}^{v}_{i}\\}_{v=1}^{V}$) at
stage-2a. At stages 2b and 3, we apply this loss term only to the parameters
of the Dirichlet distribution concerning the BCF combined opinion (i.e.,
$\bar{\bm{\alpha}}_{i}$). Please note that for the BCF combined opinion, both
the functional opinion and the referral opinion are required. Therefore, when
applying back-propagation (BP) with regard to the combined opinion, there will
be gradients for both the referral network and the functional network. We
update different types of networks at different stages. For example, we only
update the functional network at stage-2b and only update the referral network
at stage-3.

Since the parameters of the referral network are randomly initialised at the
beginning, we adopt the ace loss from above with a new smoothed label to warm-
up those parameters. For each $i$-th instance on $v$-th view, the hard label
$z^{v}_{i}$ is generated by

$z^{v}_{i}=\begin{cases}1&\text{if }\hat{y}_{i}^{v}={y}_{i}\\\
0&\text{otherwise}\end{cases}$ (7)

Following [26], we apply label smoothing with smoothing factor $\eta=0.9$ to
the hard label. The association between one-hot encoded hard label
$\mathbf{z}^{v}_{i}$ of target $z^{v}_{i}$ and smooth label is
$\mathring{\mathbf{z}}^{v}_{i}=\mathbf{z}^{v}_{i}\odot\eta+\eta/2$. since the
smoothed label could provide training signals for neurons of both target and
non-target labels, we simply adopt
$\sum_{v}L_{ace}(\ddot{\bm{\alpha}}_{i}^{v})$ with smoothed labels
$\\{\mathring{\mathbf{z}}_{i}^{v}\\}_{v=1}^{V}$ in the first warm-up stage,
and

$L_{ace}(\ddot{\bm{\alpha}}_{i}^{v})=\sum_{j=1}^{2}\mathring{\mathbf{z}}_{ij}^{v}(\psi(\ddot{\bm{\alpha}}_{i1}^{v}+\ddot{\bm{\alpha}}_{i2}^{v})-\psi(\ddot{\bm{\alpha}}_{ij}^{v}))$
(8)

where $\ddot{\bm{\alpha}}_{i}^{v}$ is the parameters of Beta distribution of
referral opinion and $\psi$ is the digamma function.

##  3 Related Work

Input: Multi-view dataset:
$\mathcal{D}=\\{\\{\mathbf{x}^{v}_{i}\\}_{v=1}^{V},y_{i}\\}_{i=1}^{N}$.

/*Training*/

Initialize: Initialize the parameters $\acute{\theta}$ of the Functional
networks; initialize the parameters $\ddot{\theta}$ of the Referral networks.

/*Stage-1 Warm-up Referral Network*/

for  _minibatch_ do

for  _$v=1:V$ _ do

$\ddot{\bm{e}}^{v}\leftarrow$ Referral Evidential network batch output;

Obtain $\ddot{\mathbf{\bm{\alpha}}}^{v}\leftarrow\ddot{\mathbf{e}}^{v}+1$ ;

end for

Obtain overall loss by summing losses calculated by Eq. (8) of all
$\\{\ddot{\mathbf{\bm{\alpha}}}^{v}\\}_{v=1}^{V}$;

Update the parameters $\ddot{\theta}$ by gradient descent with the loss from
above;

end for

/*Stage-2 Update Functional Network*/

for  _minibatch_ do

/*Substage-2a*/

for  _$v=1:V$ _ do

$\acute{\mathbf{e}}^{v}\leftarrow$ Functional Evidential network batch output;

Obtain $\acute{\mathbf{\bm{\alpha}}}^{v}\leftarrow\acute{\mathbf{e}}^{v}+1$ ;

end for

Obtain overall loss by summing losses calculated by Eq. (6) of all
$\\{\acute{\mathbf{\bm{\alpha}}}^{v}\\}_{v=1}^{V}$;

Update the parameters $\acute{\theta}$ by gradient descent with the loss from
above;

/*Substage-2b*/

for  _$v=1:V$ _ do

$\ddot{\mathbf{e}}^{v}\leftarrow$ Referral Evidential network batch output;

$\acute{\mathbf{e}}^{v}\leftarrow$ Functional Evidential network batch output;

Obtain $\ddot{\omega}^{v}$ and $\acute{\omega}^{v}$ by Eq. (1) with
$\ddot{\mathbf{e}}^{v}$ and $\acute{\mathbf{e}}^{v}$, respectively ;

end for

Obtain joint opinion $\bar{\omega}$ by Eq. (5) and $\bar{\bm{\alpha}}$ of this
opinion by reversing Eq. (1);

Obtain loss by Eq. (6) with $\bar{\bm{\alpha}}$ and update the parameters
$\acute{\theta}$ with gradient descent;

end for

/*Stage-3 Adjust Referral Network*/

By repeating Stage-2b only and update $\ddot{\theta}$ instead of
$\acute{\theta}$;

/*Stage-4 Adjust Functional Network*/

By repeating entire Stage-2;

Output: Functional and Referral networks parameters.

/*Test*/

Calculate the joint belief and the uncertainty mass through fusing discounted
opinions by Eq. (5).

Algorithm 1 Algorithm for Training and Testing

Multi-View Classification leverages multiple data sources, offering varied
perspectives on the same object, to enhance the classification performance.
Recent advancements in MVC have focused on generating noise-robust
representations through cluster-based [14, 34, 40], self-representation-based
[12], and partially view-aligned [35, 13] methods, harnessing the expressive
power of deep neural networks. However, noise-robust representations may not
fully resolve conflicts in opinions for a given data instance, as they lack a
mechanism to quantify the trustworthiness of each view. Our method addresses
this limitation by introducing an evidence network that evaluates the
reliability of view-specific predictions and adjusts the final predictions
according to the degree of trust.

Trusted Multi-View Classification has emerged as a crucial area and a pivotal
domain within Multi-View Learning. This research area aims to enhance the
accuracy and dependability of classification models by integrating data from
multiple views, guided by their prediction confidence and epistemic
uncertainty. The seminal work, Trusted Multi-View Classification (TMC) [8],
introduced the fusion of different views from an opinion perspective using the
Dempher-Shafer Combination rule. Building upon TMC, [9] extended the approach
by incorporating the pseudo-view, a concatenation of all other views,
resulting in improved performance. Subsequent studies by [25] and [36]
explored alternative opinion fusion methods. Concurrent research efforts, such
as those by [19] and [20], focus on multiview uncertainty estimation,
enhancing the model’s reliability.

Conflictive Multi-View Classification argues that existing work primarily
focusing on either learning joint aligned representations or better
quantifying uncertainty overlook the problem of potential contradictory in the
prediction space. Recognizing this gap, the pioneer work by [36] highlighted
this issue and introduced the Degree of Conflict loss. This loss quantifies
the disparity between different predictions in the prediction space while
accounting for uncertainty, aiming to mitigate conflict-related challenges.
However, this approach may inadvertently lead correct predictions to converge
towards incorrect ones, potentially jeopardizing model stability. In contrast,
our method can generate more accurate predictions with properly estimated
uncertainty.

##  4 Experiments

###  4.1 Experimental Setup

Datasets. Following [8, 9, 19, 36], we conducted experiments on six benchmark
datasets:
Handwritten,444https://archive.ics.uci.edu/ml/datasets/Multiple+Features
Caltech101 [4],
PIE,555http://www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-Pie/Home.html
Scene15 [3], HMDB [23] and CUB [32] with train-test split of 80% vs. 20%. As
we utilized the exact same datasets, we direct readers to [8] for further
details regarding these datasets. A summary of these datasets is provided in
the Appendix A.7.

Baseline Methods. We aim to resolve conflicts among predictions of different
views, so we consider the methods that generate view-specific predictions
which could have potential conflicts, and thus included following baselines:
Fusion by Majority Voting (F-Mode) and Fusion by Probability Averaging
(F-Avg), which are two commonly used fusion methods in most MVC methods. We
also consider existing trust-worthy baselines, MGP [19] TMC [8], and the
conflict resolution pioneering work ECML [36]. Our method can be extended to
leverage the pseudo view, as demonstrated by its application to ETMC [9], an
extended version of TMC that incorporates pseudo views. All methods were run
on a single 24GB RTX3090 card for fair comparison.

###  4.2 Experiment Results and Analysis

We compared our method with baselines with different metrics. For each
individual metric, mean and standard deviation from ten runs with different
random seeds are reported. In all tables, the best-performing method is
highlighted in bold, and the second-best method is underlined.

Table 2: Top-1 accuracy on test split. Dataset | Handwritten | Caltech101 | PIE | Scene15 | HMDB | CUB | AVG
---|---|---|---|---|---|---|---
F-Mode | 99.10±0.20 | 94.13±0.08 | 79.41±0.00 | 62.45±0.11 | 51.70±0.41 | 70.25±0.38 | 76.13
F-Avg | 99.25±0.00 | 95.59±0.06 | 90.59±0.29 | 76.21±0.09 | 71.49±0.35 | 92.75±0.53 | 87.65
MGP | 99.60±0.10 | 94.42±0.20 | 90.13±0.87 | 74.30±0.41 | 73.97±0.15 | 90.79±1.03 | 87.03
ECML | 99.57±0.11 | 94.25±0.08 | 91.40±0.47 | 64.34±0.11 | 72.90±0.11 | 92.58±0.25 | 85.84
TMC | 99.63±0.13 | 94.30±0.13 | 87.43±0.90 | 73.99±0.19 | 73.30±0.18 | 92.50±0.37 | 86.60
TF(ours) | 99.68±0.11 | 95.26±0.10 | 93.31±0.40 | 77.83±0.32 | 74.35±0.09 | 93.33±0.75 | 88.96
ETMC | 99.75±0.00 | 94.41±0.11 | 91.69±0.47 | 78.41±0.20 | 74.01±0.19 | 93.67±0.41 | 88.74
ETF(ours) | 99.98±0.07 | 95.07±0.08 | 94.63±0.34 | 82.01±0.17 | 75.55±0.15 | 94.08±0.38 | 90.22

Prediction Accuracy. Similar to [8, 9, 19, 36], we first evaluated the model
performance on the test split by Top-1 Classification Accuracy, as shown in
Table 2. Building on the strengths of pseudo view, our method (ETF)
consistently outperforms all trustworthy-based methods, and gains the best
average performance over six datasets compared with all baselines. For
example, on the PIE and Scene15 datasets, the use of trust boosts the accuracy
of ETMC by 2.94% and 3.60%, respectively. Moreover, ETF surpasses the
pioneering conflict resolving method ECML by a substantial margin of 3.23% on
PIE, 9.66% on Scene15 and 2.65% on HMDB, highlighting better power of
conflicts handling of our method. It is worth noting that Caltech101
inherently has lower level of conflicts, as corroborated by high accuracy and
Fleiss’ Kappa scores (Table 3) of all baselines. Nevertheless, ETF maintains
the compatible performance with the best one, F-Avg (a minor decrease of
0.52%), and still outperforms other trustworthy methods, e.g., improve the
accuracy of ETMC by 0.66%.

When compared to well-established trustworthy methods like TMC, MGP, and ECML
without pseudo views, our method TF consistently demonstrates superior
performance across all datasets. For example, our proposed trust discounting
method enhance TMC’s performance by 3.84% on Scene15 and 5.88% on PIE, while
also achieving the highest Top-1 accuracy on other datasets. Notably, our
method TF, even without incorporating pseudo views, exhibits comparable
performance to ETMC with pseduo views. For instance, TF outperforms ETMC on
three datasets (Caltech101, PIE, and HMDB) out of a total of six.

Table 3: Fleiss’ Kappa on test splits. Dataset | Handwritten | Caltech101 | PIE | Scene15 | HMDB | CUB | AVG
---|---|---|---|---|---|---|---
F-Mode | 0.63±0.04 | 0.97±0.00 | 0.38±0.00 | 0.42±0.00 | 0.56±0.00 | 0.71±0.01 | 0.61
F-Avg | 0.54±0.03 | 0.97±0.00 | 0.37±0.01 | 0.42±0.00 | 0.55±0.01 | 0.58±0.06 | 0.57
MGP | 0.59±0.05 | 0.94±0.00 | 0.21±0.01 | 0.33±0.00 | 0.51±0.00 | 0.43±0.07 | 0.50
ECML | 0.42±0.05 | 0.95±0.00 | 0.40±0.01 | 0.26±0.00 | 0.53±0.01 | 0.44±0.07 | 0.50
TMC | 0.54±0.07 | 0.94±0.01 | 0.23±0.02 | 0.30±0.01 | 0.52±0.01 | 0.37±0.19 | 0.48
TF(ours) | 0.65±0.02 | 0.95±0.00 | 0.36±0.01 | 0.39±0.00 | 0.54±0.00 | 0.51±0.10 | 0.57
ETMC | 0.66±0.01 | 0.84±0.00 | 0.28±0.04 | 0.37±0.00 | -0.15±0.04 | 0.45±0.10 | 0.41
ETF(ours) | 0.76±0.02 | 0.95±0.00 | 0.48±0.01 | 0.48±0.01 | 0.65±0.00 | 0.64±0.03 | 0.66

Reliable Prediction. To further validate the effectiveness of our proposed
method, we evaluate it with two additional metrics, Fleiss’ Kappa [6] and our
proposed metric, Multi-View Agreement with Ground Truth (MVAGT). Fleiss’ Kappa
is a statistical measure for assessing the reliability of agreement between
different raters, with scores closer to 1 indicating higher agreement among
the different predictions. As depicted in Table 3, our method (ETF) achieves
the highest Fleiss’ Kappa score on four datasets (Handwritten, PIE, Scene15,
HMDB and CUB). Even through ETF does not rank first on the remaining two
datasets (the third on Caltech101 and the second on CUB), it remains the most
generalizable model with the highest average Fleiss’ Kappa (0.66). It’s worth
noting that while our method assumes the existence of conflicts, Caltech101 is
a dataset with fewer conflicts, which explains the performance discrepancy in
Table 2. Nevertheless, ETF still outperforms other trustworthiness-based
methods and enhances the robustness of ETMC with an improvement of
approximately 13% on Caltech101. Moreover, it’s essential to highlight that
ETMC exhibits extremely poor agreement on HMDB with a negative value of -0.15.
However, by applying our method, ETF significantly improves performance by an
absolute value of 0.8. This underscores the relative robustness of our method
across different datasets.

Table 4: MVAGT on test split. Dataset | Handwritten | Caltech101 | PIE | Scene15 | HMDB | CUB
---|---|---|---|---|---|---
F-Mode | 88.87±1.73 | 94.13±0.08 | 79.41±0.00 | 62.54±0.11 | 51.70±0.41 | 70.25±0.38
F-Avg | 18.78±5.89 | 93.89±0.24 | 17.06±1.22 | 27.70±0.36 | 51.18±0.51 | 59.50±5.25
MGP | 81.37±5.73 | 91.55±0.29 | 63.20±2.31 | 52.10±0.41 | 50.43±0.42 | 42.50±9.26
ECML | 74.08±0.61 | 91.05±0.27 | 78.46±1.19 | 41.91±0.31 | 50.95±0.48 | 48.58±5.36
TMC | 81.58±6.57 | 90.27±0.38 | 51.54±3.00 | 51.42±0.46 | 50.37±0.45 | 43.25±14.8
TF(ours) | 88.97±0.61 | 92.01±0.22 | 80.59±0.75 | 60.41±0.52 | 52.47±0.35 | 54.33±7.54
ETMC | 98.10±0.17 | 92.41±0.32 | 75.15±4.13 | 73.75±0.45 | 8.45±1.09 | 91.08±1.06
ETF(ours) | 98.53±0.08 | 94.47±0.12 | 90.37±0.40 | 79.18±0.38 | 71.43±0.32 | 91.17±0.67

While a multi-view classification (MVC) classifier with both high accuracy and
high Fleiss’ Kappa score generally suggests good reliability, high Fleiss’
Kappa scores without reference to the ground truth label might be misleading,
particularly in cases where the majority of views agree on the same incorrect
class. Therefore, we propose a new evaluation metric (MVAGT), specifically
tailored for conflict MVC scenarios. MVAGT assesses correctness on the test
split by verifying that more than half of the views make correct decisions.
Since majority agreement is often more reasonable for final decisions in real-
world scenarios, evaluating methods using MVAGT ensures the reasonableness of
the fused decision. Further details about MVAGT are provided in Appendix A.6.
As depicted in Table. 4, ETF demonstrates superior performance compared to
other methods. Moreover, ETF exhibits good generalizability across different
datasets, where ETMC experiences significant decreases (e.g., HMDB) or other
methods alternately occupy the second-best position.

AUC-ROC for Uncertainty-Aware Prediction. Following prior work [5], we assess
uncertainty by approaching it as the task of determining whether to trust the
model’s prediction, evaluating the model’s ability to identify incorrect
predictions based on uncertainty scores. Specifically, we employed AUC-ROC to
measure the model’s discriminate power in distinguishing incorrect predictions
using uncertainty scores. As shown in Table 5, TF and ETF consistently
demonstrate the best performance in uncertainty-based classification across
five out of the six datasets, showcasing their robust generalizability.
Despite a slight performance decrease on the CUB dataset, our method (ETF)
still maintains the second-best result, outperforming other approaches,
whether incorporating pseudo views or not. One possible reason for the
decrease is that our method may increase the belief mass of ground truth label
for some incorrectly predicted instances, thus correcting the incorrect
predictions. However, the decrease in uncertainty is relatively small (e.g.,
0.5 for incorrect changes to 0.4 for correct). An example is shown in the
Appendix B.2.

Table 5: AUC-ROC scores for identifying incorrect predictions using uncertainty scores. Dataset | Handwritten | Caltech101 | PIE | Scene15 | HMDB | CUB
---|---|---|---|---|---|---
MGP | 99.29±0.30 | 87.62±0.90 | 88.43±0.67 | 63.92±1.96 | 82.87±0.60 | 58.20±11.4
ECML | 79.05±5.62 | 86.31±0.50 | 87.51±0.49 | 60.50±0.25 | 81.63±0.15 | 57.30±8.50
TMC | 99.23±0.22 | 87.33±0.47 | 90.16±0.99 | 62.60±0.54 | 82.63±0.48 | 64.80±10.5
TF(ours) | 99.32±0.35 | 88.99±0.54 | 95.90±0.08 | 64.56±2.02 | 83.59±0.23 | 53.52±14.3
ETMC | 99.30±0.19 | 88.35±0.63 | 93.02±1.40 | 66.49±0.44 | 85.42±0.34 | 72.56±8.11
ETF(ours) | 99.90±0.30 | 88.70±0.54 | 92.47±1.19 | 70.44±1.10 | 86.23±0.49 | 64.41±3.54

Handwritten
---

Caltech101
---

PIE
---

Scene15
---

HMDB
---

CUB
---

Figure 3: Performance comparison of simulated conflict predictions.

Simulating Conflicting Predictions with Noisy Instances. Following a similar
approach as [8, 19, 36], we introduce varying levels of Gaussian noise (e.g.,
with standard deviations of 10 and 100) to randomly selected half of the views
for each instance in the dataset, creating noisy views. The original views are
then replaced by these noisy views, and the model is trained using the updated
dataset. Consequently, the model’s predictions based on the noisy views may
conflict with the predictions from the views without explicitly introducing
noise. As illustrated in Figure 3 and Appendix B.3, our methods (ETF and TF)
consistently outperform others with or without pseudo views, demonstrating the
effectiveness of TD and our training algorithm’s ability to handle conflicting
views.

##  5 Conclusion

In this paper, we introduced a theoretically-funded approach for resolving
conflicts in Multi-View Classification. This approach is built on top of the
principle of trust discounting in Subjective Logic, where the computational
trust, aka referral trust, is represented as a Binomial opinion with a Beta
probability density function.The functional trust is then discounted by the
amount computed as a function of the degree of trust. We demonstrated through
extensive experiments that the proposed trust discounting method not only
benefits classification accuracy but also increases consistency among
different views, providing a new reliable approach to handling conflicts in
MVC. A possible limitation of our work is that the warm-up loss is not optimal
solution. We leave it for future work.

## References

  * Atrey et al. [2010] Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb El Saddik, and Mohan S Kankanhalli.  Multimodal fusion for multimedia analysis: a survey.  _Multimedia systems_ , 16:345–379, 2010.
  * Federici et al. [2020] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata.  Learning robust representations via multi-view information bottleneck.  In _International Conference on Learning Representations_ , 2020.
  * Fei-Fei and Perona [2005] Li Fei-Fei and Pietro Perona.  A bayesian hierarchical model for learning natural scene categories.  In _Computer Society Conference on Computer Vision and Pattern Recognition_ , volume 2, pages 524–531, 2005.
  * Fei-Fei et al. [2004] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In _Conference on Computer Vision and Pattern Recognition workshop_ , pages 178–178, 2004.
  * Filos et al. [2019] Angelos Filos, Sebastian Farquhar, Aidan N Gomez, Tim GJ Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Arnoud de Kroon, and Yarin Gal.  A systematic comparison of bayesian deep learning robustness in diabetic retinopathy tasks.  _arXiv preprint arXiv:1912.10481_ , 2019.
  * Fleiss [1971] Joseph L Fleiss.  Measuring nominal scale agreement among many raters.  _Psychological bulletin_ , 76(5):378, 1971.
  * Goodfellow et al. [2020] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Han et al. [2021] Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou.  Trusted multi-view classification.  In _International Conference on Learning Representations_ , 2021.
  * Han et al. [2022] Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou.  Trusted multi-view classification with dynamic evidential fusion.  _IEEE transactions on pattern analysis and machine intelligence_ , 45(2):2551–2566, 2022.
  * Hardoon et al. [2004] David R Hardoon, Sandor Szedmak, and John Shawe-Taylor.  Canonical correlation analysis: An overview with application to learning methods.  _Neural computation_ , 16(12):2639–2664, 2004.
  * Hjelm et al. [2019] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio.  Learning deep representations by mutual information estimation and maximization.  In _International Conference on Learning Representations_ , 2019.
  * Hou et al. [2020] Dongdong Hou, Yang Cong, Gan Sun, Jiahua Dong, Jun Li, and Kai Li.  Fast multi-view outlier detection via deep encoder.  _IEEE Transactions on Big Data_ , 8(4):1047–1058, 2020.
  * Huang et al. [2020] Zhenyu Huang, Peng Hu, Joey Tianyi Zhou, Jiancheng Lv, and Xi Peng.  Partially view-aligned clustering.  _Advances in Neural Information Processing Systems_ , 33:2892–2902, 2020.
  * Huang et al. [2023] Zongmo Huang, Yazhou Ren, Xiaorong Pu, Shudong Huang, Zenglin Xu, and Lifang He.  Self-supervised graph attention networks for deep weighted multi-view clustering.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 7936–7943, 2023.
  * Jøsang [2018] Audun Jøsang.  _Subjective Logic: A formalism for reasoning under uncertainty_.  Springer Publishing Company, Incorporated, 2018.
  * Jøsang et al. [2012] Audun Jøsang, Tanja Ažderska, and Stephen Marsh.  Trust transitivity and conditional belief reasoning.  In _Trust Management VI: 6th IFIP WG 11.11 International Conference, IFIPTM 2012, Surat, India, May 21-25, 2012. Proceedings 6_ , pages 68–83, 2012.
  * Jøsang et al. [2013] Audun Jøsang, Paulo CG Costa, and Erik Blasch.  Determining model correctness for situations of belief fusion.  In _Proceedings of the 16th International Conference on Information Fusion_ , pages 1886–1893, 2013.
  * Jøsang et al. [2015] Audun Jøsang, Magdalena Ivanovska, and Tim Muller.  Trust revision for conflicting sources.  In _2015 18th International Conference on Information Fusion (Fusion)_ , pages 550–557, 2015.
  * Jung et al. [2022] Myong Chol Jung, He Zhao, Joanna Dipnall, Belinda Gabbe, and Lan Du.  Uncertainty estimation for multi-view data: the power of seeing the whole picture.  _Advances in Neural Information Processing Systems_ , 35:6517–6530, 2022.
  * Jung et al. [2023] Myong Chol Jung, He Zhao, Joanna Dipnall, and Lan Du.  Beyond unimodal: Generalising neural processes for multimodal uncertainty estimation.  _Advances in Neural Information Processing Systems_ , 36, 2023.
  * Kang et al. [2020] Hengyuan Kang, Liming Xia, Fuhua Yan, Zhibin Wan, Feng Shi, Huan Yuan, Huiting Jiang, Dijia Wu, He Sui, Changqing Zhang, et al.  Diagnosis of coronavirus disease 2019 (covid-19) with structured latent multi-view representation learning.  _IEEE transactions on medical imaging_ , 39(8):2606–2614, 2020.
  * Kingma and Ba [2015] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_ , 2015.
  * Kuehne et al. [2011] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre.  Hmdb: a large video database for human motion recognition.  In _2011 International Conference on Computer Vision_ , pages 2556–2563, 2011.
  * Liang et al. [2024] Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency.  Foundations and trends in multimodal machine learning: Principles, challenges, and open questions.  _ACM Comput. Surv._ , 2024.
  * Liu et al. [2022] Wei Liu, Xiaodong Yue, Yufei Chen, and Thierry Denoeux.  Trusted multi-view deep learning with opinion aggregation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 7585–7593, 2022.
  * Müller et al. [2019] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  _Advances in neural information processing systems_ , 32, 2019.
  * Niknejad [2022] Mohammad Taghi Niknejad.  Scaphoid fracture - Case Study.  https://radiopaedia.org/cases/99029, 2022.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Sensoy et al. [2018] Murat Sensoy, Lance Kaplan, and Melih Kandemir.  Evidential deep learning to quantify classification uncertainty.  _Advances in neural information processing systems_ , 31, 2018.
  * Shafer [1976] Glenn Shafer.  _A mathematical theory of evidence_ , volume 42.  Princeton university press, 1976.
  * Tafti and Byerly [2020] Amin Tafti and Doug W Byerly.  X-ray radiographic patient positioning.  _StatPearls_ , 2020.
  * Wah et al. [2011] C Wah, S Branson, P Welinder, P Perona, and S Belongie.  The Caltech-UCSD Birds-200-2011 dataset.  Technical report, California Institute of Technology, 2011.
  * Wang et al. [2015] Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes.  On deep multi-view representation learning.  In _International conference on machine learning_ , pages 1083–1092. PMLR, 2015.
  * Wen et al. [2023a] Jie Wen, Chengliang Liu, Gehui Xu, Zhihao Wu, Chao Huang, Lunke Fei, and Yong Xu.  Highly confident local structure based consensus graph learning for incomplete multi-view clustering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15712–15721, 2023a.
  * Wen et al. [2023b] Yi Wen, Siwei Wang, Qing Liao, Weixuan Liang, Ke Liang, Xinhang Wan, and Xinwang Liu.  Unpaired multi-view graph clustering with cross-view structure matching.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2023b.
  * Xu et al. [2024] Cai Xu, Jiajun Si, Ziyu Guan, Wei Zhao, Yue Wu, and Xiyue Gao.  Reliable conflictive multi-view learning.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 38:16129–16137, 2024.
  * Yurtsever et al. [2020] Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda.  A survey of autonomous driving: Common practices and emerging technologies.  _IEEE access_ , 8:58443–58469, 2020.
  * Zhang et al. [2019] Changqing Zhang, Yeqing Liu, and Huazhu Fu.  Ae2-nets: Autoencoder in autoencoder networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2577–2585, 2019.
  * Zhang et al. [2023a] Chaoyang Zhang, Zhengzheng Lou, Qinglei Zhou, and Shizhe Hu.  Multi-view clustering via triplex information maximization.  _IEEE Transactions on Image Processing_ , 2023a.
  * Zhang et al. [2023b] Pei Zhang, Siwei Wang, Liang Li, Changwang Zhang, Xinwang Liu, En Zhu, Zhe Liu, Lu Zhou, and Lei Luo.  Let the data choose: Flexible and diverse anchor graph fusion for scalable multi-view clustering.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 11262–11269, 2023b.
  * Zheng et al. [2021] Lecheng Zheng, Yu Cheng, Hongxia Yang, Nan Cao, and Jingrui He.  Deep co-attention network for multi-view subspace learning.  In _Proceedings of the Web Conference 2021_ , pages 1528–1539, 2021.

##  Appendix A Proofs and Derivations

###  A.1 Calculation of Predictive Probability

According to Subjective Logic (SL) [15], the predictive probability $p_{k}$
for class $k$, can be calculated by

$p_{k}=b_{k}+a_{k}*u$ (9)

where $b_{k}$ is the belief mass for $k$-th label, $u$ is the predictive
uncertainty or epistemic uncertainty [29]. We usually assume the prior $a_{k}$
conforms to a uniform discrete distribution, i.e., $a_{k}=1/K$.

Above equation is identical to

$p_{k}=\frac{\alpha_{k}}{S}$ (10)

where $\alpha_{k}$ is the Dirichlet concentration parameter for $k$-th label,
and $S$ is the Dirichlet strength, i.e., $S=\sum_{k}\alpha_{k}$.

###### Proof.

$\displaystyle p_{k}$ $\displaystyle=$ $\displaystyle b_{k}+a_{k}*u$
$\displaystyle=$ $\displaystyle b_{k}+\frac{1}{K}*\frac{K}{S}$
$\displaystyle=$ $\displaystyle\frac{e_{k}}{S}+\frac{1}{S}$ $\displaystyle=$
$\displaystyle\frac{\alpha_{k}}{S}$

∎

Since Beta Distribution is 2-dimensional Dirichlet Distribution, above
equations for calculating probabilities of multinomial opinions could also be
applied to binomial opinions.

###  A.2 Alternative Representation of Belief Constraint Fusion(BCF)

###### Proof.

We the proof for Eq. (3) as follows,

$\displaystyle e_{k}$ $\displaystyle=$ $\displaystyle S*b_{k}$
$\displaystyle=$ $\displaystyle
S\,\frac{1}{1-C}(b_{k}^{1}b_{k}^{2}+b_{k}^{1}u^{2}+b_{k}^{2}u^{1})$
$\displaystyle=$ $\displaystyle
S\,\frac{1-\sum_{k}b_{k}}{u^{1}u^{2}}(b_{k}^{1}b_{k}^{2}+b_{k}^{1}u^{2}+b_{k}^{2}u^{1})$
$\displaystyle=$
$\displaystyle(S-S*\sum_{k}b_{k})\,\frac{1}{u^{1}u^{2}}(b_{k}^{1}b_{k}^{2}+b_{k}^{1}u^{2}+b_{k}^{2}u^{1})$
$\displaystyle=$
$\displaystyle(S-\sum_{k}e_{k})\,\frac{1}{u^{1}u^{2}}(b_{k}^{1}b_{k}^{2}+b_{k}^{1}u^{2}+b_{k}^{2}u^{1})$
$\displaystyle=$ $\displaystyle
K\,\frac{1}{u^{1}u^{2}}(b_{k}^{1}b_{k}^{2}+b_{k}^{1}u^{2}+b_{k}^{2}u^{1})$
$\displaystyle=$ $\displaystyle
K\,\frac{1}{u^{1}u^{2}}(\frac{e_{k}^{1}e_{k}^{2}}{S^{1}S^{2}}+\frac{e_{k}^{1}u^{2}}{S^{1}}+\frac{e_{k}^{2}u^{1}}{S^{2}})$
$\displaystyle=$ $\displaystyle
K(\frac{e_{k}^{1}e_{k}^{2}}{K*K}+\frac{e_{k}^{1}u^{2}}{Ku^{2}}+\frac{e_{k}^{2}u^{1}}{Ku^{1}})$
$\displaystyle=$
$\displaystyle\frac{e_{k}^{1}e_{k}^{2}}{K}+e_{k}^{1}+e_{k}^{2}$

∎

###  A.3 Dirichlet Evidence Updating by Trust Discounting (TD)

As mentioned earlier, the TD in Definition 2 is corresponds to update
Dirichlet evidence using following equation,

$\breve{e}_{k}=\frac{\ddot{p}_{t}\acute{u}}{1-\ddot{p}_{t}+\ddot{p}_{t}\acute{u}}\,\acute{e}_{k}$
(11)

where $\ddot{p}_{t}$ is the probability representing trust degree and
$\acute{u}$ is the uncertainty for functional opinion. $\acute{e}_{k}$ is
Dirichlet evidence of functional opinion, and $\breve{e}_{k}$ is Dirichlet
evidence after discounting.

###### Proof.

$\displaystyle\breve{\bm{e}}_{k}$ $\displaystyle=$
$\displaystyle\breve{\bm{b}}_{k}*\breve{S}$ $\displaystyle=$
$\displaystyle\frac{\ddot{p_{t}}\acute{b_{k}}K}{\breve{u}}$ $\displaystyle=$
$\displaystyle\frac{\ddot{p_{t}}\acute{b_{k}}K}{1-\ddot{p_{t}}+\ddot{p_{t}}\acute{u}}$
$\displaystyle=$
$\displaystyle\frac{\ddot{p_{t}}}{1-\ddot{p_{t}}+\ddot{p_{t}}\acute{u}}\frac{\acute{e_{k}}}{\acute{S}}K$
$\displaystyle=$
$\displaystyle\frac{\ddot{p_{t}}}{1-\ddot{p_{t}}+\ddot{p_{t}}\acute{u}}\frac{K}{\acute{S}}\acute{e_{k}}$
$\displaystyle=$
$\displaystyle\frac{\ddot{p_{t}}\acute{u}}{1-\ddot{p_{t}}+\ddot{p_{t}}\acute{u}}\,\acute{e_{k}}$

∎

###  A.4 Detailed Proof of Propositions

###### Proof.

Proof details of Proposition 1. We use $g$ to denote the index of ground-truth
label. Recall that scalar probability $\ddot{p}_{t}$ represents the degree of
trust as mentioned before. With at least one view’s prediction is correct, the
belief mass of ground truth label of BCF combined opinion, which integrates
through trust-discounted opinions, is as follows,

$\displaystyle\bar{b}_{g}$ $\displaystyle=$
$\displaystyle\frac{1}{1-\breve{C}}(\breve{b}^{1}_{g}\breve{b}^{2}_{g}+\breve{b}^{1}_{g}\breve{u}^{2}+\breve{b}^{2}_{g}\breve{u}^{1})$
$\displaystyle=$
$\displaystyle\frac{1}{1-\breve{C}}((\acute{b}^{1}_{g}\ddot{p}^{1}_{t})(\acute{b}^{2}_{g}\ddot{p}^{2}_{t})+\acute{b}^{1}_{g}\ddot{p}^{1}_{t}\breve{u}^{2}+\acute{b}^{2}_{g}\ddot{p}^{2}_{t}\breve{u}^{1})$

And we have the discounted uncertainty

$\displaystyle\breve{u}$ $\displaystyle=$ $\displaystyle
1-\ddot{p}_{t}(\sum_{k}\acute{b}_{k})$ $\displaystyle=$ $\displaystyle
1-\ddot{p}_{t}(1-\acute{u})$ $\displaystyle=$ $\displaystyle
1-\ddot{p}_{t}+\ddot{p}_{t}*\acute{u}$

In cases where the belief mass of the ground truth label is maximal (i.e.,
predicted label is the ground truth label), our referral network should
generate a high trust value, that is $\ddot{p}_{t}\to 1$ when
$\acute{b}_{g}=max(\mathbf{\acute{b}})$, and otherwise $\ddot{p}_{t}\to 0$.
Therefore, $\breve{u}\to\acute{u}$ when
$\acute{b}_{g}=max(\mathbf{\acute{b}})$, and $\breve{u}\to 1$ as well.
Therefore, the fused term will be dominant by correctly predicted views as
$\ddot{p}_{t}\to 1$ as well as $\breve{u}\to 1$ which contributes in cross
multiplication. Subsequently,

$\displaystyle\bar{b}_{g}$ $\displaystyle=$
$\displaystyle\frac{1}{1-\breve{C}}((\acute{b}^{1}_{g}\ddot{p}^{1}_{t})(\acute{b}^{2}_{g}\ddot{p}^{2}_{t})+\acute{b}^{1}_{g}\ddot{p}^{1}_{t}\breve{u}^{2}+\acute{b}^{2}_{g}\ddot{p}^{2}_{t}\breve{u}^{1})$
$\displaystyle\geq$
$\displaystyle\frac{1}{1-\breve{C}}((\acute{b}^{1}_{k}\ddot{p}^{1}_{t})(\acute{b}^{2}_{k}\ddot{p}^{2}_{t})+\acute{b}^{1}_{k}\ddot{p}^{1}_{t}\breve{u}^{2}+\acute{b}^{2}_{k}\ddot{p}^{2}_{t}\breve{u}^{1}(\text{equality
holds iif. }k=z))$ $\displaystyle=$
$\displaystyle\frac{1}{1-\breve{C}}(\breve{b}^{1}_{k}\breve{b}^{2}_{k}+\breve{b}^{1}_{k}\breve{u}^{2}+\breve{b}^{2}_{k}\breve{u}^{1})=\bar{b}_{k}$

∎

###### Proof.

Proof details of Proposition 2. Let $\bar{u}$ and $\bar{u}^{\prime}$ denote
the uncertainty of BCF combined opinion with or without Trust Discounting,
respectively.

$\displaystyle\bar{u}$ $\displaystyle=$
$\displaystyle\frac{1}{\sum^{K}_{k=1}(\frac{\breve{b}^{1}_{k}\breve{b}^{2}_{k}}{\breve{u}^{1}\breve{u}^{2}}+\frac{\breve{b}^{1}_{k}}{\breve{u}^{1}}+\frac{\breve{b}^{2}_{k}}{\breve{u}^{2}})+1}$
$\displaystyle=$
$\displaystyle\frac{1}{\sum^{K}_{k=1}(\frac{\acute{b}^{1}_{k}\ddot{p}^{1}_{t}\acute{b}^{2}_{k}\ddot{p}^{2}_{t}}{(\acute{u}^{1}\ddot{p}^{1}_{t}+1-\ddot{p}^{1}_{t})(\acute{u}^{2}\ddot{p}^{1}_{t}+1-\ddot{p}^{2}_{t})}+\frac{\acute{b}^{1}_{k}\ddot{p}^{1}_{t}}{\acute{u}^{1}\ddot{p}^{1}_{t}+1-\ddot{p}^{1}_{t}}+\frac{\acute{b}^{2}_{k}\ddot{p}^{2}_{t}}{\acute{u}^{2}\ddot{p}^{1}_{t}+1-\ddot{p}^{2}_{t}})+1}$
$\displaystyle=$
$\displaystyle\frac{1}{\sum^{K}_{k=1}(\frac{\acute{b}^{1}_{k}\acute{b}^{2}_{k}}{(\frac{\acute{u}^{1}}{\ddot{p}^{2}_{t}}+\frac{1}{\ddot{p}^{1}_{t}\ddot{p}^{2}_{t}}-\frac{1}{\ddot{p}^{2}_{t}})(\frac{\acute{u}^{2}}{\ddot{p}^{1}_{t}}+\frac{1}{\ddot{p}^{1}_{t}\ddot{p}^{2}_{t}}-\frac{1}{\ddot{p}^{1}_{t}})}+\frac{\acute{b}^{1}_{k}}{\acute{u}^{1}+\frac{1}{\ddot{p}^{1}_{t}}-1}+\frac{\acute{b}^{2}_{k}}{\acute{u}^{2}+\frac{1}{\ddot{p}^{2}_{t}}-1})+1}$
$\displaystyle=$
$\displaystyle\frac{1}{\sum^{K}_{k=1}(\frac{\acute{b}^{1}_{k}\acute{b}^{2}_{k}}{(\frac{\acute{u}^{1}}{\ddot{p}^{2}}+\frac{1-\ddot{p}^{1}}{\ddot{p}^{1}\ddot{p}^{2}})(\frac{\acute{u}^{2}}{\ddot{p}^{1}}+\frac{1-\ddot{p}^{2}}{\ddot{p}^{1}\ddot{p}^{2}})}+\frac{\acute{b}^{1}_{k}}{\acute{u}^{1}+\frac{1}{\ddot{p}^{1}}-1}+\frac{\acute{b}^{2}_{k}}{\acute{u}^{2}+\frac{1}{\ddot{p}^{2}}-1})+1}$
$\displaystyle\geq$
$\displaystyle\frac{1}{\sum^{K}_{k=1}(\frac{\acute{b}^{1}_{k}\acute{b}^{2}_{k}}{\acute{u}^{1}\acute{u}^{2}}+\frac{\acute{b}^{1}_{k}}{\acute{u}^{1}}+\frac{\acute{b}^{2}_{k}}{\acute{u}^{2}})+1}=\bar{u}^{\prime}$

∎

###  A.5 Loss Functions and Hyperparameters for Optimization

Recall that the probability density function (pdf) of the Dirichlet
distribution, $\text{Dir}(\mathbf{p}\mid\bm{\alpha})$, is given by:

$\text{Dir}(\mathbf{p}\mid\bm{\alpha})=\frac{1}{B(\bm{\alpha})}\prod_{i=1}^{K}p_{i}^{\alpha_{i}-1}$

where:

  * •

$\mathbf{p}=(p_{1},p_{2},\ldots,p_{K})$ is a probability vector, such that
$\sum_{k=1}^{K}p_{k}=1$ and $p_{k}\geq 0$ for all $k$.

  * •

$\bm{\alpha}=(\alpha_{1},\alpha_{2},\ldots,\alpha_{K})$ is a vector of
concentration parameters, with $\alpha_{k}>0$.

  * •

$B(\bm{\alpha})$ is the multivariate Beta function, defined as
$B(\bm{\alpha})=\frac{\prod_{k=1}^{K}\Gamma(\alpha_{k})}{\Gamma\left(\sum_{k=1}^{K}\alpha_{k}\right)}$.

  * •

$\Gamma(\cdot)$ is the Gamma function.

Our loss function for Dirichlet Parameters $\bm{\alpha}$ is

$\displaystyle
L(\bm{\alpha})=L_{ace}(\bm{\alpha})+\lambda\text{D}_{KL}[\text{Dir}(\mathbf{p}|\bm{\tilde{\alpha}})||\text{Dir}(\mathbf{p}|\bm{1})]$
(12)

Recall that $\tilde{\bm{\alpha}}=\mathbf{y}+(1-\mathbf{y})\odot\bm{\alpha}$ is
the Dirichlet parameters after removing non-misleading evidence from predicted
parameters $\bm{\alpha}$ and $\mathbf{p}$ is the projected probability, i.e.,
$\mathbf{p}=\frac{\bm{\alpha}}{S}$. $\odot$ is the element-wise product and
$\lambda_{o}$ is the annealing factor. We follow [29] , and use
$\lambda_{o}=min(1.0,o/10)$, where $o$ is the index of the current epoch.

Specifically, the $ace$ loss represents the Bayes risk for Cross-Entropy loss
when using a Dirichlet distribution.

$\displaystyle L_{ace}(\bm{\alpha})$ $\displaystyle=$
$\displaystyle\int\left[\sum_{k=1}^{K}-y_{k}log(p_{k})\right]\frac{1}{B(\bm{\alpha})}\prod_{k=1}^{K}({p_{k}})^{\alpha_{k}-1}d\mathbf{p}$
(13) $\displaystyle=$
$\displaystyle\sum_{k=1}^{K}y_{k}(\psi(S)-\psi(\alpha_{k}))$

Where $\psi$ is the digamma function.

Recall that our referral network will generate the evidence for binomial
opinion, and the evidence will be converted into parameters of Beta
Distribution, i.e., $Beta(\alpha_{0},\alpha_{1})$ Subsequently, by replacing
the Dirichlet Distribution with Beta Distribution, and the label $y_{k}$ in
above equation with another label, we can have the $ace$ loss for Beta
Distribution, as Eq. (8).

Table 6: TF and ETF hyper-parameters Hyper-parameter | Handwritten | Caltech101 | PIE | Scene15 | HMDB | CUB
---|---|---|---|---|---|---
lr | 3e-3 | 1e-4 | 3e-3 | 1e-2 | 3e-4 | 1e-3
rlr | 3e-4 | 3e-5 | 1e-3 | 3e-3 | 1e-4 | 3e-4
weight-decay | 1e-4 | 1e-4 | 1e-4 | 1e-4 | 1e-4 | 1e-4
warm-up epochs | 1 | 1 | 1 | 1 | 1 | 1

The KL divergence loss is

$\displaystyle\text{D}_{KL}\left[\text{Dir}(\mathbf{p}\mid\bm{\alpha})\parallel\text{Dir}(\mathbf{p}\mid\bm{1})\right]$
$\displaystyle=\log\left(\frac{\Gamma\left(\sum_{k=1}^{K}\alpha_{k}\right)}{\Gamma(K)\prod_{k=1}^{K}\Gamma(\alpha_{k})}\right)+\sum_{k=1}^{K}(\alpha_{k}-1)\left[\psi(\alpha_{k})-\psi\left(\sum_{j=1}^{K}\alpha_{j}\right)\right]$
(14)

The hyper-parameters for training TF and ETF has been shown in in Table 6.
Concretely, "lr" is the learning rate for functional networks, "rlr" indicates
the learning rate for referral networks. For the "lr", we follow ETMC [9], and
used same strategy to select learning rate for the functional nets. When
tuning the learning rate for referral networks, we follow a basic principle of
starting with a value less than or equal to the base learning rate, and then
gradually decreasing the learning rate of referral network by a factor of 3.
For fair comparison, we used same learning rate for functional networks for
evidence-based methods, except MGP [19], where we followed their paper.

We implement our proposed methods (TF and ETF) using Pytorch (version 1.13)
[28] and used Adam optimizer [22] to update network parameters during training
on all datasets, with weight decay 1e-4 following [9].

###  A.6 Multi-View Agreement with Ground Truth (MVAGT)

The MVAGT (Multi-View Agreement with Ground Truth) is a novel evaluation
metric designed specifically for multi-view classification problems with
conflicting views. It assesses the model’s performance on the test set by
considering the ground truth labels, thus providing a more reliable and
realistic measure of the model’s ability to handle view disagreements. The
rationality behind MVAGT lies in its alignment with real-world scenarios,
where the majority agreement among multiple views is often considered more
reasonable for the final decision. In the presence of view conflicts, a model
that can make predictions consistent with the majority of views is deemed more
trustworthy and reliable. By evaluating models using MVAGT, we can examine the
reasonableness of the fused decision and assess the model’s capability to
handle view conflicts effectively. Mathematically, MVAGT calculates the
accuracy of the model on the test set as follows:

$\text{MVAGT}=\frac{1}{M}\sum_{i=1}^{M}\mathbbm{1}\left(\sum_{v=1}^{V}\mathbbm{1}((\hat{y}^{v}_{i}=y_{i})>\frac{V}{2}\right)$
(15)

where $M$ is the total number of test samples, $V$ is the number of views,
$\hat{y}^{v}_{i}$ is the predicted label of the $i$-th sample from the $v$-th
view, $y^{i}$ is the ground truth label of the $i$-th sample, and
$\mathbbm{1}(\cdot)$ is the indicator function that returns 1 if the condition
is satisfied and 0 otherwise.

Table 7: Summary of Datasets Dataset | Size | K | Dimensions
---|---|---|---
HandWritten | 2000 | 10 | 240/76/216/47/64/6
Caltech101 | 8677 | 101 | 4096/4096
PIE | 680 | 68 | 484/256/279
Scene15 | 4485 | 15 | 20/59/40
HMDB | 6718 | 51 | 1000/1000
CUB | 600 | 10 | 1024/300

###  A.7 Summary of Dataset

We provide the summary of the dataset in Table 7, we direct readers to [8] for
further details regarding these datasets.

##  Appendix B Supplementary Insights and Additional Analysis

###  B.1 Reduce Conflicts by Trust Fusion

TMC
---

TF
---

ETMC
---

ETF
---

Figure 4: Conflict Ratio on Scene15, Four Methods TMC, TF, ETMC, ETF are
compared. GT, Pred, PS are ground-truth view, prediction view, pseudo-view
respectively, View 1, 2 and 3 are corresponding to feature view 1, 2 and 3.

We calculate the Conflict Ratio (CR) by normalizing the number of times that
the $v$-th view prediction is different from $w$-th view, i.e.,
$\text{CR}(\hat{\bm{y}}^{v},\hat{\bm{y}}^{w})=\frac{1}{M}\sum_{i=1}^{M}\mathbbm{1}(\hat{y}^{v}_{i}\neq\hat{y}^{w}_{i})$,
where $M$ is total number of test instances, $\hat{y}^{w}_{i}$ is the
predicted label of $i$-th instance on $w$-th view, and $\mathbbm{1}$ is the
indicator function that returns 1 if the condition is satisfied and 0
otherwise. By applying Trust Discounting, both TMC’s and ETMC’s conflicts
between different views are significant reduced. As an example, the CR on
Scene15 is visualized by heatmap, shown in Figure 4. The colors in the heatmap
generated by our method are noticeably more blue (or less red) than those of
the baselines, indicating that the conflict ratio has been reduced by our
method.

###  B.2 Explanation for the decrease of AUC-ROC for Uncertainty-Aware
Prediction on CUB dataset

Combined View
---

Pseudo View
---

Feat View 1
---

Feat View 2
---

Figure 5: Bar chart for belief mass on each label and predictive uncertainy on
one instance of CUB. GT indicates the ground truth label of the selected
instance.

As the error case displayed in Figure 5, ETF corrects the error prediction
made by ETMC. However, even though the combined prediction is correct after
applying trust discounting, the predictive uncertainty is still relatively
high. If ETF corrects previously incorrect predictions but assigns them
relatively high uncertainty scores (e.g., 0.4), it may lead to a decrease in
the AUC-ROC for predictive uncertainty. This is because AUC-ROC evaluates the
model’s ability to discriminate between correct and incorrect predictions
based on uncertainty scores. Correcting predictions while maintaining high
uncertainty scores can make it more challenging for the model to distinguish
between correct and incorrect predictions, resulting in a lower AUC-ROC score,
even though the accuracy improves.

Handwritten
---

Caltech101
---

PIE
---

Scene15
---

HMDB
---

CUB
---

Figure 6: Performance comparison of simulated conflict predictions.

###  B.3 Simulating Conflicting Predictions with Noisy Instances

Besides the one we show in Figure 3, we also plot the model performance for
evidence-based methods that do not incorporate pseudo views, as shown in
Figure 6. Our method TF consistently outperforms other methods like TMC and
ECML.

Handwritten
---

Caltech101
---

PIE
---

Scene15
---

HMDB
---

CUB
---

Figure 7: Performance comparison of simulated conflict predictions.

###  B.4 Ablation Study of Hyper-parameters

We presented results using a non-over-tuned empirical value of 1 for the
hyperparameter ’warm-up epochs’ in all previous experiments. These results
indicated our model’s robust performance, as it consistently outperformed
baseline models. Here we demonstrate the effect of this hyperparameter using
more finely graded values, starting from 0 (which indicates no warm-up stage)
and increasing steadily, for example, to 2, 5, and 10.

From Figure 7, we can find that incorporating warm-up stage (warm-up epochs >
0) can generally results in better model performance. For some datasets (e.g.
HMDB), increasing the number of warm-up epochs further improves performance
compared to the results previously reported. This observation suggests that
adjusting this value based on the specific dataset can lead to enhanced
performance.

##  Appendix C Technical Requirements and Execution

Table 8: Execution Time for ETF and TF on 24GB RTX3090 (Single run) Method | Handwritten | Caltech101 | PIE | Scene15 | HMDB | CUB
---|---|---|---|---|---|---
TF | 4min30s | 8min40sec | 55s | 4min50sec | 5min30s | 40sec
ETF | 5min | 11min | 1min20sec | 7min30sec | 8min | 50sec

###  C.1 Execution Time

We report the execution time of single run of the training process, which is
shown in Table 8. For example, 1min20sec in the table for ETF on PIE means
training ETC on PIE dataset using one 24GB RTX3090 GPU card will cost about 1
minute and 20 seconds. In our experiment, we will repeat this step 10 time for
calculating the mean and standard error.