# Learning semantic image quality for fetal ultrasound from noisy ranking
annotation

Manxi Lin1,∗, Jakob Ambsdorf2,4,∗, Emilie Pi Fogtmann Sejer3, Zahra Bashir3,
Chun Kit Wong1,4,
Paraskevas Pegios1,4, Alberto Raheli1, Morten Bo Søndergaard Svendsen2,3, Mads
Nielsen2,4,
Martin Grønnebæk Tolsgaard,3, Anders Nymark Christensen1, Aasa Feragen1,4 1DTU
Compute, Technical University of Denmark, Kgs Lyngby, Denmark
2Department of Computer Science, University of Copenhagen, Copenhagen, Denmark
3Copenhagen Academy of Medical Education and Simulation (CAMES),
Rigshospitalet, Copenhagen, Denmark
4Pioneer Centre for AI, Copenhagen, Denmark
Email: {manli<EMAIL_ADDRESS><EMAIL_ADDRESS>∗Equal contribution

###### Abstract

We introduce the notion of semantic image quality for applications where image
quality relies on semantic requirements. Working in fetal ultrasound, where
ranking is challenging and annotations are noisy, we design a robust coarse-
to-fine model that ranks images based on their semantic image quality and
endow our predicted rankings with an uncertainty estimate. To annotate
rankings on training data, we design an efficient ranking annotation scheme
based on the merge sort algorithm. Finally, we compare our ranking algorithm
to a number of state-of-the-art ranking algorithms on a challenging fetal
ultrasound quality assessment task, showing the superior performance of our
method on the majority of rank correlation metrics.

##  I Introduction

Image quality is crucial for an array of medical imaging tasks [1, 2], but
acquiring high-quality images is challenging in terms of cost, time,
equipment, and operator skill. While in computer vision, image quality tends
to encapsulate low-level properties like sharpness or artefacts [3], the
quality of medical images like fetal ultrasound pose complex requirements on
semantic image content [2, 4]. Here, images of "sufficient quality" satisfy a
set of standardized requirements [5], leading us to define a new _semantic
image quality_ paradigm.

We consider image quality assessment as a ranking task, as opposed to existing
classification of images as either good enough or not, via regression on a
global score. Classification is too coarse, as the "good enough" image might
be unattainable if the patient is too ill to sit still; if the operator is not
sufficiently skilled, or if high patient BMI or fetal position inhibits high-
quality acquisition. We still want the best image quality possible, which
requires finely ranking one image as better than another. Moreover,
applications such as navigation and auto-capture for ultrasound acquisition
assistance also depend on detecting small improvements in image quality.

Figure 1: Fetal ultrasound _standard planes_ are particular 2D cross sections
of the fetal anatomy, defined by the presence, absence, and visual quality of
specific organs. Above, 4 femur planes are sorted by their standard plane
quality – which is challenging, as the assessment depends on multiple
simultaneous criteria.

While this could be done by regressing on a quality score, we hypothesize that
annotated scores or categories can be imprecise and biased [6], as the
complexity of semantic image quality makes the task naturally challenging. We
observe this through intra-observer inconsistency in annotated scores and
rankings, also supported by previous work [4]. However, when instead
annotating rankings, the annotators do not need to sense the scale of sample
difference during annotation. The annotators are thus asked to perform a
simpler task - instead of ’how much this image is better than the other’, the
annotators only need to answer ’is this image better than the other’. This is
supported by literature on human decision-making, which indicates that humans
make better assessments when asked to rank than to score. Motivated by this,
we collected image quality rankings rather than image quality scores [7], and
learned image _ranking_ directly from _ranking_ annotations, rather than rank
images implicitly by training regression or classification models.

In this paper, we studied the problem of ranking images according to their
semantic quality. We contribute:

  * •

Instead of classifying images as "good enough" or not, we propose an ordinal-
regression-based network to learn semantic image quality ranking in a coarse-
to-fine manner.

  * •

Modelling of the predictions’ ranking uncertainty.

  * •

Validation against state-of-the-art on a fetal ultrasound image quality
assessment task.

  * •

In order to train algorithms for ranking, we propose a ranking annotation
scheme based on the merge sort algorithm, enabling easy annotation of ranking.

Note that our developed methodology applies to any semantic image quality
assessment application.

##  II Related Work

_Learning-to-rank_ (LTR) models sort samples according to their relevance for
a given query [8]. Pointwise [9] approaches, which score single samples,
suffer from lack of context, whereas pairwise [10] or listwise [11, 12, 8]
approaches predict the relative order of pairs or complete lists,
respectively. Validation metrics for listwise ranking, e.g. Normalized
Discounted Cumulative Gain (NDCG), are not differentiable and cannot be used
as loss functions. Existing work thus optimize an upper bound [13, 14] or
approximate the metric [8]. SoDeep uses a sorting approximation of non-
differentiable metrics trained on synthetic data [15]. While LTR models are
popular for image quality assessment [6, 16, 17], they are mostly used as
auxiliary tasks for regressing quality scores. An exception is [6], which
generates different levels of image corruption and learns image quality from
pairwise ranking. For ultrasound, LTR has been used as an auxiliary loss for
regression [3, 2], but synthetic noise is difficult to generate for fetal
ultrasound quality assessment [2]. We instead learn to rank semantic image
quality directly from noisy ranking annotations.

_Relative attribute learning_ quantifies the strength of attributes relative
to other images [18]. This is exemplified by attributes such as ’young’,
’natural’, and ’smiling’, which are easy to compare between two images, but
harder to numerically score. Existing approaches include siamese networks
optimized with a Hinge or rank SVM loss [19, 20]; pairwise ranking treated as
binary classification as in RankNet [21]; or learning robust relative
attributes from noisy data by excluding noisy pairs [22]. Our _semantic image
quality_ is a relative attribute. Relative attributes are usually modeled for
sampled subsets, often pairs. We, conversely, learn to rank semantic image
quality from full ranking annotations of the dataset, coupled with an approach
to assess the uncertainty of the learned rankings.

Figure 2: Network architecture of the proposed ORBNet.

_Ordinal regression_ , situated between regression and classification,
predicts pointwise ordered class labels, rather than determining the ranking
of a set. Classical approaches reformulate ordinal regression as multiple
binary classification tasks [23, 24, 25, 26, 27], but these methods cannot
guarantee ordinal consistency. In [28] weight-sharing constraints are imposed
in the output layer of a deep network, giving theoretical guarantees for
consistent confidence scores and regularization. In [29] a robust deep ordinal
regression method is proposed along with loss correction for class-dependent
label noise [30]. We use ordinal regression to learn a coarse global ranking.

##  III Method

###  III-A Learning to Rank with ORBNet

We aim to rank a dataset $D=\\{\mathbf{x}_{i},y_{i}\\}_{i=i}^{n}$, where
$\mathbf{x}_{i}$ is the $i$-th image and $y_{i}\in\mathbb{N}$ its annotated
rank. For simplicity, we assume no ties, i.e. $y_{i}\neq y_{j}\ \forall\ i\neq
j$. We tackle this by learning an intermediate score $\mathbf{x}_{i}\mapsto
s_{i}$, which is easily converted to a rank $\hat{y}_{i}$. For this, we
propose an ordinal-regression-based neural network (ORBNet) (Fig 2), with
three parts: (1) a pre-trained feature extractor $\theta$ gathering high-level
semantic features $f_{i}$ from $\mathbf{x}_{i}$ (see Sec. IV-A), (2) an
ordinal module $\psi$ to get a coarse, global ranking of the extracted
features, and (3) an offset module $\zeta$ giving a fine-grained, local
scoring.

#### Coarse global ranking with ordinal regression

We first learn a reliable rough ranking via ordinal regression, by splitting
ranking labels into $m$ bins. Each ranking label $y_{i}$ thus is encoded as a
set of binary labels ${\\{b_{ij}\\}}_{j=1}^{m-1}$, where

$b_{ij}=\begin{cases}1&\textrm{if }y_{i}>=j\tau\\\ 0&\textrm{otherwise,}\\\
\end{cases}$ (1)

where $\tau=\frac{n}{m}$ is the range of each bin. The ordinal mapping
$\psi\colon f_{i}\mapsto l_{i}$ maps to a coarse global ranking
$l_{i}\in\mathbb{R}^{m-1}$, binning the image $x_{i}$ within the whole
dataset. Each coordinate of $l_{i}$, if activated by the sigmoid $\sigma$,
represents the probability of $x_{i}$ larger than the left bound of the bin.
We train $l_{i}$ to align with ${\\{b_{ij}\\}}_{j=1}^{m-1}$ using binary
cross-entropy $l_{coarse}^{i}=-\sum_{j=1}^{m-1}(log(\sigma(l_{ij}))\cdot
b_{ij}+(1-log(\sigma(l_{ij})))\cdot(1-b_{ij}))$, where $l_{ij}$ is the $j$-th
element of $l_{i}$. The bin to which the image is assigned defines a score
$\bar{s}_{i}=\tau\cdot\sum_{j=1}^{m-1}l_{ij}$, which is within
$[0,\frac{n(m-1)}{m}]$, i.e., the left bound of the bin the image belongs to,
in a normalized scale. We implement $\psi$ as a multi-layer perceptron (MLP);
the output layer biases are not shared [28].

#### Fine-tune local ranking by learning an offset

The effect of noise is reduced through our rough binned ranking, but we also
lose fine-grained information. We therefore add an offset module $\zeta$
learning small bin-wise offsets. The offset module
$\zeta\colon\\{f_{i},l_{i}\\}\mapsto\widetilde{s}$ is an MLP taking the
concatenation $(f_{i},l_{i})$ as input, outputting an offset
$\widetilde{s}\in\mathbb{R}$ which is limited to a local range, to retain the
effect of the predicted global order. This gives a final predicted quality
score $s_{i}=\bar{s}_{i}+\tau\cdot\sigma(\widetilde{s})\in[0,1]$\- for
$x_{i}$, which is used to train the offset layer with RankNet [10]. For each
pair of images in $D$,
$l_{fine}^{(i,j)}=-(log(\sigma(s_{i}-s_{j}))p_{ij}+(1-log(\sigma(s_{i}-s_{j})))(1-p_{i,j}))$,
where

$p_{ij}=\begin{cases}1,&y_{i}>y_{j}\\\ 0,&y_{i}<y_{j}\\\
0.5,&y_{i}=y_{j}\end{cases}.$ (2)

The ORBNet is trained end-to-end using $l=l_{coarse}+l_{fine}$.

###  III-B Assessing Ranking Uncertainty

As our ranking annotations are noisy, we also expect uncertain predictions.
While the variance of the predicted ranks is easy to quantify, this would not
help the user, as the position of an image in the quality-ranked dataset is
not clinically relevant. Instead, we choose to assess the uncertainty of
binary decisions of the form "which of two images is better?"

We evaluate the uncertainty of pairwise ORBNet rankings using MC Dropout [31]
at inference time. Given a pair of images $\mathbf{x}^{a}$ and
$\mathbf{x}^{b}$, we use $N$ forward passes of the model with Dropout enabled,
to obtain multiple predicted ranking scores $\widetilde{y}_{a}$ for each image
$\mathbf{x}^{a}$. The share of predicted rankings $\mathbf{x}^{a}$ over
$\mathbf{x}^{b}$ in the form of a Bernoulli distribution
$P(\widetilde{y}_{a}>y_{b})$ is then interpreted as a confidence in ranking
$\mathbf{x}_{a}$ over $\mathbf{x}_{b}$.

###  III-C Annotating Rankings of Entire Datasets via Merge Sort

Considering the combinatorial complexity that is introduced when ranking
entire lists, annotating complete dataset rankings is expensive, as every pair
of items needs to be compared. In this paper, we propose a scheme for
annotating a full dataset ranking based on the merge sort algorithm,
illustrated in Fig. 3:

Figure 3: Illustration of the annotation process.

  1. 1.

Divide the original dataset into sub-sets of size $n$, where $n$ is a
controllable hyperparameter;

  2. 2.

Sort each sub-set;

  3. 3.

Until the entire dataset is merged, do:

    1. (a)

Select the two highest ranking instances from two adjacent sub-sets;

    2. (b)

Pick the higher ranking instance of the two, remove it from the sub-set and
append it to a new list;

    3. (c)

When all sub-sets have been merged, continue by merging the new larger lists;

The proposed annotation scheme reduces the annotation workload compared to
directly sorting an entire list (e.g. in a drag-and-drop interface) by
choosing an appropriate value for $n$ and limiting the merge process to an
iterative selection of the higher-quality image. The value of $n$ is selected
based on (a) fitting all images in sufficient size and resolution on a screen,
and (b) not overwhelming the working memory of the annotator (typically $7\pm
2$ items [32]), to ensure that the initial ranking can be performed quickly
and with a low error-rate. In practice, we determined $n=6$ to be suitable for
our use case on a 27-inch screen. The initial sorting of $n$-lets increases
the efficiency of the annotation process by reducing the number of merge
iterations compared to starting with image pairs.

##  IV Experiments

###  IV-A Data and Implementation

We consider 3rd-trimester fetal ultrasound screening, including 300 femur
images, 300 abdomen images, and 264 head images from a national fetal
ultrasound screening database [33]. These images were annotated following Sec.
III-C (annotator A1, MD, Ph.D. fellow in Fetal Medicine: femur and head
images, annotator A2, Fetal Medicine consultant, Professor: abdomen images),
defining semantic image quality via compliance with the ISUOG guidelines [5].
To analyze intra- and inter-annotator ranking consistency, a random sample of
60 head images was additionally annotated by A1 (several days after performing
the first trial) as well as by A2. We kept these 60 head images as a clean
test set and trained the model on the rest of the head images, where $10\%$
images were left for validation.

The property bottleneck of a pre-trained progressive concept bottleneck
network (PCBM) [33], which are both representative and human-interpretable via
the ISUOG criteria [5], was used as a feature extractor. This generates
features ${\\{f_{i}\\}}_{i=1}^{n}$, used as input for the following stages in
our ORBNet: The ordinal module is a three-layer MLP including 512, 128, and
$m-1=9$ neurons in each hidden layer, with $m=10$ selected empirically. The
offset module is another three-layer MLP containing 512, 128, and 1 neurons
per hidden layer. The first two layers of the offset module share weights with
the ordinal module.

The ordinal and offset modules were trained with SGD for 30 epochs with
momentum 0.9, batch size 32, and L2 regularization 1e-4. The learning rate was
initialized with 1e-2 and multiplied by 0.1 after 10 epochs without
improvement. All hyperparameters were selected via ablation experiments.

(a)  (b)  (c)  (d)  (e)  (f)  (g)

Figure 4: Validating ranking uncertainty: The plots correspond to anchor
images $x_{a}$ the $1^{st},10^{th},20^{th},30^{th},40^{th},50^{th}$, shown in
blue – with queries $x_{q}$ of systematically increasing rank (dots – color
indicates predicted rank, $y$-axis indicates confidence).

###  IV-B Validating Ranking Performance

#### Baselines

Six existing methods were used as baselines: RankNet [10], Hinge loss [19],
ListNet [11], SoDeep [15], along with the ‘traditional’ image quality
assessment method RankIQA [17] and a standard regression model using an L1
loss to regress the percentile rank of the image. We constructed the baseline
model of a three-layer MLP including 512, 128, and 1 neurons in each hidden
layer, which was trained in mini-batches. RankNet and Hinge loss were applied
to all the image pairs within a batch. ListNet optimized the permutation
probability distribution of the predicted scores to ground truth rankings. We
trained two ListNet models: ’ListNet-local’ was optimized to the rankings of
samples within the minibatch, and ’ListNet-global’ optimized to the global
rankings of samples within the full training set. We trained an LSTM as the
sorter for SoDeep. All the baseline models were trained with their optimal
hyperparameter settings.

TABLE I: Performance on the clean test set. Here, NDCG@n means top-n
normalized discounted cumulative gain.

Method | SPC | PAcc | PRC | KTC | NDCG@3 | NDCG@5
---|---|---|---|---|---|---
RankNet | $0.594$ | $71.9$ | $0.616$ | $0.437$ | 0.881 | 0.952
Hinge loss | $0.602$ | $71.7$ | $0.604$ | $0.434$ | $0.790$ | $0.936$
ListNet-loc. | $0.588$ | $70.2$ | $0.602$ | $0.405$ | $0.675$ | $0.740$
ListNet-glob. | $0.585$ | $70.1$ | $0.599$ | $0.402$ | $0.676$ | $0.742$
SoDeep | $0.613$ | $71.9$ | $0.639$ | $0.438$ | $0.746$ | $0.841$
RankIQA | $0.065$ | $50.2$ | $0.124$ | $0.003$ | $0.538$ | $0.649$
Regression | $0.210$ | $57.1$ | $0.195$ | $0.141$ | $0.271$ | $0.287$
ORBNet (ours) | 0.652 | 73.8 | 0.673 | 0.477 | $0.837$ | $0.925$

#### Validation on a cleaned test set

To minimize the effect of noise on validation, we first validate model
performance on 60 head images annotated with scores by annotator A1 repeated
at two different time points, which were averaged to produce a robust
annotator rank. Tab. I shows performance evaluated with five metrics: Spearman
correlation of ground truth and predicted rankings (SPC), overall
classification accuracy of every pair in the dataset (PAcc), Pearson
correlation of predicted and annotated image scores (PRC), Kendall’s $\tau$
coefficient (KTC), and the top-3 and top-5 normalized discounted cumulative
gain. ORBNet outperforms all baselines on SPC, PAcc, PRC, and KTC. On NDCG, it
is slightly outperformed by RankNet and Hinge Loss but outperforms the
remaining baselines by far.

#### 10-fold cross-validation

We also conducted a 10-fold cross-validation on all the anatomies, assessing
model performance via Spearman correlation, see Tab. II. ORBNet outperforms
baselines considerably on Head and Abdomen.

TABLE II: Spearman correlation of ground truth and predicted ranking under 10-fold cross-validation.  Method | Head | Abdomen | Femur
---|---|---|---
RankNet | $0.68\pm 0.07$ | $0.58\pm 0.12$ | $0.42\pm 0.13$
Hinge loss | $0.70\pm 0.06$ | $0.58\pm 0.13$ | $0.42\pm 0.15$
ListNet-local | $0.54\pm 0.12$ | $0.50\pm 0.14$ | $\textbf{0.50}\pm 0.14$
ListNet-global | $0.53\pm 0.14$ | $0.51\pm 0.13$ | $0.32\pm 0.16$
SoDeep | $0.66\pm 0.13$ | $0.58\pm 0.13$ | $0.45\pm 0.15$
RankIQA | $0.01\pm 0.07$ | $-0.22\pm 0.11$ | $0.20\pm 0.06$
Regression | $-0.02\pm 0.18$ | $0.09\pm 0.19$ | $0.07\pm 0.15$
ORBNet (ours) | $\textbf{0.81}\pm 0.09$ | $\textbf{0.67}\pm 0.15$ | $0.44\pm 0.14$

#### Validating ranking uncertainty

We compare a fixed anchor image $\mathbf{x}_{a}$ to query images
$\mathbf{x}_{q}$ with ground truth ranking scores $y_{q}$ from the entire test
set. Confidence was estimated using dropout probability 0.5 with 10 passes per
query image. Fig. 4 illustrates pairwise confidences of ranking
$\mathbf{x}_{a}$ (blue dotted line) over a systematically varying
$\mathbf{x}_{q}$ (dots), ordered by increasing query image rank, to show how
ranking uncertainty depends on how similarly the anchor and query images are
ranked by the annotator. This experiment is presented using the
$1^{st},10^{th},20^{th},30^{th},40^{th},50^{th}$, and last ranked image in our
clean test set as anchor image $x_{a}$. Note how, when the anchor image
(dotted blue line) is ranked very high or low, most pairs have a high
confidence ranking, whereas the confidence is lower – and the uncertainty
higher – for neighboring images when the query image is ranked in the middle
of the dataset.

#### Model variance

To assess stability, we performed 10-fold cross-validation with 10 different
random seeds. Tab. III shows the mean and standard deviation of SPC across the
$100$ models per anatomy, indicating relatively robust ranking performance.

TABLE III: Mean and standard deviation of SPC of the ground truth and predicted image ranking in 10-fold cross validation. Head | Abdomen | Femur
---|---|---
$0.7987\pm 0.09$ | $0.6648\pm 0.16$ | $0.4369\pm 0.14$

##  V Discussion, Limitations, and Conclusion

Our model is built on the assumption that annotated image quality rankings are
noisy. This is confirmed by the intra-rater consistency for annotator A1 which
gives an SPC of $0.794$ between the two repeated rankings of the $n=60$ head
image dataset and an inter-rater consistency of $0.549$ with annotator A2.
Thus, while annotations were performed by experts and based on the same
criteria, both intra- and inter-rater consistency is limited, emphasizing that
fine-grained semantic image quality assessment is inherently noisy and
subjective.

#### Efficient ranking annotation via merge sort

Our proposed annotation scheme effectively achieves a complete ranking of the
dataset. It does not, however, alleviate annotation noise. In part, images
that are difficult to compare to each other should ideally be oversampled to
reduce annotation noise. Further, as we only compare image pairs once, we
propagate annotation errors as later rankings depend on wrong decisions.
Future work should therefore consider noise reduction while keeping the
annotation process efficient for straightforward comparisons.

#### Annotation uncertainty

While we quantify ranking uncertainty in this study, we have not explored or
modeled the underlying sources of uncertainty. In particular, our consistency
assessment indicates aleatoric uncertainty stemming from the annotations,
while MC dropout only captures epistemic uncertainty. Thus, discovering and
explicitly modeling different sources of uncertainty are important open
problems.

#### Conclusion

We address semantic image quality assessment via image ranking for a
challenging fetal ultrasound task. Fetal ultrasound images are usually
assessed as either acceptable or not. This goal may be unattainable in
clinical practice, but we still want the best image quality possible. Thus, we
have introduced a simple scheme for annotating global ranking of image sets.
We propose an ordinal-regression-based neural network to learn image ranking
in a coarse-to-fine manner, and utilize MC Dropout to quantify ranking
uncertainty. We show that our model outperforms baselines in the challenging
fetal ultrasound quality assessment task.

## Compliance with ethical standards

This study is part of a larger project on AI support for clinicians carrying
out fetal ultrasound examinations. Approvals from The Danish Data Protection
Agency (Protocol No. P-2019-310) and by The Danish Patient Safety Authority
(Protocol No. 3-3031-2915/1) have been obtained for the project. The project
was submitted to the Regional Ethics Committee, which has assessed that the
study is exempt from The Scientific Ethical Treatment Act (jr. nr. 21024445).

## Acknowledgment

This work was supported by the DIREC project EXPLAIN-ME (9142-00001B), the
Novo Nordisk Foundation through the Center for Basic Machine Learning Research
in Life Science (NNF20OC0062606), and the Pioneer Centre for AI, DNRF grant nr
P1.

## References

  * [1] C. Ozer, A. Guler, A. T. Cansever, and I. Oksuz, “Explainable image quality assessment for medical imaging,” _arXiv preprint arXiv:2303.14479_ , 2023.
  * [2] J. Lyu, S. H. Ling, S. Banerjee, J. Zheng, K.-L. Lai, D. Yang, Y.-P. Zheng, X. Bi, S. Su, and U. Chamoli, “Ultrasound volume projection image quality selection by ranking from convolutional ranknet,” _Computerized Medical Imaging and Graphics_ , vol. 89, p. 101847, 2021.
  * [3] C. Chen, X. Yang, R. Huang, X. Hu, Y. Huang, X. Lu, X. Zhou, M. Luo, Y. Ye, X. Shuang _et al._ , “Fine-grained correlation loss for regression,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VIII_. Springer, 2022, pp. 663–672.
  * [4] F. Zarb, L. Rainford, and M. F. McEntee, “Image quality assessment tools for optimization of ct images,” _Radiography_ , vol. 16, no. 2, pp. 147–153, 2010.
  * [5] L. Salomon, Z. Alfirevic, F. Da Silva Costa, R. Deter, F. Figueras, T. a. Ghi, P. Glanc, A. Khalil, W. Lee, R. Napolitano _et al._ , “Isuog practice guidelines: ultrasound assessment of fetal biometry and growth,” _Ultrasound in obstetrics & gynecology_, vol. 53, no. 6, pp. 715–723, 2019.
  * [6] F. Gao, D. Tao, X. Gao, and X. Li, “Learning to rank for blind image quality assessment,” _IEEE transactions on neural networks and learning systems_ , vol. 26, no. 10, pp. 2275–2290, 2015.
  * [7] A.-W. Harzing, J. Baldueza, W. Barner-Rasmussen, C. Barzantny, A. Canabal, A. Davila, A. Espejo, R. Ferreira, A. Giroud, K. Koester _et al._ , “Rating versus ranking: What is the best way to reduce response and language bias in cross-national research?” _International business review_ , vol. 18, no. 4, pp. 417–432, 2009.
  * [8] R. Swezey, A. Grover, B. Charron, and S. Ermon, “Pirank: Scalable learning to rank via differentiable sorting,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 21 644–21 654, 2021.
  * [9] K. Crammer and Y. Singer, “Pranking with ranking,” _Advances in neural information processing systems_ , vol. 14, 2001.
  * [10] C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender, “Learning to rank using gradient descent,” in _Proceedings of the 22nd international conference on Machine learning_ , 2005, pp. 89–96.
  * [11] Z. Cao, T. Qin, T.-Y. Liu, M.-F. Tsai, and H. Li, “Learning to rank: from pairwise approach to listwise approach,” in _Proceedings of the 24th international conference on Machine learning_ , 2007, pp. 129–136.
  * [12] Y. Lan, Y. Zhu, J. Guo, S. Niu, and X. Cheng, “Position-aware listmle: A sequential learning process for ranking.” in _UAI_ , 2014, pp. 449–458.
  * [13] H. Valizadegan, R. Jin, R. Zhang, and J. Mao, “Learning to rank by optimizing ndcg measure,” _Advances in neural information processing systems_ , vol. 22, 2009.
  * [14] Y. Yue, T. Finley, F. Radlinski, and T. Joachims, “A support vector method for optimizing average precision,” in _Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval_ , 2007, pp. 271–278.
  * [15] M. Engilberge, L. Chevallier, P. Pérez, and M. Cord, “Sodeep: a sorting deep net to learn ranking loss surrogates,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 792–10 801.
  * [16] K. Ma, W. Liu, T. Liu, Z. Wang, and D. Tao, “dipiq: Blind image quality assessment by learning-to-rank discriminable image pairs,” _IEEE Transactions on Image Processing_ , vol. 26, no. 8, pp. 3951–3964, 2017.
  * [17] X. Liu, J. Van De Weijer, and A. D. Bagdanov, “Rankiqa: Learning from rankings for no-reference image quality assessment,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 1040–1049.
  * [18] D. Parikh and K. Grauman, “Relative attributes,” in _2011 International Conference on Computer Vision_. IEEE, 2011, pp. 503–510.
  * [19] X. Yang, T. Zhang, C. Xu, S. Yan, M. S. Hossain, and A. Ghoneim, “Deep relative attributes,” _IEEE Transactions on Multimedia_ , vol. 18, no. 9, pp. 1832–1842, 2016.
  * [20] S. A. A. Ahmed and B. Yanikoglu, “Relative attribute classification with deep-ranksvm,” in _Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event, January 10–15, 2021, Proceedings, Part II_. Springer, 2021, pp. 659–671.
  * [21] Y. Souri, E. Noury, and E. Adeli, “Deep relative attributes,” in _Computer Vision–ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part V 13_. Springer, 2017, pp. 118–133.
  * [22] Q. Xu, Z. Yang, Y. Jiang, X. Cao, Y. Yao, and Q. Huang, “Not all samples are trustworthy: Towards deep robust svp prediction,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 6, pp. 3154–3169, 2020.
  * [23] K. Crammer and Y. Singer, “Pranking with ranking,” in _Advances in Neural Information Processing Systems_ , T. Dietterich, S. Becker, and Z. Ghahramani, Eds., vol. 14. MIT Press, 2001. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2001/file/5531a5834816222280f20d1ef9e95f69-Paper.pdf
  * [24] L. Shen and A. K. Joshi, “Ranking and reranking with perceptron,” _Machine Learning_ , vol. 60, pp. 73–96, 2005.
  * [25] W. Chu and S. S. Keerthi, “New approaches to support vector ordinal regression,” in _Proceedings of the 22nd international conference on Machine learning_ , 2005, pp. 145–152.
  * [26] D. Cao, Z. Lei, Z. Zhang, J. Feng, and S. Z. Li, “Human age estimation using ranking svm,” in _Biometric Recognition: 7th Chinese Conference, CCBR 2012, Guangzhou, China, December 4-5, 2012. Proceedings 7_. Springer, 2012, pp. 324–331.
  * [27] K. Antoniuk, V. Franc, and V. Hlaváč, “V-shaped interval insensitive loss for ordinal classification,” _Machine Learning_ , vol. 103, pp. 261–283, 2016.
  * [28] W. Cao, V. Mirjalili, and S. Raschka, “Rank consistent ordinal regression for neural networks with application to age estimation,” _Pattern Recognition Letters_ , vol. 140, pp. 325–331, 2020.
  * [29] B. Garg and N. Manwani, “Robust deep ordinal regression under label noise,” in _Asian Conference on Machine Learning_. PMLR, 2020, pp. 782–796.
  * [30] G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and L. Qu, “Making deep neural networks robust to label noise: A loss correction approach,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1944–1952.
  * [31] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in _international conference on machine learning_. PMLR, 2016, pp. 1050–1059.
  * [32] G. A. Miller, “The magical number seven, plus or minus two: Some limits on our capacity for processing information.” _Psychological review_ , vol. 63, no. 2, p. 81, 1956.
  * [33] M. Lin, A. Feragen, Z. Bashir, M. G. Tolsgaard, and A. N. Christensen, “I saw, i conceived, i concluded: Progressive concepts as bottlenecks,” _arXiv preprint arXiv:2211.10630_ , 2022.