# Memory-Assisted Sub-Prototype Mining for Universal Domain Adaptation

Yuxiang Lai Xinghong Liu Tao Zhou Yi Zhou
Southeast University Nanjing University of Science and Technology
Correspondence to Yi Zhou (yizhou.szcn@gmail.com).

###### Abstract

Universal domain adaptation aims to align the classes and reduce the feature
gap between the same category of the source and target domains. The target
private category is set as the unknown class during the adaptation process, as
it is not included in the source domain. However, most existing methods
overlook the intra-class structure within a category, especially in cases
where there exists significant concept shift between the samples belonging to
the same category. When samples with large concept shift are forced to be
pushed together, it may negatively affect the adaptation performance.
Moreover, from the interpretability aspect, it is unreasonable to align visual
features with significant differences, such as fighter jets and civil
aircraft, into the same category. Unfortunately, due to such semantic
ambiguity and annotation cost, categories are not always classified in detail,
making it difficult for the model to perform precise adaptation. To address
these issues, we propose a novel Memory-Assisted Sub-Prototype Mining (MemSPM)
method that can learn the differences between samples belonging to the same
category and mine sub-classes when there exists significant concept shift
between them. By doing so, our model learns a more reasonable feature space
that enhances the transferability and reflects the inherent differences among
samples annotated as the same category. We evaluate the effectiveness of our
MemSPM method over multiple scenarios, including UniDA, OSDA, and PDA. Our
method achieves state-of-the-art performance on four benchmarks in most cases.

##  1 Introduction

Unsupervised Domain Adaptation (UDA) [15, 22, 41, 44, 9, 19, 21] has become a
crucial research area of transfer learning, as it allows models trained on a
specific dataset to be applied to related but distinct domains. However,
traditional UDA methods are limited by the assumption that the source and
target domains have to share the same label space. This assumption is
problematic in real-world scenarios where the target distribution is complex,
open, and diverse. Universal Domain Adaptation (UniDA) represents a strategy
to address the limitations of traditional unsupervised domain adaptation
methods. In the UniDA, the target domain has a different label set than the
source domain. The goal is to correctly classify target domain samples
belonging to the shared classes in the source label set, while any samples not
conforming to the source label set are treated as "unknown". The term
"universal" characterizes UniDA as not relying on prior knowledge about the
label sets of the target domain. UniDA relaxes the assumption of a shared
class space while aiming to learn domain-invariant features across a more
broad range of domains.

Figure 1: Illustration of our motivation. (a) Examples of concept shift and
intra-class diversity in DA benchmarks. For the class of alarm clocks, we find
that digital clocks, pointer clocks, and alarm bells should be set in
different sub-classes. For the class of airplane, we find that images
containing more than one plane, single jetliner, and turboprop aircraft should
be differently treated for adaptation. (b) Previous methods utilize one-hot
labels to guide classifying without considering the intra-class distinction.
Consequently, the model forces all samples from the same class to converge
towards a single center, disregarding the diversity in the class. Our method
clusters samples with large intra-class differences into separate sub-class,
providing a more accurate representation. (c) During domain adaptation by our
design, the samples in the target domain can also be aligned near the sub-
class centers with similar features rather than just the class centers
determined by labels.

Despite being widely explored, most existing universal domain adaptation
methods [24, 47, 40, 39, 6, 34, 8, 26] overlook the internal structure
intrinsically presented within each image category. These methods aim to align
the common classes between the source and target domains for adaptation but
usually train a model to learn the class "prototype" representing each
annotated category. This is particularly controversial when significant
concept shift exists between samples belonging to the same category. These
differences can lead to sub-optimal feature learning and adaptation if the
intra-class structure is neglected during training. Since this type of
semantic ambiguity without fine-grained category labels occurs in almost all
of the DA benchmarks, all the methods will encounter this issue.

In this paper, we aim to propose a method to learn the detailed intra-class
distinction and mine "sub-prototypes" for better alignment and adaptation.
This kind of sub-prototype is the further subdivision of each category-level
prototype, which represents the "sub-class" of the annotated categories. The
main idea of our proposed approach lies in its utilization of a learnable
memory structure to learn sub-prototypes for their corresponding sub-classes.
This can optimize the construction and refinement of the feature space,
bolstering the classifier’s ability to distinguish class-wise relationships
and improve the model’s transferability across domains. A comparison between
our proposed sub-prototypes mining approach and previous methods is
illustrated in Figure 1. In previous methods, samples within a category were
forced to be aligned together in the feature space regardless of whether there
exist significant differences among them because the labels were one-hot
encoded. Contrastively, our sub-prototypes’ feature space distinguishes sub-
classes with apparent differences within the category, thus improving the
model’s accuracy of domain adaptation and interpretability.

Our proposed approach, named memory-assisted sub-prototype mining (MemSPM), is
inspired by the memory mechanism works [17, 10, 45, 36]. In our approach, the
memory generates sub-prototypes that embody sub-classes learned from the
source domain. During testing of the target samples, the encoder produces
embedding that is compared to source domain sub-prototypes learned in the
memory. Subsequently, an embedding for the query sample is generated through
weighted sub-prototype sampling in the memory. This results in reduced domain
shift before the embedding is passed to the classifier. Our proposal of mining
sub-prototypes, which are learned from the source domain memory, improves the
universal domain adaptation performance by promoting more refined visual
concept alignment.

MemSPM approach has been evaluated on four benchmark datasets (Office-31 [37],
Office-Home [46], VisDA [33], and Domain-Net [32]), under various category
shift scenarios, including PDA, OSDA, and UniDA. Our MemSPM method achieves
state-of-the-art performance in most cases. Moreover, we design a
visualization module for the sub-prototype learned by our memory to
demonstrate the interpretability of MemSPM. Our contributions can be
highlighted as follows:

  * •

We study the UniDA problem from a new aspect, which focuses on the negative
impacts caused by overlooking the intra-class structure within a category when
simply adopting one-hot labels.

  * •

We propose Memory-Assisted Sub-Prototype Mining(MemSPM), which explores the
memory mechanism to learn sub-prototypes for improving the model’s adaption
performance and interpretability. Meanwhile, visualizations reveal the sub-
prototypes stored in memory, which demonstrate the interpretability of MemSPM
approach.

  * •

Extensive experiments on four benchmarks verify the superior performance of
our proposed MemSPM compared with previous works.

##  2 Related Work

Closed-Set Domain Adaptation (CSDA). To mitigate the performance degradation
caused by the closed-set domain shift, [16, 29, 48] introduce adversarial
learning methods with the domain discriminator, aiming to minimize the domain
gap between source and target domains. Beyond the use of the additional domain
discriminator, some studies [41, 23, 50, 30, 13] have explored the use of two
task-specific classifiers, otherwise referred to as bi-classifier, to
implicitly achieve the adversarial learning. However, the previously mentioned
methods for CSDA cannot be directly applied in scenarios involving the
category shift.

Partial Domain Adaptation (PDA). PDA posits that private classes are exclusive
to the source domain. Representative PDA methods, such as those discussed in
[3, 49], employ domain discriminators with weight adjustments or utilize
source samples based on their resemblance to the target domain [5]. Methods
incorporating residual correction blocks in PDA have been introduced by Li et
al. and Liang et al. [25, 27]. Other research [7, 11, 38] explores the use of
Reinforcement Learning for source data selection within the context of PDA.

Open-Set Domain Adaptation (OSDA). Saito et al. [42] developed a classifier
inclusive of an additional “unknown” class intended to differentiate
categories unique to the target domain. Liu et al. [28] and Shermin et al.
[43] propose assigning individual weights to each sample depending on their
importance during domain adaptation. Jang et al. [20] strive to align the
source and target-known distributions, while concurrently distinguishing the
target-unknown distribution within the feature alignment process. The above
PDA and OSDA methods are limited to specific category shift.

Universal Domain Adaptation (UniDA) You et al. [47] proposed Universal
Adaptation Network (UAN) to deal with the UniDA setting that the label set of
the target domain is unknown. Li et al. [24] proposed Domain Consensus
Clustering to differentiate the private classes rather than treat the unknow
classes as one class. Saito et al. [40] suggested that using the minimum
inter-class distance in the source domain as a threshold can be an effective
approach for distinguishing between “known” and “unknown” samples in the
target domain. However, most existing methods [24, 47, 40, 39, 6, 34, 8, 26]
overlook the intra-class distinction within one category, especially in cases
where there exists significant concept shift between the samples belonging to
the same category.

##  3 Proposed Methods

###  3.1 Preliminaries

In unsupervised domain adaptation, we are provided with labeled source samples
$\mathcal{D}^{s}=\\{x^{s}_{i},y^{s}_{i})\\}^{n^{s}}_{i=1}$ and unlabeled
target samples $\mathcal{D}^{t}=\\{(x^{t}_{i})\\}^{n^{t}}_{i=1}$. As the label
set for each domain in UniDA setting may not be identical, we use $C_{s}$ and
$C_{t}$ to represent label sets for the two domains, respectively. Then, we
denote $C=C_{s}\cap C_{t}$ as the common label set. $\hat{C}_{s}$,
$\hat{C}_{t}$ are denoted as the private label sets of the source domain and
target domain, respectively. We aim to train a model on $\mathcal{D}^{s}$ and
$\mathcal{D}^{t}$ to classify target samples into $|C|+1$ classes, where
private samples are treated as unknown classes.

Our method aims to address the issue of intra-class concept shift that often
exists within the labeled categories in most datasets, which is overlooked by
previous methods. Our method enables the model to learn an adaptive feature
space that better aligns fine-grained sub-class concepts, taking into account
the diversity present within each category. Let $X$ denote the input query,
$Z$ denote the embedding extracted by the encoder, $L$ denote the data labels,
$\hat{Z}$ denotes the embedding obtained from the memory, $\hat{X}$ denote the
visualization of the memory, $\hat{L}$ denotes the prediction of the input
query, and the $K$ denotes the top-$K$ relevant sub-prototypes, respectively.
The overall pipeline is presented in Figure 2. More details will be described
in the following sub-sections.

Figure 2: Our model first utilizes a fixed pre-trained model as the encoder to
extract input-oriented embedding given an input sample. The extracted input-
oriented embedding is then compared with sub-prototypes learned in memory to
find the closest $K$. These $K$ are then weighted-averaged into a task-
oriented embedding to represent the input, and used for learning downstream
tasks. During the UniDA process, we adopt the cycle-consistent matching method
on the task-oriented embedding $\hat{Z}$ generated from the memory. Moreover,
a decoder is designed to reconstruct the image, allowing for visualizing of
the sub-prototypes in memory and verifying the effectiveness of sub-class
learning.

###  3.2 Input-Oriented Embedding vs. Task-Oriented Embedding

Usually, the image feature extracted by a visual encoder is directly used for
learning downstream tasks. We call this kind of feature as input-oriented
embedding. However, it heavily relies on the original image content. Since
different samples of the same category always vary significantly in their
visual features, categorization based on the input-oriented embedding is
sometimes unattainable. In our pipeline, we simply adopt a CLIP-based [35]
pre-trained visual encoder to extract the input-oriented embeddings, which is
not directly used for learning our downstream task.

In our MemSPM, we propose to generate task-oriented embedding, which is
obtained by serving input-oriented embedding as a query to retrieve the sub-
prototypes from our memory unit. We define
$f^{fixed}_{encode}(\cdot):X\rightarrow Z$ to represent the fixed pre-trained
encoder and $f^{UniDA}_{class}(\cdot):\hat{Z}\rightarrow\hat{L}$ to represent
the UniDA classifier. The input-oriented embedding $Z$ is used to retrieve the
relevant sub-prototypes from the memory. The task-oriented embedding $\hat{Z}$
is obtained using the retrieved sub-prototypes for classification tasks. In
conventional ways, $\hat{Z}=Z$, which means the $\hat{Z}$ is obtained directly
from $Z$. Our method obtains the $\hat{Z}$ by retrieving the sub-prototypes
from the memory, which differentiates $\hat{Z}$ from $Z$ and eliminates the
domain-specific information from the target domain during the testing phase.
As a result, it improves the performance of $f^{UniDA}_{class}(\cdot)$ when
performing UniDA.

###  3.3 Memory-Assisted Sub-Prototype Mining

The memory module proposed in MemSPM consists of two key components: a memory
unit responsible for learning sub-prototypes, and an attention-based
addressing [18] operator to obtain better task-oriented representation
$\hat{Z}$ for the query, which is more domain-invariant.

####  3.3.1 Memory Structure with Partitioned Sub-Prototype

The memory in MemSPM is represented as a matrix, denoted by
$M\in\mathbb{R}^{N\times S\times D}$, where $N$ indicates the number of memory
items stored, $S$ refers to the number of sub-prototypes partitioned in each
memory item, and $D$ represents the dimension of each sub-prototype. For
convenience, we assume $D$ is the same to the dimension of
$Z\in\mathbb{R}^{C}$ ( $\mathbb{R}^{D}$=$\mathbb{R}^{C}$). Let the vector
$m_{i,j}$, $\forall i\in[N]$ denote the $i\text{-th}$ row of $M$, where $[N]$
denotes the set of integers from 1 to $N$, $\forall j\in[S]$ denote the
$j\text{-th}$ sub-prototype of $M$ items, where $[S]$ denotes the set of
integers from 1 to $S$. Each $m_{i}$ denotes a memory item. Given a embedding
$Z\in\mathbb{R}^{D}$, the memory module obtains $\hat{Z}$ through a soft
addressing vector $W\in\mathbb{R}^{1\times 1\times N}$ as follows:

$\hat{Z}=W\cdot M=\Sigma^{N}_{i=1}w_{i,j=s_{i}}\cdot m_{i,j=s_{i}}\text{,}$
(1) $w_{i,j=s_{i}}=\text{argmax}(w_{i,j},dim=1)\text{,}$ (2)

where $W$ is a vector with non-negative entries that indicate the maximum
attention weight of each item’s sub-prototype, $s_{i}$ denotes the index of
the sub-prototype in the $i\text{-th}$ item, and $w_{i,j=s_{i}}$ denotes the
$i,j=s_{i}$-th entry of $W$. The hyperparameter $N$ determines the maximum
capacity for memory items and the hyperparameter $S$ defines the number of
sub-prototypes in each memory item. The effect of different settings of hyper-
parameters is evaluated in Section 4.

####  3.3.2 Sub-Prototype Addressing and Retrieving

In MemSPM, the memory $M$ is designed to learn the sub-prototypes to represent
the input-oriented embedding $Z$. We define the memory as a content
addressable memory [17, 10, 45, 36] that allows for direct referencing of the
content of the memory being matched. The sub-prototype is retrieved by
attention weights $W$ which are computed based on the similarity between the
sub-prototypes in the memory items and the input-oriented embedding $Z$. To
calculate the weight $w_{i,j}$, we use a softmax operation:

$w_{i,j}=\frac{\exp(d(z,m_{i,j}))}{\Sigma^{N}_{n=1}\Sigma^{S}_{s=1}\exp(d(z,m_{n,s}))}\text{,}$
(3)

where $d(\cdot,\cdot)$ denotes cosine similarity measurement. As indicated by
Eq. 1 and 3, the memory module retrieves the sub-prototype that is most
similar to $Z$ from each memory item in order to obtain the new representation
embedding $\hat{Z}$. As a consequence of utilizing the adaptive threshold
addressing technique(Section 3.3.3), only the top-$K$ can be utilized to
obtain a task-oriented embedding $\hat{Z}$, that serves to represent the
encoded embedding $Z$.

####  3.3.3 Adaptive Threshold Technique for More Efficient Memory

Limiting the amount of sub-prototypes retrieved can enhance memory utilization
and avoid negative impacts on unrelated sub-prototypes during model parameter
updates. Despite the natural reduction in the number of selected memory items,
the attention-based addressing mechanism may still lead to the combination of
small attention-weight items into the output embedding $\hat{Z}$, which has a
negative impact on the classifier and sub-prototypes in the memory. Therefore,
it is necessary to impose a mandatory quantity limit on the amount of the
relevant sub-prototypes retrieved. To address this issue, we apply an adaptive
threshold operation to restrict the amount of sub-prototypes retrieved in a
forward process.

$\hat{w}_{i,j=s_{i}}=\begin{cases}w_{i,j=s_{i}},&w_{i,j=s_{i}}>\lambda\\\
0,&\text{other}\\\ \end{cases}$ (4)

where $\hat{w}_{i,j=s_{i}}$ denotes the $i,j=s_{i}$-th entry of $\hat{w}$, the
$\lambda$ denotes the adaptive threshold:

$\lambda=\text{argmin}(topk(w))\text{.}$ (5)

Directly implementing the backward for the discontinuous function in Eq. 4 is
not an easy task. For simplicity, we use the method [17]that rewrites the
operation using the continuous ReLU activation function as:

$\hat{w}_{i,j=s_{i}}=\frac{\max(w_{i,j=s_{i}}-\lambda)\cdot
w_{i,j=s_{i}}}{\left|w_{i,j=s_{i}}-\lambda\right|+\epsilon}\text{,}$ (6)

where max$(\cdot,0)$ is commonly referred to as the ReLU activation function,
and $\epsilon$ is a small positive scalar. The prototype $\hat{Z}$ will be
obtained by $\hat{Z}=\hat{W}\cdot M$. The adaptive threshold addressing
encourages the model to represent embedding $Z$ using fewer but more relevant
sub-prototypes, leading to learning more effective features in memory and
reducing the impact on irrelevant sub-prototypes.

###  3.4 Visualization and Interpretability

We denote $f^{unfixed}_{decode}(\cdot):\hat{Z}\rightarrow\hat{X}$ to represent
the decoder. The decoder is trained to visualize what has been learned in the
memory by taking the retrieved sub-prototype as input. From an
interpretability perspective, each encoded embedding $Z$ calculates the cosine
similarity to find the top-$K$ fitting sub-prototype representation for the
given input-oriented embedding. Then, these sub-prototypes are combined to
represent the $Z$ in $\hat{Z}$. The sub-prototype in this process can be
regarded as the visual description for the input embedding $Z$. In other
words, the input image is much like the sub-classes represented by these sub-
prototypes. In this way, samples with significant intra-class differences will
be matched to different sub-prototypes, thereby distinguishing different sub-
classes. The use of a reconstruction auxiliary task can visualize the sub-
prototypes in memory to confirm whether our approach has learned intra-class
differences for the annotated category. The results of this visualization are
demonstrated in Figure 3.

###  3.5 Cycle-Consistent Alignment and Adaption

Once the sub-prototypes are mined through memory learning, the method of
cycle-consistent matching, inspired by DCC [24], is employed to align the
embedding $\hat{Z}$. The cycle-consistent matching is preferred due to it can
provide a better fit to the memory structure compared to other UniDA methods.
The other method, One-vs-All Network (OVANet), proposed by Saito et al. [40],
needs to train the memory multiple times, which can lead to significant
computational overhead. In brief, the Cycle-Consistent Alignment provides a
solution by iteratively learning a consensus set of clusters between the two
domains. The consensus clusters are identified based on the similarity of the
prototypes, which is measured using a similarity metric. The similarity metric
is calculated on the feature representations of the prototypes. For unknown
classes, we set the size $N$ of our memory during the initial phase to be
larger than the number of possible sub-classes that may be learned in the
source domain. This size is a hyperparameter that is adjusted based on the
dataset size. Redundant sub-prototypes are invoked to represent the $\hat{Z}$,
when encountering unknown classes, allowing for an improved distance
separation between unknown and known classes in the feature space.

Training Objective. The adaptation loss in our training is similar to that of
DCC, as $\mathcal{L}_{DA}$:

$\mathcal{L}_{DA}=\mathcal{L}_{ce}+\lambda_{1}\mathcal{L}_{cdd}+\lambda_{2}\mathcal{L}_{reg}\text{,}$
(7)

where the $\mathcal{L}_{ce}$ denotes the cross-entropy loss on source samples,
$\mathcal{L}_{cdd}$ denotes the domain alignment loss and $\mathcal{L}_{reg}$
denotes the regularizer. For the auxiliary reconstruction task, we add a mean-
squared-error (MSE) loss function, denoted as $\mathcal{L}_{rec}$. Thus, the
model is optimized with:

$\mathcal{L}=\mathcal{L}_{DA}+\lambda_{3}\mathcal{L}_{rec}=\mathcal{L}_{ce}+\lambda_{1}\mathcal{L}_{cdd}+\lambda_{2}\mathcal{L}_{reg}+\lambda_{3}\mathcal{L}_{rec}\text{.}$
(8)

##  4 Experiments

###  4.1 Datasets and Evaluation Metrics

We first conduct the experiments in the UniDA setting [47] where private
classes exist in both domains. Moreover, we also evaluate our approach on two
other sub-cases, namely Open-Set Domain Adaptation (OSDA) and Partial Domain
Adaptation (PDA).

Table 1: H-score ($\%$) comparison in UniDA scenario on DomainNet, VisDA and
Office-31,some results are cited from [24, 34]

Method | Pretrain | DomainNet | VisDA | Office-31
---|---|---|---|---
17 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | P2R | P2S | R2P | R2S | S2P | S2R | Avg | S2R | A2D | A2W | D2A | D2W | W2A | W2D | Avg
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
UAN [47] | ImageNet | 41.9 | 39.1 | 43.6 | 38.7 | 38.9 | 43.7 | 41.0 | 34.8 | 59.7 | 58.6 | 60.1 | 70.6 | 60.3 | 71.4 | 63.5
CMU [14] | 50.8 | 45.1 | 52.2 | 45.6 | 44.8 | 51.0 | 48.3 | 32.9 | 68.1 | 67.3 | 71.4 | 79.3 | 72.2 | 80.4 | 73.1
DCC [24] | 56.9 | 43.7 | 50.3 | 43.3 | 44.9 | 56.2 | 49.2 | 43.0 | $\mathbf{88.5}$ | 78.5 | 70.2 | 79.3 | 75.9 | 88.6 | 80.2
OVANet [40] | 56.0 | 47.1 | 51.7 | 44.9 | 47.4 | 57.2 | 50.7 | 53.1 | 85.8 | 79.4 | 80.1 | 95.4 | 84.0 | 94.3 | 86.5
UMAD [26] | 59.0 | 44.3 | 50.1 | 42.1 | 32.0 | 55.3 | 47.1 | 58.3 | 79.1 | 77.4 | 87.4 | 90.7 | $\mathbf{90.4}$ | 97.2 | 87.0
GATE [8] | 57.4 | 48.7 | 52.8 | 47.6 | 49.5 | 56.3 | 52.1 | 56.4 | 87.7 | 81.6 | 84.2 | 94.8 | 83.4 | 94.1 | 87.6
UniOT [6] | 59.3 | 47.8 | 51.8 | 46.8 | 48.3 | 58.3 | 52.0 | 57.3 | 83.7 | $\mathbf{85.3}$ | 71.4 | 91.2 | 70.9 | 90.84 | 82.2
GLC [34] | $\mathbf{63.3}$ | 50.5 | 54.9 | 50.9 | 49.6 | 61.3 | 55.1 | 73.1 | 81.5 | 84.5 | $\mathbf{89.8}$ | 90.4 | 88.4 | 92.3 | 87.8
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
GLC [34] | ViT-B/16 | 51.2 | 44.5 | 55.6 | 43.1 | 47.0 | 39.1 | 46.8 | $\mathbf{80.3}$ | 80.5 | 80.4 | 77.5 | $\mathbf{95.6}$ | 77.7 | 96.9 | 84.8
DCC [24] | 61.1 | 38.8 | 51.8 | 49.3 | 49.1 | 60.3 | 52.2 | 61.2 | 82.2 | 76.9 | 83.6 | 75.2 | 85.8 | 88.7 | 82.1
MemSPM+DCC | 62.4 | $\mathbf{52.8}$ | $\mathbf{58.5}$ | $\mathbf{53.3}$ | $\mathbf{50.4}$ | $\mathbf{62.6}$ | $\mathbf{56.7}$ | 79.3 | 88.0 | 84.6 | 88.7 | 87.6 | 87.9 | 94.3 | $\mathbf{88.5}$

Table 2: H-score ($\%$) comparison in UniDA scenario on Office-Home, some
results are cited from [24, 34]

Method | Pretrain | Office-Home
---|---|---
15 |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Ar2Cl | Ar2Pr | Ar2Rw | Cl2Ar | Cl2Pr | Cl2Rw | Pr2Ar | Pr2Cl | Pr2Rw | Rw2Ar | Rw2Cl | Rw2Pr | Avg
|  |  |  |  |  |  |  |  |  |  |  |  |  |
UAN [47] | ImageNet | 51.6 | 51.7 | 54.3 | 61.7 | 57.6 | 61.9 | 50.4 | 47.6 | 61.5 | 62.9 | 52.6 | 65.2 | 56.6
CMU [14] | 56.0 | 56.9 | 59.2 | 67.0 | 64.3 | 67.8 | 54.7 | 51.1 | 66.4 | 68.2 | 57.9 | 69.7 | 61.6
DCC [24] | 58.0 | 54.1 | 58.0 | 74.6 | 70.6 | 77.5 | 64.3 | 73.6 | 74.9 | $\mathbf{81.0}$ | 75.1 | 80.4 | 70.2
OVANet [40] | 62.8 | 75.6 | 78.6 | 70.7 | 68.8 | 75.0 | 71.3 | 58.6 | 80.5 | 76.1 | 64.1 | 78.9 | 71.8
UMAD [26] | 61.1 | 76.3 | 82.7 | 70.7 | 67.7 | 75.7 | 64.4 | 55.7 | 76.3 | 73.2 | 60.4 | 77.2 | 70.1
GATE [8] | 63.8 | 75.9 | 81.4 | 74.0 | 72.1 | 79.8 | 74.7 | 70.3 | 82.7 | 79.1 | 71.5 | 81.7 | 75.6
UniOT [6] | 67.2 | 80.5 | 86.0 | 73.5 | 77.3 | 84.3 | 75.5 | 63.3 | 86.0 | 77.8 | 65.4 | 81.9 | 76.6
GLC [34] | 64.3 | 78.2 | 89.8 | 63.1 | 81.7 | $\mathbf{89.1}$ | 77.6 | 54.2 | $\mathbf{88.9}$ | 80.7 | 54.2 | 85.9 | 75.6
|  |  |  |  |  |  |  |  |  |  |  |  |  |
GLC [34] | CLIP | 79.4 | 88.9 | $\mathbf{90.8}$ | $\mathbf{76.3}$ | 84.7 | 89.0 | $\mathbf{71.5}$ | 72.9 | 85.7 | 78.2 | 79.4 | 90.0 | 82.6
DCC [24] | 62.6 | 88.7 | 87.4 | 63.3 | 68.5 | 79.3 | 67.9 | 63.8 | 82.4 | 70.7 | 69.8 | 87.5 | 74.4
MemSPM+DCC | $\mathbf{78.1}$ | $\mathbf{90.3}$ | 90.7 | 81.9 | $\mathbf{90.5}$ | 88.3 | 79.2 | $\mathbf{77.4}$ | 87.8 | 78.8 | $\mathbf{76.2}$ | $\mathbf{91.6}$ | $\mathbf{84.2}$

Table 3: The division on label set, Common Class ($C$) / Source-Private Class
($\hat{C}_{s}$) / Target Private Class ($\hat{C}_{t}$).

Dataset | Class Split$(C/\hat{C}_{s}/\hat{C}_{t})$
---|---
4 |  |  |
| PDA | OSDA | UniDA
|  |  |
Office-31 | 10 / 21 / 0 | 10 / 0 / 11 | 10 / 10 / 11
OfficeHome | 25 / 40 / 0 | 25 / 0 / 40 | 10 / 5 / 50
VisDA | 6 / 6 / 0 | 6 / 0 / 6 | 6 / 3 / 3
DomainNet | —— | —— | 150 / 50 / 145

Datasets. Our experiments are conducted on four datasets:

Office-31 [37], which contains 4652 images from three domains (DSLR, Amazon,
and Webcam); OfficeHome [46], a more difficult dataset consisting of 15500
images across 65 categories and 4 domains (Artistic images, Clip-Art images,
Product images, and Real-World images); VisDA [33], a large-scale dataset with
a synthetic source domain of 15K images and a real-world target domain of 5K
images; and DomainNet [32], the largest domain adaptation dataset with
approximately 600,000 images. Similar to previous studies [14], we evaluate
our model on three subsets of DomainNet (Painting, Real, and Sketch).

As in previous work [24, 41, 2, 4, 47], we divide the label set into three
groups: common classes $C$, source-private classes $\hat{C}_{s}$, and target-
private classes $\hat{C}_{t}$. The separation of classes for each of the four
datasets is shown in Table 3 and is determined according to alphabetical
order.

Evaluation Metrics. We report the average results of three runs. For the PDA
scenario, we calculate the classification accuracy over all target samples.
The usual metrics adopted to evaluate OSDA are the average class accuracy over
the known classes $OS^{*}$, and the accuracy of the unknown class $UNK$. In
the OSDA and UniDA scenarios, we consider the balance between “known” and
“unknown” categories and report the H-score [1]:

$\text{H-score}=2\times\frac{OS^{*}\times UNK}{OS^{*}+UNK}\text{,}$ (9)

which is the harmonic mean of the accuracy of “known” and “unknown” samples.

Implementation Details. Our implementation is based on PyTorch [31]. We use
CLIP [12] as the backbone pretrained by CLIP [35] for the MemSPM is hard to
train with a randomly initialized encoder. The classifier consists of two
fully-connected layers, which follow the previous design [4, 47, 41, 14, 24].
The weights in the $\mathcal{L}$ are empirically set as $\lambda_{1}=0.1$,
$\lambda_{2}=3$ and $\lambda_{3}=0.5$ following DCC [24]. For a fair
comparison, we also adopt CLIP as backbone for DCC [24] and state-of-art
method GLC [34]. We use the official code of DCC [24] and GLC [34] (Links in
Appendix D).

Table 4: H-score ($\%$) comparison in OSDA scenario on Office-Home, VisDA and
Office-31, some results are cited from [24, 34]

Method | Pretrain | Office-Home | Office-31 | VisDA
---|---|---|---|---
17 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Ar2Cl | Ar2Pr | Ar2Rw | Cl2Ar | Cl2Pr | Cl2Rw | Pr2Ar | Pr2Cl | Pr2Rw | Rw2Ar | Rw2Cl | Rw2Pr | Avg | Avg | Avg
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
OSBP [41] | ImageNet | 55.1 | 65.2 | 72.9 | 64.3 | 64.7 | 70.6 | 63.2 | 53.2 | 73.9 | 66.7 | 54.5 | 72.3 | 64.7 | 83.7 | 52.3
CMU [14] | 55.0 | 57.0 | 59.0 | 59.3 | 58.2 | 60.6 | 59.2 | 51.3 | 61.2 | 61.9 | 53.5 | 55.3 | 57.6 | 65.2 | 54.2
DCC [24] | 56.1 | 67.5 | 66.7 | 49.6 | 66.5 | 64.0 | 55.8 | 53.0 | 70.5 | 61.6 | 57.2 | 71.9 | 61.7 | 72.7 | 59.6
OVANet [40] | 58.6 | 66.3 | 69.9 | 62.0 | 65.2 | 68.6 | 59.8 | 53.4 | 69.3 | 68.7 | 59.6 | 66.7 | 64.0 | 91.7 | 66.1
UMAD [26] | 59.2 | 71.8 | 76.6 | 63.5 | 69.0 | 71.9 | 62.5 | 54.6 | 72.8 | 66.5 | 57.9 | 70.7 | 66.4 | 89.8 | 66.8
GATE [8] | 63.8 | 70.5 | 75.8 | 66.4 | 67.9 | 71.7 | 67.3 | 61.5 | 76.0 | 70.4 | 61.8 | 75.1 | 69.0 | 89.5 | 70.8
ROS [6] | 60.1 | 69.3 | 76.5 | 58.9 | 65.2 | 68.6 | 60.6 | 56.3 | 74.4 | 68.8 | 60.4 | 75.7 | 66.2 | 85.9 | 66.5
GLC [34] | 65.3 | 74.2 | 79.0 | 60.4 | 71.6 | 74.7 | 63.7 | 63.2 | 75.8 | 67.1 | 64.3 | 77.8 | 69.8 | 89.0 | 72.5
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
GLC [34] | CLIP | 68.4 | 81.7 | 84.5 | $\mathbf{76.0}$ | $\mathbf{82.4}$ | $\mathbf{83.8}$ | 69.9 | 59.6 | 84.6 | $\mathbf{73.3}$ | $\mathbf{66.8}$ | 83.9 | 76.2 | 90.1 | $\mathbf{81.6}$
DCC [24] | 62.9 | 73.3 | 78.4 | 49.8 | 69.2 | 75.0 | 59.3 | 61.5 | 80.9 | 68.1 | 62.5 | 80.0 | 68.4 | 81.9 | 66.2
MemSPM+DCC | $\mathbf{69.7}$ | $\mathbf{83.2}$ | $\mathbf{85.2}$ | 72.0 | 79.2 | 81.2 | $\mathbf{72.3}$ | $\mathbf{66.7}$ | $\mathbf{85.2}$ | 72.7 | 66.0 | $\mathbf{84.5}$ | $\mathbf{76.5}$ | $\mathbf{95.6}$ | 79.7

Table 5: H-score ($\%$) comparison in PDA scenario on Office-Home, VisDA and
Office-31, some results are cited from [24, 34]

Method | Pretrain | Office-Home | Office-31 | VisDA
---|---|---|---|---
17 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Ar2Cl | Ar2Pr | Ar2Rw | Cl2Ar | Cl2Pr | Cl2Rw | Pr2Ar | Pr2Cl | Pr2Rw | Rw2Ar | Rw2Cl | Rw2Pr | Avg | Avg | Avg
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
ETN [5] | ImageNet | 59.2 | 77.0 | 79.5 | 62.9 | 65.7 | 75.0 | 68.3 | 55.4 | 84.4 | 75.7 | 57.7 | 84.5 | 70.4 | 96.7 | 59.8
BA3US [27] | 60.6 | $\mathbf{83.2}$ | $\mathbf{88.4}$ | 71.8 | 72.8 | 83.4 | 75.5 | 61.6 | 86.5 | 79.3 | 62.8 | $\mathbf{86.1}$ | $\mathbf{76.0}$ | $\mathbf{97.8}$ | 54.9
DCC [24] | 54.2 | 47.5 | 57.5 | $\mathbf{83.8}$ | 71.6 | $\mathbf{86.2}$ | 63.7 | $\mathbf{65.0}$ | 75.2 | $\mathbf{85.5}$ | $\mathbf{78.2}$ | 82.6 | 70.9 | 93.3 | 72.4
OVANet [40] | 34.1 | 54.6 | 72.1 | 42.4 | 47.3 | 55.9 | 38.2 | 26.2 | 61.7 | 56.7 | 35.8 | 68.9 | 49.5 | 74.6 | 34.3
UMAD [26] | 51.2 | 66.5 | 79.2 | 63.1 | 62.9 | 68.2 | 63.3 | 56.4 | 75.9 | 74.5 | 55.9 | 78.3 | 66.3 | 89.5 | 68.5
GATE [8] | 55.8 | 75.9 | 85.3 | 73.6 | 70.2 | 83.0 | 72.1 | 59.5 | $\mathbf{84.7}$ | 79.6 | 63.9 | 83.8 | 74.0 | 93.7 | 75.6
GLC [34] | 55.9 | 79.0 | 87.5 | 72.5 | 71.8 | 82.7 | $\mathbf{74.9}$ | 41.7 | 82.4 | 77.3 | 60.4 | 84.3 | 72.5 | 94.1 | 76.2
|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
GLC [34] | CLIP | 63.2 | 80.7 | 86.5 | 76.0 | 77.9 | 84.1 | 74.5 | 56.8 | $\mathbf{84.7}$ | 79.8 | 57.4 | 83.0 | 75.4 | 91.5 | 86.2
DCC [24] | 59.4 | 78.8 | 83.2 | 61.95 | $\mathbf{78.6}$ | 79.3 | 64.2 | 44.4 | 82.9 | 76.5 | 70.7 | 84.6 | 72.1 | 93.7 | 79.8
MemSPM+DCC | $\mathbf{64.7}$ | 81.1 | 84.5 | 74.8 | 74.7 | 77.5 | 58.7 | 60.3 | 84.2 | 70.3 | 77.2 | 85.8 | 74.5 | 94.4 | $\mathbf{87.9}$

###  4.2 Comparison with State-of-The-Arts

We compare our method with previous state-of-the-art algorithms in three sub-
cases of unsupervised domain adaptation, namely, object-specific domain
adaptation (OSDA), partial domain adaptation (PDA), and universal domain
adaptation (UniDA).

Results on UniDA. In the most challenging setting, i.e. UniDA, our MemSPM
approach achieves state-of-the-art performance. Table 7 shows the results on
DomainNet, VisDA, and Office-31, and the result of Office-Home is summarized
in Table 2. We mainly compare with GLC and DCC using ViT-B/16 as the backbone.
On Office-31, the MemSPM+DCC outperforms the previous state-of-art method GLC
by $3.7\%$ and surpasses the DCC by $6.4\%$. On visda, our method surpasses
the DCC by a huge margin of $16.1\%$. Our method also surpasses the GLC by
$9.9\%$ and the DCC by $4.5\%$ on DomainNet. On the Office-Home, we surpass
the DCC by $9.8\%$ and the GLC by $3.7\%$.

Results on OSDA and PDA. In table 4 and table 5, we present the results on
Office-Home, Office-31, and VisDA under OSDA and PDA scenarios. In the OSDA
scenario, MemSPM+DCC still achieves state-of-the-art performance.
Specifically, MemSPM+DCC obtains $95.6\%$ H-score on Office-31, with an
improvement of $5.5\%$ compared to GLC and $13.7\%$ compared to DCC. In the
PDA scenario, MemSPM still achieves comparable performance compared to methods
tailored for PDA. The MemSPM+DCC surpasses the DCC by $8.1\%$ on the VisDA.

###  4.3 Ablation Studies

Visualization with Reconstruction and tSNE We first visualize what the memory
learns from Office-Home by sampling a single sub-prototype and adapting an
auxiliary reconstruction task: $X\rightarrow\hat{X}$. We also provide the tSNE
of the $\hat{Z}$ which retrieves the most related sub-prototypes. The
visualization is shown in Figure 3. The tSNE visualization depicts the
distribution of sub-classes within each category, indicative of MemSPM’s
successful mining of sub-prototypes. The reconstruction visualization shows
what has been learned by MemSPM, demonstrating its ability to capture intra-
class diversity.

Figure 3: (a) The tSNE visualization shows the feature space of the sub-
classes belonging to each category, which demonstrates the MemSPM mining the
sub-prototypes successfully. (b) The results of different values of $S$ and
$N$. (c) The reconstruction visualization shows what has been learned in the
memory, which demonstrates the intra-class diversity has been learned by
MemSPM.

Memory-Assisted Sub-Prototype Mining (MemSPM) Impact. As shown in Tables 7, 2,
4, and 5, MemSPM+DCC outperforms DCC across UniDA, OSDA, and PDA scenarios.
MemSPM significantly enhances DCC performance with CLIP as the backbone. CLIP
is chosen because MemSPM’s memory module, with a large latent space
initialized by a random normal distribution, faces challenges in retrieving
diverse sub-prototypes early in training. CLIP’s learned global feature space
addresses this issue.

Sensitivity to Hyper-parameters. We conducted experiments on the VisDA dataset
under the UniDA setting to demonstrate the impact of hyperparameters $S$ and
$N$ on the performance of our method. The impact of $S$ is shown in Figure 3.
When $S\geq 20$, the performance achieves a comparable level. At the same
time, the performance of the model is not sensitive to the value of $N$, when
$S=30$.

Effect of CLIP-based Feature. As shown in Table 6, we have conducted
experiments to compare ViT-B/16 (pre-trained by CLIP), ViT-B/16 (pre-trained
on ImageNet), and ViT-B/16 (without pre-training). The performance of MemSPM
on Officehome using ViT-B/16 (ImageNet) is 76.7% (H-score), which is 7.5%
lower than MemSPM using ViT-B/16 (pre-trained on CLIP). Additionally, the
ViT-B/16 (without pre-training) only achieves 64.3%, which is 19.9% lower than
that using ViT-B/16 (pre-trained on CLIP).

Effect of Adaptive Threshold As shown in Table 6, to demonstrate the
effectiveness of the adaptive threshold, we find a best-performed fixed
threshold of 0.005 through experiments. It limits the memory to learn sub-
prototypes, which only achieved 73.9% (H-score) on Officehome.

Effect of Loss As shown in Table 6, We experimented with loss contributions.
$\mathcal{L}_{ce}$ for classification is essential; removing
$\mathcal{L}_{cdd}$ led to a 4.4% drop (79.8%). Optimal coefficients for
$\mathcal{L}_{ce}$ ($\lambda_{1}=0.1$) and $\mathcal{L}_{cdd}$
($\lambda_{2}=3$) achieves the best performance. The reconstruction loss
($\mathcal{L}_{rec}$) slightly improved performance, mainly for visualizing
sub-prototypes.

Table 6: Ablation Studies

Method | Pretrain | Ar2Cl | Ar2Pr | Ar2Rw | Cl2Ar | Cl2Pr | Cl2Rw | Pr2Ar | Pr2Cl | Pr2Rw | Rw2Ar | Rw2Cl | Rw2Pr | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
CLIP-Baseline | CLIP | 64.6 | 84.3 | 78.1 | 73.7 | 88.2 | 86.5 | 68.1 | 68.7 | 89.6 | 68.5 | 69.4 | 86.6 | 77.2
DCC+MemSPM | ImageNet | 57.1 | 85.0 | 88.4 | 60.8 | 61.1 | 85.2 | 83.5 | 76.1 | 87.5 | 82.7 | 77.3 | 76.4 | 76.7
DCC+MemSPM | CLIP | 78.1 | 90.3 | 90.7 | 81.9 | 90.5 | 88.3 | 79.2 | 77.4 | 87.8 | 78.8 | 76.2 | 91.6 | 84.2
DCC+MemSPM | None | 50.7 | 78.4 | 85.6 | 50.2 | 60.7 | 67.1 | 58.2 | 44.1 | 77.9 | 67.1 | 50.3 | 81.7 | 64.33
Fixed Threshold=0.005 DCC+MemSPM | CLIP | 64.6 | 86.7 | 87.4 | 63.3 | 68.5 | 79.3 | 65.9 | 65.8 | 81.4 | 70.7 | 68.8 | 85.5 | 73.9
DCC+MemSPM Without Lcdd | CLIP | 75.9 | 75.4 | 86.4 | 80.1 | 71.6 | 87.5 | 70.1 | 87.1 | 88.7 | 74.2 | 73.5 | 88.8 | 79.8

##  5 Conclusion

In this paper, we propose the Memory-Assisted Sub-Prototype Mining (MemSPM)
method, which can learn the intra-class diversity by mining the sub-prototypes
to represent the sub-classes. Compared with the previous methods, which
overlook the intra-class structure by using the one-hot label, our MemSPM can
learn the class feature from a more subdivided sub-class perspective to
improve adaptation performance. At the same time, the visualization of the
tSNE and reconstruction demonstrates the sub-prototypes have been well learned
as we expected. Our MemSPM method exhibits superior performance in most cases
compared with previous state-of-the-art methods on four benchmarks.

## References

  * [1] Silvia Bucci, Mohammad Reza Loghmani, and Tatiana Tommasi.  On the effectiveness of image rotation for open set domain adaptation.  In Proceedings of the European Conference on Computer Vision, pages 422–438, 2020.
  * [2] Pau Panareda Busto, Ahsan Iqbal, and Juergen Gall.  Open set domain adaptation for image and action recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2):413–429, 2018.
  * [3] Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Michael I Jordan.  Partial transfer learning with selective adversarial networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2724–2732, 2018.
  * [4] Zhangjie Cao, Lijia Ma, Mingsheng Long, and Jianmin Wang.  Partial adversarial domain adaptation.  In Proceedings of the European Conference on Computer Vision, pages 135–150, 2018.
  * [5] Zhangjie Cao, Kaichao You, Mingsheng Long, Jianmin Wang, and Qiang Yang.  Learning to transfer examples for partial domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2985–2994, 2019.
  * [6] Wanxing Chang, Ye Shi, Hoang Tuan, and Jingya Wang.  Unified optimal transport framework for universal domain adaptation.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 29512–29524. Curran Associates, Inc., 2022.
  * [7] Jin Chen, Xinxiao Wu, Lixin Duan, and Shenghua Gao.  Domain adversarial reinforcement learning for partial domain adaptation.  IEEE Transactions on Neural Networks and Learning Systems, 33(2):539–553, 2020.
  * [8] Liang Chen, Yihang Lou, Jianzhong He, Tao Bai, and Minghua Deng.  Geometric anchor correspondence mining with uncertainty modeling for universal domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16134–16143, 2022.
  * [9] Yanbei Chen, Xiatian Zhu, and Shaogang Gong.  Deep reconstruction-classification networks for unsupervised domain adaptation.  In Proceedings of the European Conference on Computer Vision, pages 597–613, 2016.
  * [10] Yanbei Chen, Xiatian Zhu, and Shaogang Gong.  Semi-supervised deep learning with memory.  In Proceedings of the European Conference on Computer Vision, pages 268–283, 2018.
  * [11] Zhihong Chen, Chao Chen, Zhaowei Cheng, Boyuan Jiang, Ke Fang, and Xinyu Jin.  Selective transfer with reinforced transfer network for partial domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12706–12714, 2020.
  * [12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [13] Zhekai Du, Jingjing Li, Hongzu Su, Lei Zhu, and Ke Lu.  Cross-domain gradient discrepancy minimization for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3937–3946, 2021.
  * [14] Bo Fu, Zhangjie Cao, Mingsheng Long, and Jianmin Wang.  Learning to detect open classes for universal domain adaptation.  In Proceedings of the European Conference on Computer Vision, pages 567–583, 2020.
  * [15] Yaroslav Ganin and Victor Lempitsky.  Unsupervised domain adaptation by backpropagation.  In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1180–1189. PMLR, 2015.
  * [16] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The journal of machine learning research, 17(1):2096–2030, 2016\.
  * [17] Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel.  Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1705–1714, 2019.
  * [18] Alex Graves, Greg Wayne, and Ivo Danihelka.  Neural turing machines.  arXiv preprint arXiv:1410.5401, 2014.
  * [19] Tzu Ming Harry Hsu, Wei Yu Chen, Cheng-An Hou, Yao-Hung Hubert Tsai, Yi-Ren Yeh, and Yu-Chiang Frank Wang.  Unsupervised domain adaptation with imbalanced cross-domain data.  In Proceedings of the IEEE International Conference on Computer Vision, 2015.
  * [20] JoonHo Jang, Byeonghu Na, Dong Hyeok Shin, Mingi Ji, Kyungwoo Song, and Il-Chul Moon.  Unknown-aware domain adversarial learning for open-set domain adaptation.  Advances in Neural Information Processing Systems, 35:16755–16767, 2022.
  * [21] Tarun Kalluri, Astuti Sharma, and Manmohan Chandraker.  Memsac: Memory augmented sample consistency for large scale domain adaptation.  In Proceedings of the European Conference on Computer Vision, pages 550–568, 2022.
  * [22] Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G. Hauptmann.  Contrastive adaptation network for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2019.
  * [23] Chen-Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht.  Sliced wasserstein discrepancy for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10285–10295, 2019.
  * [24] Guangrui Li, Guoliang Kang, Yi Zhu, Yunchao Wei, and Yi Yang.  Domain consensus clustering for universal domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9757–9766, 2021.
  * [25] Shuang Li, Chi Harold Liu, Qiuxia Lin, Qi Wen, Limin Su, Gao Huang, and Zhengming Ding.  Deep residual correction network for partial domain adaptation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(7):2329–2344, 2020.
  * [26] Jian Liang, Dapeng Hu, Jiashi Feng, and Ran He.  Umad: Universal model adaptation under domain and category shift.  arXiv preprint arXiv:2112.08553, 2021.
  * [27] Jian Liang, Yunbo Wang, Dapeng Hu, Ran He, and Jiashi Feng.  A balanced and uncertainty-aware approach for partial domain adaptation.  In Proceedings of the European Conference on Computer Vision, pages 123–140, 2020.
  * [28] Hong Liu, Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Qiang Yang.  Separate to adapt: Open set domain adaptation via progressive separation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2927–2936, 2019.
  * [29] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.  Conditional adversarial domain adaptation.  Advances in Neural Information Processing Systems, 31, 2018.
  * [30] Zhihe Lu, Yongxin Yang, Xiatian Zhu, Cong Liu, Yi-Zhe Song, and Tao Xiang.  Stochastic classifiers for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9111–9120, 2020.
  * [31] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [32] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.  Moment matching for multi-source domain adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
  * [33] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko.  Visda: The visual domain adaptation challenge.  arXiv preprint arXiv:1710.06924, 2017.
  * [34] Sanqing Qu, Tianpei Zou, Florian Röhrbein, Cewu Lu, Guang Chen, Dacheng Tao, and Changjun Jiang.  Upcycling models under domain and category shift.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  * [35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 2021.
  * [36] Jack Rae, Jonathan J Hunt, Ivo Danihelka, Timothy Harley, Andrew W Senior, Gregory Wayne, Alex Graves, and Timothy Lillicrap.  Scaling memory-augmented neural networks with sparse reads and writes.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  * [37] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell.  Adapting visual category models to new domains.  In Proceedings of the European Conference on Computer Vision, pages 213–226, 2010.
  * [38] Aadarsh Sahoo, Rameswar Panda, Rogerio Feris, Kate Saenko, and Abir Das.  Select, label, and mix: Learning discriminative invariant feature representations for partial domain adaptation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4210–4219, 2023.
  * [39] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko.  Universal domain adaptation through self supervision.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 16282–16292. Curran Associates, Inc., 2020.
  * [40] Kuniaki Saito and Kate Saenko.  Ovanet: One-vs-all network for universal domain adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9000–9009, 2021.
  * [41] Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.  Maximum classifier discrepancy for unsupervised domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [42] Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada.  Open set domain adaptation by backpropagation.  In Proceedings of the European Conference on Computer Vision, pages 153–168, 2018.
  * [43] Tasfia Shermin, Guojun Lu, Shyh Wei Teng, Manzur Murshed, and Ferdous Sohel.  Adversarial network with multiple classifiers for open set domain adaptation.  IEEE Transactions on Multimedia, 23:2732–2744, 2020.
  * [44] Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon.  A dirt-t approach to unsupervised domain adaptation.  arXiv preprint arXiv:1802.08735, 2018.
  * [45] Sainbayar Sukhbaatar, arthur szlam, Jason Weston, and Rob Fergus.  End-to-end memory networks.  In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
  * [46] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan.  Deep hashing network for unsupervised domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
  * [47] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.  Universal domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2720–2729, 2019.
  * [48] Chaohui Yu, Jindong Wang, Yiqiang Chen, and Meiyu Huang.  Transfer learning with dynamic adversarial adaptation network.  In 2019 IEEE International Conference on Data Mining, pages 778–786. IEEE, 2019.
  * [49] Jing Zhang, Zewei Ding, Wanqing Li, and Philip Ogunbona.  Importance weighted adversarial nets for partial domain adaptation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8156–8164, 2018.
  * [50] Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan.  Domain-symmetric networks for adversarial domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5031–5040, 2019.

Appendix

Appendix

In the supplementary material, we provide additional visualization results,
limitations, potential negative societal impacts and compute requirements of
the MemSPM. In the pursuit of reproducible research, we will make the demo and
network weights of our code available to the public.

This supplementary is organized as follows:

  * •

Section A: Notations

  * •

Section B: Limitation

  * •

Section C: Comparison Between Related Prototype Concepts

  * •

Section D: Implementation details

    * $\circ$

Baseline details

    * $\circ$

Compute requirements

  * •

Section E: Visualization Results

  * •

Section F: Details of Domain Consensus Clustering

  * •

Section G: Potential societal impact

##  Appendix A Notations

Table 7:

| Symbol | Description
---|---|---
|  |
Model | $f^{fixed}_{encode}(\cdot)$ | Fixed image encoder
$f^{unfixed}_{decode}(\cdot)$ | Unfixed reconstruction decoder
$f^{UniDA}_{class}$ | UniDA classifier
$M$ | Memory unit
$W$ | Weight vector
|  |
Space | $\mathcal{D}^{s}$ | Labeled source dataset
$\mathcal{D}^{t}$ | Unlabeled target dataset
$C$ | Common label set
$C_{s}$ | Source label set
$C_{t}$ | Target label set
$\hat{C_{s}}$ | Source private label set
$\hat{C_{t}}$ | Target private label set
|  |
Samples | $X$ | Input image
$\hat{X}$ | Reconstruction of image
$Z$ | Input-oriented embedding
$\hat{Z}$ | Task-oriented embedding
$L$ | Label of the image
$\hat{L}$ | Prediction of image
|  |
Measures | $w_{i,j}$ | Attention weight measurement between $Z$ and sub-prototype
$d(\cdot,\cdot)$ | Cosine similarity measurement
$\hat{w_{i,j}}$ | Adaptive threshold operation on $w_{i,j}$
|  |
Hyperparameters | $N$ | Number of memory items
$S$ | Number of sub-prototypes partitioned in each memory item
$D$ | Dimension of each sub-prototype
| $K$ | Top-K relevant sub-prototypes of $Z$

##  Appendix B Limitation

Training the memory unit of MemSPM is challenging when adopting the commonly
used ResNet-50 as the backbone. This is due to the memory unit’s composition
of massive randomly initialized tensors. During the early stage of training,
there is a lack of discriminability in the input-oriented embedding, which
leads to addressing only a few sub-prototypes. This decoupling of the memory
unit from the input data necessitates using a better pre-trained model
(ViT-B/16 pre-trained on CLIP) and fixing the encoder to reduce computation
requirements. Additionally, the number of sub-prototypes in one memory item
might need to be adjusted for the diversity of the category.

##  Appendix C Comparsion Between Related Prototype Concepts

The related concept of the prototype is mentioned in some previous works
[kundu2022subsidiary, liu2022psdc], there are clear differences between theirs
and our MemSPM.

First, the meaning of prototype is different between [kundu2022subsidiary] and
ours. In the [kundu2022subsidiary], the subsidiary prototype is extracted from
randomly cropped images, which means the subsidiary prototypes only represent
the low-level, morphological, and partial features of the image. These
subsidiary prototypes don’t have complete semantic knowledge, and the method
can’t learn the concept shift in the same category. Moreover, they still used
the labeled category directly for alignment and adaptation. These prototypes
can’t represent some part of the samples in one category.

In contrast, the MemSPM allows memory items to extract complete semantic
knowledge and maintain domain-invariant knowledge. To accomplish this, we use
input-oriented embedding, which involves comparing the entire image feature
with memory items. The memory can then sample a task-oriented embedding that
represents the semantic knowledge of the input-oriented embedding. Our
approach is designed to obtain a domain-invariant and semantic feature for
categories with significant domain shifts. As a result, each sub-prototype can
represent a sub-class in one category.

Second, the purpose of [liu2022psdc] is very different from our MemSPM. They
aim to learn differences among unknown classes, which is like the DCC. It
still extracts features and aligns the class across different domains directly
based on one-hot labels, and is not concerned with the concept shift and
difference in one category. However, our method can mine the sub-classes in
one category when there exist significant concept shifts, reflecting the
inherent differences among samples annotated as the same category. This helps
universal adaptation with a more fine-grained alignment or to make significant
decisions without human supervision.

##  Appendix D Implementation details

DCC. We use ViT-B/16 [12] as the backbone. The classifier is made up of two FC
layers. We use Nesterov momentum SGD to optimize the model, which has a
momentum of $0.9$ and a weight decay of $5\text{e-}4$. The learning rate
decreases by a factor of $(1+\alpha\frac{i}{N})^{-\beta}$, where $i$ and $N$
represent current and global iteration, respectively, and we set $\alpha=10$
and $\beta=0.75$. We use a batch size of 36 and the initial learning rate is
set as $1\text{e-}4$ for Office-31, and $1\text{e-}3$ for Office-Home and
DomainNet. We use the settings detailed in [24]. PyTorch [31] is used for
implementation.

GLC. We use ViT-B/16 [12] as the backbone. The SGD optimizer with a momentum
of $0.9$ is used during the target model adaptation phase of GLC [34]. The
initial learning rate is set to $1\text{e-}3$ for Office-Home and
$1\text{e-}4$ for both VisDA and DomainNet. The hyperparameter $\rho$ is fixed
at $0.75$ and $|L|$ at 4 across all datasets, while $\eta$ is set to $0.3$ for
VisDA and $1.5$ for Office-Home and DomainNet, which corresponds to the
settings detailed in [34]. PyTorch [31] is used for implementation.

Existing code used.

  * •

DCC [24]:
https://github.com/Solacex/Domain-Consensus-Clustering

  * •

GLC [34]:
https://github.com/ispc-lab/GLC

  * •

PyTorch [31]:
https://pytorch.org/

Existing datasets used.

  * •

Office-31 [37]:
https://www.cc.gatech.edu/∼judy/domainadapt

  * •

Office-Home [46]:
https://www.hemanthdv.org/officeHomeDataset.html

  * •

DomainNet [32]:
http://ai.bu.edu/M3SDA

  * •

VisDA [33]:
http://ai.bu.edu/visda-2017/

Compute Requirements. For our experiments, we used a local desktop machine
with an Intel Core i5-12490f, a single Nvidia RTX-3090 GPU, and 32GB of RAM.
When we adapt the batch-size used in DCC [24], our MemSPM only occupies 4GB of
GPU memory during training as the result of fixing the encoder.

##  Appendix E Details of Domain Consensus Clustering

Domain Consensus Clustering (DCC). They leverage Contrastive Domain
Discrepancy (CDD) to facilitate the alignment over identified common samples
in a class-aware style. They impose $L_{\text{CDD}}$ to minimize the intra-
class discrepancies and enlarge the inter-class gap. Consequently, the
enhanced discriminability, in turn, enables DCC to perform more accurate
clustering. Details of CDD are provided in:
https://openaccess.thecvf.com/content/CVPR2021/supplemental/Li_Domain_Consensus_Clustering_CVPR_2021_supplemental.pdf.

##  Appendix F Visualization

We provide more results of visualization in Figure 4 and Figure 5 to reveal
sub-prototypes stored in the memory unit, which demonstrates that our MemSPM
approach can learn the intra-class concept shift.

Figure 4:  The reconstruction visualization shows what has been learned in the
memory, which demonstrates the intra-class diversity has been learned by
MemSPM.

Figure 5:  The tSNE visualization shows the distribution of the retrieved sub-
prototypes and demonstrates that the sub-classes have been learned by MemSPM.

##  Appendix G Potential Societal Impact

Our finding of the intra-class concept shift may influence future work on
domain adaption or other tasks. They can optimize the construction and
refinement of the feature space by considering the intra-class distinction.
The MemSPM also provides a method that can be used to demonstrate the
interpretability of the model for further deployment. However, the utilization
of the MemSPM method for illegal purposes may be facilitated by its increased
availability to organizations or individuals. The MemSPM method may be
susceptible to adversarial attacks as all contemporary deep learning systems.
Although we demonstrate increased performance and interpretability compared to
the state-of-the-art methods, negative transfer is still possible in extreme
cases of domain shift or category shift. Therefore, our technique should not
be employed in critical applications or to make significant decisions without
human supervision.


Modified Newton’s gravity in Finsler Space as a possible alternative to dark
matter hypothesis
Zhe Chang<EMAIL_ADDRESS>and Xin Li<EMAIL_ADDRESS>
Institute of High Energy Physics, Chinese Academy of Sciences
P. O. Box 918(4), 100049 Beijing, China

###### Abstract

A modified Newton’s gravity is obtained as the weak field approximation of the
Einstein’s equation in Finsler space. It is found that a specified Finsler
structure makes the modified Newton’s gravity equivalent to the modified
Newtonian dynamics (MOND). In the framework of Finsler geometry, the flat
rotation curves of spiral galaxies can be deduced naturally without invoking
dark matter.

PACS numbers: 02.40.-k, 04.25.Nx, 95.35.+d

There are a great variety of observations which show that the rotational
velocity curves of all spiral galaxies tend to some constant values[1]. These
include the Oort discrepancy in the disk of the Milky Way[2], the velocity
dispersions of dwarf Spheroidal galaxies[3], and the flat rotation curves of
spiral galaxies[4]. These facts violate sharply the prediction of Newtonian
dynamics or Newton’s gravity.

The most widely adopted way to resolve these difficulties is the dark matter
hypothesis. It is assumed that all visible stars are surrounded by massive
nonluminous matters. Though it explains the flat rotation curves of spiral
galaxies, the hypothesis has its own weakness. No theory predicts these
matters, and they behave in such ad hoc way. There are a lot of possible
candidates of dark matter (such as axion, neutrino et al), but none of them
satisfactory. Up to now, all of them either undetected or excluded by
observation.

Because of these troubles induced by dark matter, some models have been built
for alternative of the dark matter hypothesis. Their main ideas are to assume
that the Newtonian gravity or Newton’s dynamics is invalid in galactic scale.
In particular, two models explain well the flat rotational curves successfully
without invoking dark matter. One is higher-order gravitational theory[5]. The
gravitational potential was supposed[6] of the Yukawa form,

$\displaystyle\varphi(r)=-\frac{GM}{r(1+\alpha)}\left(1+\alpha
e^{r/r_{0}}\right).$ (1)

Another is the famous MOND[7]. It assumed that the Newtonian dynamics does not
hold in galactic scale. The particular form of MOND is given as

$\begin{array}[]{l}m\mu\left(\displaystyle\frac{a}{a_{0}}\right)\mathbf{a}=\mathbf{F},\\\\[11.38092pt]
\displaystyle\lim_{x\gg 1}\mu(x)=1,~{}~{}~{}\lim_{x\ll 1}\mu(x)=x,\end{array}$
(2)

where $a_{0}$ is at the order of $10^{-8}$ cm/s2. At beginning, as a
phenomenological model, MOND explains well the flat rotation curves with a
simple formula and a new parameter. In particular, it deduce naturally a well-
known global scaling relation for spiral galaxies, the Tully-Fisher
relation[8]. By introducing several scalar, vector and tensor fields,
Bekenstein[9] rewrote the MOND into a covariant formulation. He showed that
the MOND satisfies all four classical tests on Einstein’s general relativity
in Solar system.

These models seem appealing in theoretical interest and fit well the empirical
data of the flat rotation curves. However, it has been pointed out that
solving dark matter problem by means of metric theories is too difficult. The
Yukawa term(1) is mediated by a spin 1 vector particle[6]. Such a
contribution[10] may be proportional to baryonic charge. The baryonic charge
varies from one body to another, this violates the weak equivalence principle.
MOND, as a very successful phenomenological model, still face problems yet.
Reproducing the MOND force law[11] requires any completely stable, metric-
based theory of gravity to become conformally invariant in the weak field
limit, and the prospects for a formulation with a very weak instability.

In this Letter, we present another possible alternative to the dark matter
hypothesis. The major property of the flat rotational curves is the velocity
of a particle in a circular orbit around a finite spiral galaxy becomes
independent of the radius of the orbit at large radii. This empirical fact
makes us consider the Finsler geometry. In Finsler geometry, the intrinsic
curve is not only the function of position but also the function of velocity.
And in the framework of Finsler geometry, the four-velocity vector is treated
as independent variable[12]. The explicit DISIM${}_{b}(2)$ invariant
Finslerian line element and the respective Lie algebra was first proposed
thirty years ago[13, 14, 15]. Finsler geometry is a natural and fundamental
generalization of Riemann geometry.

The gravity in Finsler space has been investigated for a long time[16, 17, 18,
19]. In this Letter, we begin with the field equation in Berwald space-a
special space in Finsler space. The gravitational field equation in Berwald-
Finsler space has been written down explicitly[20],

$\displaystyle\left[Ric_{\mu\nu}-\frac{1}{2}g_{\mu\nu}S\right]+\left\\{\frac{1}{2}B^{~{}\alpha}_{\alpha~{}\mu\nu}+B^{~{}\alpha}_{\mu~{}\nu\alpha}\right\\}=8\pi
GT_{\mu\nu}.$ (3)

We will study the weak field approximation of the field equation (3). Results
obtained should be compared with Newton’s gravity. We wish the modified
Newton’s gravity got as the weak field approximation of the field equation (3)
in Finsler space describes the odd behavior of the rotation curves of spiral
galaxies.

Before dealing with the field equation, we introduce some basic notation of
Finsler geometry[21]. Denote by $T_{x}M$ the tangent space at $x\in M$, and by
$TM$ the tangent bundle of $M$. Each element of $TM$ has the form $(x,y)$,
where $x\in M$ and $y\in T_{x}M$. The natural projection $\pi:TM\rightarrow M$
is given by $\pi(x,y)\equiv x$. A Finsler structure of $M$ is a function

$\displaystyle F:TM\rightarrow[0,\infty)$

with the following properties:
(i) Regularity: F is $C^{\infty}$ on the entire slit tangent bundle
$TM\backslash 0$.
(ii) Positive homogeneity : $F(x,\lambda y)=\lambda F(x,y)$ for all
$\lambda>0$.
(iii) Strong convexity: The $n\times n$ Hessian matrix

$\displaystyle g_{\mu\nu}\equiv\frac{\partial}{\partial
y^{\mu}}\frac{\partial}{\partial y^{\nu}}\left(\frac{1}{2}F^{2}\right)$

is positive-definite at every point of $TM\backslash 0$.

Finsler geometry has its genesis in integrals of the form

$\displaystyle\int^{r}_{s}F\left(x^{1},\cdots,x^{n};\frac{dx^{1}}{dt},\cdots,\frac{dx^{n}}{dt}\right)dt.$
(4)

Throughout the Letter, the lowering and raising of indices are carried out by
the fundamental tensor $g_{\mu\nu}$ defined above, and its inverse
$g^{\mu\nu}$.

In Finsler manifold, there exist a unique linear connection - the Chern
connection[22]. It is torsion freeness and metric-compatibility,

$\displaystyle\Gamma^{\alpha}_{\mu\nu}=\gamma^{\alpha}_{\mu\nu}-g^{\alpha\lambda}\left(A_{\lambda\mu\beta}\frac{N^{\beta}_{\nu}}{F}-A_{\mu\nu\beta}\frac{N^{\beta}_{\lambda}}{F}+A_{\nu\lambda\beta}\frac{N^{\beta}_{\mu}}{F}\right),$
(5)

where $\gamma^{\alpha}_{\mu\nu}$ is the formal Christoffel symbols of the
second kind with the same form of Riemannian connection, $N^{\mu}_{\nu}$ is
defined as
$N^{\mu}_{\nu}\equiv\gamma^{\mu}_{\nu\alpha}y^{\alpha}-A^{\mu}_{\nu\lambda}\gamma^{\lambda}_{\alpha\beta}y^{\alpha}y^{\beta}$
and $A_{\lambda\mu\nu}\equiv\frac{\partial}{\partial
y^{\lambda}}\frac{\partial}{\partial y^{\mu}}\frac{\partial}{\partial
y^{\nu}}\frac{F}{4}(F^{2})$ is the Cartan tensor (regarded as a measurement of
deviation from Riemannian Manifold). The curvature of Berwald - Finsler space
is given as

$\displaystyle R^{~{}\lambda}_{\kappa~{}\mu\nu}$ $\displaystyle=$
$\displaystyle\frac{\partial\Gamma^{\lambda}_{\kappa\nu}}{\partial
x^{\mu}}-\frac{\partial\Gamma^{\lambda}_{\kappa\mu}}{\partial
x^{\nu}}+\Gamma^{\lambda}_{\alpha\mu}\Gamma^{\alpha}_{\kappa\nu}-\Gamma^{\lambda}_{\alpha\nu}\Gamma^{\alpha}_{\kappa\mu}.$
(6)

The Ricci tensor on Finsler manifold was first introduced by Akbar-Zadeh[23].
In Berwald-Finsler space, it reduces to

$\displaystyle
Ric_{\mu\nu}=\frac{1}{2}(R^{~{}\alpha}_{\mu~{}\alpha\nu}+R^{~{}\alpha}_{\nu~{}\alpha\mu}).$
(7)

It is manifestly symmetric and covariant. Apparently the Ricci tensor will
reduce to the Riemann-Ricci tensor if the Cartan tensor vanish identically.
The trace of the Ricci tensor gives the scalar curvature $S\equiv
g^{\mu\nu}Ric_{\mu\nu}$. We starts from the second Bianchi identities on
Berwald-Finsler space

$\displaystyle
R^{~{}\alpha}_{\mu~{}\lambda\nu|\beta}+R^{~{}\alpha}_{\mu~{}\nu\beta|\lambda}+R^{~{}\alpha}_{\mu~{}\beta\lambda|\nu}=0,$
(8)

where the $|$ means the covariant derivative. The metric-compatibility

$\displaystyle
g_{\mu\nu|\alpha}=0~{}~{}~{}~{}\mathrm{and}~{}~{}~{}~{}g^{\mu\nu}_{~{}~{}|\alpha}=0,$
(9)

and contraction of (8) with $g^{\mu\beta}$ gives that

$\displaystyle
R^{\mu\alpha}_{~{}~{}\lambda\nu|\mu}+R^{\mu\alpha}_{~{}\nu\mu|\lambda}+R^{\mu\alpha}_{~{}\mu\lambda|\nu}=0.$
(10)

Lowering the index $\alpha$ and contracting with $g^{\alpha\lambda}$, we
obtain

$\displaystyle\left[Ric_{\mu\nu}-\frac{1}{2}g_{\mu\nu}S\right]_{|\mu}+\left\\{\frac{1}{2}B^{~{}\alpha}_{\alpha~{}\mu\nu}+B^{~{}\alpha}_{\mu~{}\nu\alpha}\right\\}_{|\mu}=0,$
(11)

where

$\displaystyle
B_{\mu\nu\alpha\beta}=-A_{\mu\nu\lambda}R^{~{}\lambda}_{\theta~{}\alpha\beta}y^{\theta}/F.$
(12)

Thus, the counterpart of the Einstein’s field equation on Berwald - Finsler
space takes the form

$\displaystyle\left[Ric_{\mu\nu}-\frac{1}{2}g_{\mu\nu}S\right]+\left\\{\frac{1}{2}B^{~{}\alpha}_{\alpha~{}\mu\nu}+B^{~{}\alpha}_{\mu~{}\nu\alpha}\right\\}=8\pi
GT_{\mu\nu}.$ (13)

To get a modified Newton’s gravity, we consider a particle moving slowly in a
weak stationary gravitational field[24]. We suppose that the metric is close
to the locally Minkowskian metric,

$\begin{array}[]{l}g_{\mu\nu}(x,y)=\eta_{\mu\nu}(y)+h_{\mu\nu}(x,y),\\\\[5.69046pt]
\eta_{\mu\nu}(y)=f(y)*{\rm diag}\\{1,-1,-1,-1\\}~{},\end{array}$ (14)

where $|h_{\mu\nu}|\ll 1$.

The field is stationary and all time derivatives of $g_{\mu\nu}$ vanish. The
Cartan tensor is symmetric in its three indices. To first order in
$h_{\mu\nu}$, we have

$\displaystyle\Gamma^{i}_{00}=\frac{1}{2f}\frac{\partial h_{00}}{\partial
x^{i}}-\frac{1}{4f^{2}}\frac{\partial f}{\partial y^{0}}y^{0}\frac{\partial
h_{00}}{\partial x^{i}},$ (15)

and

$\displaystyle Ric_{00}=\frac{\partial\Gamma^{i}_{00}}{\partial x^{i}}.$ (16)

All particles here are moving slowly. The velocity $y^{0}$ equal to the speed
of light approximately and can be regarded as constant. Then, the connection
$\Gamma^{i}_{00}$ reduces as

$\displaystyle\Gamma^{i}_{00}=\frac{1}{2f}\frac{\partial h_{00}}{\partial
x^{i}}.$ (17)

From the definition of $B_{\mu\nu\alpha\beta}$, to first order in
$h_{\mu\nu}$, we know that all components of $B$ vanish. By making use of the
non-relativistic perfect fluid approximation, we obtain the non vanished
component of energy-momentum tensor $T_{00}=\rho g_{00}$. Here $\rho$ is the
proper energy density. After the above manipulations, the velocity four-vector
reduces to the normal velocity $v$, which is defined in the manifold $M$ and
function of distance. In the above approximations, the field equation(3)
reduces to

$\displaystyle\frac{\partial}{\partial
x_{i}}\left(\frac{1}{2f(v)}\frac{\partial h_{00}}{\partial x^{i}}\right)=4\pi
G\rho f(v).$ (18)

The geodesic equation in Finsler space is given as[21]

$\displaystyle\frac{d^{2}x^{\lambda}}{d\tau^{2}}+\gamma^{\lambda}_{\mu\nu}\frac{dx^{\mu}}{d\tau}\frac{dx^{\nu}}{d\tau}=0.$
(19)

Though the form of the geodesic equation is the same with the one in Riemann
geometry, but one should notice that the connection
$\gamma^{\lambda}_{\mu\nu}$ here depends both on coordinates and velocities.
We will show the physical meaning of the velocity dependence in an indirect
way by solving the geodesic equation in weak field approximation. In the
approximation of moving slowly and weak field, the geodesic equation(19)
reduces to

$\displaystyle\frac{d^{2}t}{d\tau^{2}}$ $\displaystyle=$ $\displaystyle 0,$
$\displaystyle\frac{d^{2}x^{i}}{d\tau^{2}}$ $\displaystyle=$
$\displaystyle-\frac{1}{2f}\frac{\partial h_{00}}{\partial
x^{i}}\left(\frac{dt}{d\tau}\right)^{2}.$ (20)

The solution of first equation in (S0.Ex3) is $dt/d\tau=const.$. Dividing the
second equation in (S0.Ex3) by $(dt/d\tau)^{2}$, we obtain

$\displaystyle\frac{d^{2}x^{i}}{dt^{2}}=-\frac{1}{2f}\frac{\partial
h_{00}}{\partial x^{i}}\equiv-\frac{\partial\varphi}{\partial x^{i}},$ (21)

where $\varphi$ is the gravitational potential. Substituting the relation (21)
into equation (18), we get

$\displaystyle\nabla^{2}\varphi=4\pi G\rho f(v).$ (22)

The velocity $v$ is a function of coordinate. In spherical coordinate, we can
rewrite $f(v)$ into $\rho_{geo}(r)$. The integration of (22) gives

$\displaystyle\nabla\varphi(r)=\frac{G}{r^{2}}\int\rho(r)\rho_{geo}(r)dV=\frac{G\mathcal{M}}{r^{2}}.$
(23)

One can see clearly that the effective mass $\mathcal{M}$ is different from
the baryonic mass. It is enlarged by the factor $\rho_{geo}(r)$ due to Finsler
geometrical effect.

We suppose the energy density of baryons is constant for convenience. Denotes
$M$ the mass of visible matter.

We would like limit the metric (14) to be the form

$\begin{array}[]{l}\displaystyle f(\xi)=\frac{\frac{5}{6}\frac{3a_{0}}{4\pi
G\rho\xi}+1}{\sqrt{\frac{3a_{0}}{4\pi G\rho\xi}+1}}~{},\\\\[28.45274pt]
v=\left(\frac{4}{3}\pi Ga_{0}\rho\xi^{3}+\left(\frac{4}{3}\pi
G\rho\right)^{2}\xi^{4}\right)^{1/4}~{}.\end{array}$ (24)

Here $a_{0}$ is the deformation parameter of Finsler geometry. It is the
measurement of deviation from Riemann geometry. The deformation of Finsler
space may have cosmological significance. One wishes naturally the deformation
parameter relates with the cosmological constant $\Lambda$. In fact, Milgrom
observed[7] that $2\pi a_{0}\approx c(\sqrt{\Lambda/3})$. Eqs (22) and (23)
give the geometrical factor of the density of baryons,

$\displaystyle\rho_{geo}(r)=\frac{\frac{5}{6}+\frac{GM}{r^{2}a_{0}}}{\sqrt{(\frac{GM}{r^{2}a_{0}})^{2}+\frac{GM}{r^{2}a_{0}}}}.$
(25)

In the zero limit of the deformation parameter, familiar results on Riemann
geometry are recovered. Thus, the acceleration $a$ is expressed as

$\displaystyle
a=\nabla\varphi(r)=\frac{GM}{r^{2}}\sqrt{\frac{r^{2}+GM/a_{0}}{GM/a_{0}}}.$
(26)

We can rewrite it in a compact form,

$\displaystyle a=\frac{GM}{r^{2}}\nu\left(\frac{GM}{r^{2}a_{0}}\right),$ (27)

where

$\displaystyle\nu(x)=\sqrt{\frac{1+x}{x}}\approx\left\\{1~{}~{}~{}~{}~{}~{}~{}{\rm
for}~{}~{}x\gg 1\atop x^{-1/2}~{}~{}{\rm for}~{}~{}x\ll 1\right..$ (28)

It is obvious that the above formula is just the MOND [25]. The MOND reduces
to Newton’s gravity when $x\gg 1$. Meanwhile, in our model the metric
$\eta_{\mu\nu}$ returns to Minkowskian metric.

Persic, Salucci and Stel analyzed about 1100 rotational curves of spiral
galaxies[26]. They gave a universal formula for the rotation curves, called
Universal Rotation Curves (URC),

$\displaystyle
V_{URC}(r)=V(r_{opt})\bigg{[}\left(0.72+0.44\log\frac{L}{L_{\ast}}\right)\frac{1.97x^{1.22}}{(x^{2}+0.78^{2})^{1.43}}$
$\displaystyle+1.6e^{-0.4(L/L_{\ast})}\frac{x^{2}}{x^{2}+1.5^{2}(L/L_{\ast})^{0.4}}\bigg{]}^{1/2}{\rm
km/s},$ (29)

where $r_{opt}$ is the radius encircling 83% of the integrated light and
$x=r/r_{opt}$. The luminous matter in spiral galaxies is composed by
spheroidal bugle and an extended thin exponential disk. The spheroidal bugle
has density distribution of the form
$I(r)=I_{0}e^{-7.67(r/r_{e})^{1/4}}$(where $r_{e}$ is the half-light
radius)[27]. The thin exponential disk has density distribution of the form
$I(r)=I_{0}e^{-r\alpha}$($\alpha^{-1}$ is the disk scale-length)[28].
Phenomenological similarities between the URC and MOND have been investigated
in detail by looking for properties predicted in one framework that are also
reproducible in the other one[29]. We wish a general Finslerian metric be
found to reproduced the URC naturally. In fact, many works about dark matter
have been done, and the density profile of dark matter halos surrounding
galaxies have been given. The Finsler geometrical factor is determined by

$\displaystyle\rho_{geo}=1+\rho_{DM}/\rho.$ (30)

Acknowledgements

We would like to thank T. Chen, P. Wang, N. Wu and Y. Yu for useful
discussion. The work was supported by the NSF of China under Grant NO.
10575106.

## References

  * [1] V. T. Trimble, Ann. Rev. Astron. Astrophys. 25, 425 (1987).
  * [2] J. N. Bahcall, C. Flynn, and A. Gould, Astrophys. J. 389, 234 (1992).
  * [3] S. S. Vogt, M. Mateo, E. W. Olszewski, and M. J. Keane, Astron. J. 109, 151 (1995).
  * [4] V. C. Rubin, W. K. Ford, and N. Thonnard, Astrophys. J. 238, 471 (1980).
  * [5] C. Xu, G. F. R. Ellis and X. Wu et al. S. Afr. J. Phys. 15, 5 (1992).
  * [6] R. H. Sanders, Astron. Astrophys. 136, L21 (1984); 154, 135 (1984).
  * [7] M. Milgrom, Astrophys. J. 270, 365 (1983).
  * [8] R. B. Tully and J. R. Fisher, Astr. Ap. 54, 661 (1977).
  * [9] J. D. Bekenstein, Phys. Rev. D 70, 083509 (2004).
  * [10] V. V. Zhytnikov and J. M. Nester, Phys. Rev. Lett 73, 2950 (1994).
  * [11] M. E. Soussa and R. P. Woodard, arXiv: astro-ph/037358v1.
  * [12] Z. Chang and X. Li, Phys. Lett. B. 663, 103 (2008).
  * [13] G. Yu. Bogoslovsky, Nuovo Cimento B, 40, 99 (1977).
  * [14] G. Yu. Bogoslovsky, arXiv: gr-qc/0706.2621.
  * [15] G. W. Gibbons, J. Gomis, and C. N. Pope, arXiv: gr-qc/0707.2174.
  * [16] Y. Takano, Lett. Nuovo Cimento, 10, 747 (1974).
  * [17] S. Ikeda, Ann. der Phys., 44, 558 (1987).
  * [18] R. Tavakol, N. van den Bergh, Phys. Lett. A, 112, 23 (1985).
  * [19] G. Yu. Bogoslovsky, Phys. Part. Nucl., 24, 354 (1993).
  * [20] X. Li and Z. Chang, arXiv: gr-qc/0711.1934.
  * [21] D. Bao, S. S. Chern and Z. Shen, An Introduction to Riemann–Finsler Geometry, Graduate Texts in Mathmatics 200, Springer, New York, 2000.
  * [22] S. S. Chern, Sci. Rep. Nat. Tsing Hua Univ. Ser. A 5, 95 (1948); or Selected Papers, vol. II, 194, Springer 1989.
  * [23] H. Akbar-Zadeh, Acad. Roy. Belg. Bull. Cl. Sci. (5) 74, 281 (1988).
  * [24] S. Weinberg, Gravitation and Cosmology: Principles and Applications of the General Theory of Relativity, John Wiley & Sons New York, 1972.
  * [25] M. Milgrom, arXiv: astro-ph/0801.3133v2.
  * [26] M. Persic, P. Salucci and F. Stel, Mon. Not. R. astr. Soc., 281, 27 (1996).
  * [27] G. de Vaucouleurs, Ann. d’Astroph., 11, 247 (1948).
  * [28] K. C. Freeman, Astrophys. J. 160, 811 (1970).
  * [29] G. Gentile, arXiv: astro-ph/0805.1731v1.