In visual domain adaptation (DA), separating the domain-specific characteristics from the domain-invariant representations is an ill-posed problem. Existing methods apply different kinds of priors or directly minimize the domain discrepancy to address this problem, which lack flexibility in handling real-world situations. Another research pipeline expresses the domain-specific information as a gradual transferring process, which tends to be suboptimal in accurately removing the domain-specific properties. In this paper, we address the modeling of domain-invariant and domain-specific information from the heuristic search perspective. We identify the characteristics in the existing representations that lead to larger domain discrepancy as the heuristic representations. With the guidance of heuristic representations, we formulate a principled framework of Heuristic Domain Adaptation (HDA) with well-founded theoretical guarantees. To perform HDA, the cosine similarity scores and independence measurements between domain-invariant and domain-specific representations are cast into the constraints at the initial and final states during the learning procedure. Similar to the final condition of heuristic search, we further derive a constraint enforcing the final range of heuristic network output to be small. Accordingly, we propose Heuristic Domain Adaptation Network (HDAN), which explicitly learns the domain-invariant and domain-specific representations with the above mentioned constraints. Extensive experiments show that HDAN has exceeded state-of-the-art on unsupervised DA, multi-source DA and semi-supervised DA. The code is available at <https://github.com/cuishuhao/HDA>.

§ INTRODUCTION

In visual domain adaptation scenarios, the domain-invariant (a.k.a. transferable) representations [36], are considered to be the information bottleneck to enhance the generalizability of a model transferred from labeled source domain to unlabeled or weakly-labeled target domain. Nevertheless, without sufficient knowledge, separating domain-invariant information from the domain-specific information that is harmful for domain adaptation, is substantially an ill-posed problem with infinite solutions.

To derive feasible solutions, a major way is to impose different prior knowledge on the structure of the mapping functions. For example, the domain-invariant feature is regarded as the information shared among domains, and obtained by averaging the domain-specific metrics [37] or by a joint neural network [8] for multiple learning tasks. Regularization with well-established statistical explanation, such as $F$-norm [15], structured $L_1$-norm [3, 4] or orthogonality [2] constraints on linear or kernel mapping functions [14], is imposed on either the domain-specific or domain-invariant part to alleviate the ill-posedness. However, the learning of domain-invariant representations may still suffer from inappropriate prior knowledge, which requires both a comprehensive understanding of domain data and the flexibility in handling real-world domain adaptation problems.

In deep neural networks, the learning of domain-invariant features are directly guided by knowledge of prior distribution using diverse metrics on multiple layers [17, 18, 32, 55, 59], or by confusing the domain classifier in an adversarial manner [18, 25, 28, 33]. Some other prior-based approaches are focused on additional properties along with the training procedure, such as the transferability [10, 54] and discriminability [10, 11, 60]. Nevertheless, by disregarding the explicit modeling of domain-specific characteristics, it can not be guaranteed that the domain-invariant representations contain as few domain-specific properties as possible. Other research pipeline expresses the domain-specific information from source to target as a gradual transferring process [12], undertaken on a geodesic domain flow along the data manifolds [19, 20, 23]. In these methods, the geodesic transfer properties are enforced by reconstructing input images [6, 20], which tends to be less accurate in removing the domain-specific properties from domain-invariant representations.

In this paper, we propose a principled framework of Heuristic Domain Adaptation (HDA), with a lower error bound derived by theoretical analysis. The information in the current domain-invariant representations that leads to larger domain discrepancy is identified as the domain-specific counterpart. We assume that domain-specific representations could be more easily obtained compared with domain-invariant parts. Similar to the estimated distance in heuristic search, the explicit construction of domain-specific representations could provide heuristics on learning the domain-invariant representations. Furthermore, the distance to the destination could be approximated by the range of domain-specific representations. Similar to the admissible constraint in heuristic search, the range is expected to be lower than the ideal range of domain-specific representations during the training. In what follows, we sometimes refer to the domain-specific representation as the heuristic representation.

To perform HDA, we derive three heuristic constraints, i.e., limited similarity, independence and termination. The analysis on cosine similarity scores between domain-invariant and domain-specific representations results in the constraint of limited similarity between their initial representations. The independence between the domain-invariant and domain-specific representations could be enforced by lessening the non-gaussianity discrepancy between the two representations. We also analyze the termination condition of heuristic search, and derive the constraint of reducing the range of heuristic representations to near zero at the end. By distinguishing the two counterparts more comprehensively, a better construction of domain-invariant representation can be achieved.

Accordingly, we propose Heuristic Domain Adaptation Network (HDAN), which is composed of the fundament network and heuristic network. The heuristic network is constructed by a series of subnetworks to enhance the construction of domain-specific properties. We devise HDAN for three typical domain adaptation tasks, i.e., unsupervised DA, semi-supervised DA and multi-source DA. Experiments show that HDAN achieves state-of-the-art results on the three tasks. Our contributions could be summarized as follows:

* We propose HDA, a principled framework leveraging domain-specific representations as heuristics to obtain domain-invariant representations.

* We develop three heuristic constraints to distinguishing the two counterparts more comprehensively, and a better construction of domain-invariant representation can be achieved.

* We design HDAN to perform HDA, and achieve state-of-the-art on three DA tasks.

§ RELATED WORK

Visual domain adaptation [42] has achieved increasing research interests in the past few years. Recently, domain adaptation methods focus on different types of domain adaptation tasks, such as unsupervised DA [18, 54, 11], multi-source DA [53, 39], semi-supervised DA [43], partial DA [7] and unsupervised open domain recognition [58]. In this work, we aim to construct a general framework that is suitable for a wide range of domain adaptation tasks.

Deep domain adaptation methods incorporate various learning criteria on deep networks with two main technologies for distribution matching, i.e., moment alignment and adversarial training. Moment alignment methods [23, 32, 55] are designed with classical metrics, such as maximum mean discrepancy [32, 35], second-order statistics [48, 59] or other distance metrics calculated on task-specific representations [26]. Inspired by the Generative adversarial networks (GAN) [21], adversarial learning has been successfully explored on various tasks including domain adaptation. Domain adversarial neural network (DANN) [18] confuses the domain classifier by the gradient reversal layer to reduce the domain shift. Other methods, such as ADDA [50], CyCADA [25], MCD [45], CDAN [33] and GVB [12], further put forward more advanced adversarial domain adaptation frameworks.

Beyond distribution alignment, other types of prior knowledge [8, 4] can also be applied. The prior knowledge is regarded as heuristic information to enhance the reinforcement learning that is used to address the transfer learning problem [9]. In domain adaptation, the transferability is enhanced by minimizing the $L_1$-norm of domain-specific representation in [54, 10]. Meanwhile, the discriminability [10, 60, 11] is modeled to avoid spurious predictions. In our study, the prior knowledge is the assumption that domain-specific properties could be more easily obtained compared to domain-invariant properties. Thus similar to the estimated distance in heuristic search, we explicitly construct domain-specific representations towards more accurate domain adaptation.

In existing methods, DSN [6] and DLOW [20] reconstruct the input images to enhance the domain-specific counterpart. However, the image reconstruction demands large domain properties, resulting in great difficulties in removing the domain characteristics. GVB [12] explicitly reduces the domain discrepancy, but it lacks accurate construction of domain-specific parts. In comparison, we model the domain-specific parts from the overall representations by checking the conditions of the similarity, independence and termination in the learning process, similar to the procedure of heuristic search.

§ HEURISTIC DOMAIN ADAPTATION

In classic path search problems, the goal is to obtain the minimum-cost path. As a typical solution of path search, heuristic search estimates the distance to destination with lower time consumption. In a standard heuristic search process, e.g., $A^*$ search, the $n$th node is given a cost ${f}(n)$. ${f}(n)$ is composed of two elements, i.e., ${g}(n)$ as the cost from the start to the $n$th node, and ${h}(n)$ as the estimated cost from the $n$th node to the destination, as follows:

\begin{equation}
\begin{split}{
{f}(n)={g}(n) + {h}(n). \\
\label{search}
\end{equation}

In search problems, $A^*$ search is proved to be optimal [40, 38], if ${h}(n)$ is admissible. The admissible constraint means the value of ${h}(n)$ is smaller than the ideal estimation ${h}^{*}(n)$, i.e., the real distance from the $n$th node to the destination should be restricted as follows:

\begin{equation}
\begin{split}{
{h}(n) \le h^{*}(n). \\
\label{optimals}
\end{equation}

In domain adaptation, it is difficult to directly obtain domain-invariant representations due to the large discrepancy between the training and testing data. Thus towards more comprehensive reduction of domain discrepancy, we resort to more flexible prior knowledge to mitigate the transfer difficulties.

In a certain domain, the difficulty of constructing domain-invariant representations is more than that of constructing domain-specific representations.

As widely accepted, images from the same domain demonstrate similar domain-specific properties. For example, images taken from foggy situation constitute the domain with vast expanse of whiteness [46]. Therefore, the domain properties could be easily captured, as validated in Figure <ref>.

The assumption is regarded as the general prior knowledge in domain adaptation. Accordingly, we construct a framework which is composed of a fundament network $F$, and a heuristic network $H$, as shown in Figure <ref>. The heuristic network aims to explicitly model the discrepancy from the current representations to the ideal domain-invariant representations, denoted as heuristic representations. Similar to Eqn. <ref>, with the input $x$, subtracting heuristic representations $H(x)$ from fundament representations $F(x)$ results in the transferable representations $G(x)$, as follows:

\begin{equation}
\begin{split}{
G(x) = F(x)- {H}(x). \\
\label{network}
\end{equation}

According to Assumption <ref>, ${H}(x)$ could be more easily modeled than ${G}(x)$. Thus ${H}(x)$ could provide guidance to the construction of $G(x)$. Meanwhile, considering the admissible constraint in heuristic search in Eqn. <ref>, the heuristic representations could be similarly restrained as follows:

\begin{equation}
\begin{split}{
|{H}(x)| \le |{H}^*(x)|= |{F}(x) - {F}^*(x)|, \\
\label{optimal}
\end{equation}

where ${H}^*(x)$ and ${F}^*(x)$ are the ideal heuristic and fundament representations. The range of ${H}(x)$ should not be larger than the ideal value ${H}^*(x)$. This indicates that ${H}(x)$ should not over-estimate the distance ${F}(x)$ to ${F}^*(x)$. Since large distance is always accompanied by redundant domain properties, leading to the confusion between $G(x)$ and $H(x)$. The construction of ${H}(x)$ and ${F}(x)$ formulates the principled framework of Heuristic Domain Adaptation. More discussions between heuristic search and HDA could be found in Section 1 in Supplementary.

To show the intrinsic mechanism of HDA, we follow the theory of [5]. Consider the source and target distribution $S$ and $T$ over a family of classifiers $F$ in hypothesis space $\mathcal{H_H}$. $\epsilon_{S}(F)=\mathbb{E}_{(\mathrm{x}, \mathrm{y}) \sim S}\left|F(\mathrm{x}) - \mathrm{y}\right|$ is the risk of a hypothesis $F \in \mathcal{H_H}$ in distribution $S$. $\epsilon_{S}\left(F_{1}, F_{2}\right)=\mathbb{E}_{(\mathrm{x}, \mathrm{y}) \sim S}\left|F_{1}(\mathrm{x}) - F_{2}(\mathrm{x})\right|$ represents the disagreement between hypotheses $F_{1}, F_{2} \in \mathcal{H_H}$ in distribution $S$. The notations in distribution $T$ are similar. We denote the ideal hypothesis as $F^{*}=\arg \min _{F} \epsilon_{S}(F)+\epsilon_{T}(F)$. The probabilistic bound [5] of the target risk $\epsilon_{T}(F)$ of hypothesis $F$ is given by the source risk, ideal joint hypothesis risk and domain discrepancy risk, as follows:

\begin{equation}
\epsilon_{T}(F) \leqslant \epsilon_{S}(F)+\left[\epsilon_{S}\left(F^{*}\right)+\epsilon_{T}\left(F^{*}\right)\right]+\left|\epsilon_{S}\left(F, F^{*}\right)-\epsilon_{T}\left(F, F^{*}\right)\right|.
\label{theory0}
\end{equation}

For heuristic search, we have $F = G + H$, where $F$, $G$ and $H$ belong to the same hypothesis space $\mathcal{H_H}$. Also the goal of $F$ and $G$ is the same, thus the ideal hypothesis $F^*=G^*$. On the functional path towards the ideal hypothesis, the heuristic function $H$ models domain-specific parts on both $S$ and $T$. The domain-specific parts could also be approximately calculated by the parts in $F$ but not in $F^*$. The lower error bound in normal situation is proved in Section 2 in Supplementary. Typically, the two functions modeling domain-specific properties could be approximately regarded as positively correlated, as follows:

\begin{equation}
H = k(F-F^*) \qquad k \in (0,1],
\label{theory1}
\end{equation}

where $k$ is the positive correlation coefficient between $H$ and $F-F^*$, lower than $1$ constrained by Eqn <ref>. Substituting $H$ with $F-G$, we could obtain the relationship between $G$ and $F$ as follows:

\begin{equation}
(1-k)(F-F^*) = (G-G^*),
\label{theory2}
\end{equation}

where the equation could be achieved on distribution of both $S$ and $T$. Then the risk $\epsilon\left(G, G^{*}\right)= (1-k)\epsilon\left(F^*-F\right)$ is occupied on distributions of $S$ and $T$. Meanwhile, since both $G$ and $F$ could classify source samples correctly, the differences on source domain are small, which means $\epsilon_{S}(G)=\epsilon_{S}(F)$. Then we could calculate the target risk $\epsilon_{T}(G)$ as follows:

\begin{equation}
\begin{split}{
\epsilon_{T}(G) &\leqslant \epsilon_{S}(G)+\left[\epsilon_{S}\left(G^{*}\right)+\epsilon_{T}\left(G^{*}\right)\right]+\left|\epsilon_{S}\left(G, G^{*}\right)-\epsilon_{T}\left(G, G^{*}\right)\right| \qquad \qquad\\
&\leqslant \epsilon_{S}(F)+\left[\epsilon_{S}\left(F^{*}\right)+\epsilon_{T}\left(F^{*}\right)\right]+(1-k) \left|\epsilon_{S}\left(F, F^{*}\right)-\epsilon_{T}\left(F, F^{*}\right)\right|
\label{theory}
\end{equation}

where $k \in (0,1]$, thus $1-k \in [0,1)$. The utilization of $G$ could restrain the error upper-bound to a lower value, demonstrating the superiority of HDA.

§ HEURISTIC DOMAIN ADAPTATION NETWORK

§.§ Heuristic Constraint

In domain adaptation, the heuristic representations are supposed to estimate the divergence to the ideal domain-invariant representations by constructing domain-specific information. Methods in [6, 20] calculate domain-specific parts by reconstructing input images. The range of the domain-specific part has not been restrained, leading to confused domain-specific and domain-invariant properties. We construct the basic network performing HDA from the perspective of representation separation. Accordingly, we analyze the similarity, independence and termination conditions of the two part of representations.

First, to construct the heuristic function, we analyze the similarity between domain-invariant representations $G(x)$ and domain-specific representations $H(x)$. In high dimensional space, cosine similarity [47, 31] is a common method to reflect the relationship between the representations in deep networks. The cosine similarity between $G(x)$ and $H(x)$ could be calculated as:

\begin{equation}
\begin{split}{
cos(\theta) =  \frac{G(x) \cdot H(x)}{\left|G(x) \right|\left|H(x) \right|},
\label{theta}
\end{equation}

where $cos(\theta) \in \left[-1,1\right]$. We aim to eliminate the domain-specific parts in $G(x)$ with assistance of $H(x)$. $H(x)$ should not model any domain-specific representations, but $H(x)$ should construct the characteristics confused into $G(x)$ that are actually domain-specific parts. Thus $G(x)$ and $H(x)$ should contain similar domain-specific representations that could be achieved by making the initial cosine similarity to $1$ or $-1$. For better reducing the domain-specific parts, the initial similarity should be the minimum value $-1$. As the simplest form of minimum, we use the opposite initialization between $H(x)$ and $G(x)$. Denoting the initial value of $H(x)$ and $G(x)$ as $H_{init}(x)$ and $G_{init}(x)$, the heuristic similarity could be calculated as follows:

\begin{equation}
\begin{split}{
\label{initial}
\end{equation}

Next, we analyze the independence relationship between $G(x)$ and $H(x)$. $G(x)$ should maintain the same across different domains, while $H(x)$ is different across domains. Thus $G(x)$ and $H(x)$ should be independent. Traditionally, a typical way to separate the statistically independent components is Independent Component Analysis (ICA) [1].  ICA is based on the Central Limit Theorem, i.e., the distribution of a sum of independent random variables tends to converge to a Gaussian distribution.

Different from the ICA which learns the weight of each independent component, we aim to achieve a better separation from the existing representations $F(x)$. Specifically, as the sum of two independent variables produced by $G(x)$ and $H(x)$, $F(x)$ could obtain higher nongaussianity compared with $G(x)$ or $H(x)$. We choose a classical measurement of the nongaussianity, i.e., Kurtosis as follows:

\begin{equation}
\operatorname{kurt}(y)=\mathbb{E}_{y \in \mathcal{D}}\left[N(y)^{4}\right]-3\left\{\mathbb{E}_{y \in \mathcal{D}}\left[N(y)^{2}\right]\right\}^{2}
\end{equation}

where $y$ is the variable sampled from $\mathcal{D}$ distribution, $\mathbb{E}$ is the expectation. $N(\cdot)$ is the process of normalization, resulting in distribution with zero mean and unit variance.

ICA maximizes the nongaussianity of the weighted sum, to obtain one of the independent components. Similarly, we could maximize the nongaussianity of $F(x)$, pushing $F(x)$ to be independent variable to obtain more accurate $G(x)$ and $H(x)$. Besides, we construct more accurate $G(x)$ by reducing the discrepancy between $G(x)$ and the ideal separation from $F(x)$. To approximate the ideal separation, we propose heuristic independence, a constraint reducing the nongaussianity discrepancy between $F(x)$ or $G(x)$. The heuristic independence could be calculated by the difference reduction between $\operatorname{kurt}(F(x))$ and $\operatorname{kurt}(G(x))$, as follows:

\begin{equation}
\operatorname{kurt}(F(x))-\operatorname{kurt}(G(x)) \approx 0.
\label{independ}
\end{equation}

The proposed framework of Heuristic Domain Adaptation Network. The framework consists of a fundament network and a heuristic network. The heuristic network is further enhanced by multiple subnetworks modeling domain-specific representations. The overall loss consists of constraints on both the generator and heuristic loss functions.

Finally, we re-investigate the termination condition of heuristic search. The path search terminates when the path reaches the destination with zero heuristic cost. On the path to the destination, the heuristic loss gradually reduces. Similarly, in domain adaptation, the range of heuristic representations $H(x)$ converges to near zero, if the constructed domain-invariant representations are converged to the ideal. In the training process,  the range of heuristic representations should be progressively reduced to near zero towards more domain-invariant representations. Meanwhile, the heuristic termination reduces the range of $H(x)$ to be lower than the ideal, thus the termination constraint could also be regarded as the approximation of the constraint in Eqn. <ref>.

In some cases, the range of heuristic representations might be large for hard samples with rich domain-specific properties during the training process and also in the end. Considering that difficult samples hardly exist, the heuristic representations with large ranges in the dataset should be sparse. A common solution to ensure the sparsity is to minimize the $L_1$-norm [49]. The domain characteristic should be gradually reduced by penalizing large $L_1$-norm:

\begin{equation}
\begin{split}{
\quad \left|H(x)\right|_1 \approx 0.
\label{cons}
\end{equation}

§.§ Heuristic Loss

For better explanation of the framework, we take the unsupervised domain adaptation for example. We are given source domain $\mathcal{D}_S=\{(x_i^s,y_i^s) \}_{i=1}^{N_s}$ with $N_s$ labeled examples covering $C$ classes where $y_i^s \in \{1,...,C\}$ and target domain $\mathcal{D}_T=\{x_i^t \}_{i=1}^{N_T}$ with $N_t$ unlabeled examples that belong to the same $C$ classes. Example from either domain is represented as $x_i$.

For domain adaptation, the total loss $\mathcal{L}_{F}$ is composed of two parts, i.e., the generator loss $\mathcal{L}_{G}$ and the heuristic loss $\mathcal{L}_{H}$. The generator loss could be calculated by the sum of classification loss $\mathcal{L}_{Cls}$ and transfer loss $\mathcal{L}_{Trans}$, as follows:

\begin{equation}
\mathcal{L}_{F} = \mathcal{L}_{G} + \mathcal{L}_{H} = \mathcal{L}_{Cls} + \mathcal{L}_{Trans} + \mathcal{L}_{H}.
\label{lossum}
\end{equation}

On the generator, we directly calculate the adversarial discrepancy on classification responses, the same as [45, 12]. Despite of using the adversarial discrepancy, we should emphasize that HDA can also collaborate well with other discrepancy measurements.
We denote $L_{ce}$ as the cross-entropy loss function and $D$ as the constructed discriminator. Then the classification loss and the transfer loss could be calculated as:

\begin{equation}
\begin{split}{
\mathcal{L}_{Cls} = \mathbb{E}_{(x_i^s, y_i^s) \sim \mathcal{D}_S} &L_{ce} (G(x_i^s)),y_i^s) \\
\mathcal{L}_{Trans}= - \mathbb{E}_{x_i^s\sim \mathcal{D}_S} \log \left[D(G(x_i^s))\right] &- \mathbb{E}_{x_j^t\sim \mathcal{D}_T} \log \left[1- D(G(x_j^t))\right].
\label{cls} \end{equation}

For the heuristic loss $\mathcal{L}_{H}$, we should not only restrain $H(x_i)$ to be domain-specific, but also satisfy the constraints discussed in Sec. <ref>. It is difficult to directly restrain $H(x)$ to model the domain-specific properties, while alignment between $G(x_i^s)$ and $G(x_i^t)$ could contribute to more domain properties in $H(x_i)$, as discussed in [6, 12].
To enhance the ability in capturing domain-specific properties, we design $H(x_i)$ to be an additive ensemble of multiple subnetworks:

\begin{equation}
\begin{split}{
{H}(x_i) = \Sigma_{k=1}^M{H}^k(x_i), \\
\label{multi}
\end{equation}

where $M$ is the number of heuristic subnetworks. Each heuristic subnetwork $H^k(x_i)$ is differently initialized. Thus each $H^k(x_i)$ models the local domain-specific property, providing more subtle fitting ability to the domain-specific properties for the whole heuristic network $H(x_i)$, which is validated by experimental results in Figure <ref>. With the combination of multiple subnetworks, the domain-specific properties could be captured more accurately by $H(x_i)$ with larger $M$. A larger number of subnetworks $M$ may lead to more accurate construction of heuristic representations, and also higher risk of over-fitting. The proper number of heuristic subnetworks is shown in Table <ref>.

For heuristic similarity, considering that the summation of $H(x_i)$ and $G(x_i)$ is $F(x_i)$, the constraint in Eqn. <ref> becomes that $F(x_i)$ should be near zero for initialization. Thus the heuristic similarity is achieved by setting the initial fundament network parameters to near zero. To satisfy constraints of heuristic independence and termination, the range of heuristic representations should be reduced, similar to the constraint in GVB [12]. Since the constructed heuristic function contains multiple heuristic subnetworks, the reduction of range is achieved by minimizing the following objective function:

\begin{equation}
\begin{split}{
\mathcal{L}_{H}= \mathbb{E}_{x_i \sim (\mathcal{D}_S, \mathcal{D}_T)} \left|H(x_i)\right|_1= \mathbb{E}_{x_i \sim (\mathcal{D}_S, \mathcal{D}_T)} \left|\Sigma_{k=1}^M{H}^k(x_i)\right|_1.
\label{trans}
\end{equation}

In Figure <ref>, the termination constraint is shown to have similar evolving pattern to the independence constraint of Eqn. <ref> along the iteration number. The phenomenon is normal since a smaller range of $H(x_i)$ could lead to smaller difference between $G(x_i)$ and $F(x_i)$, which ensures similar nongaussianity on the other hand.

§ EXPERIMENTS

Accuracies (%) on Office-Home for ResNet50-based UDA.

Method    A$\rightarrow$C     A$\rightarrow$P     A$\rightarrow$R     C$\rightarrow$A     C$\rightarrow$P     C$\rightarrow$R     P$\rightarrow$A     P$\rightarrow$C     P$\rightarrow$R     R$\rightarrow$A     R$\rightarrow$C     R$\rightarrow$P     Avg

ResNet50 [24]     34.9     50.0     58.0     37.4     41.9     46.2     38.5     31.2     60.4     53.9     41.2     59.9     46.1

DAN [32]     43.6     57.0     67.9     45.8     56.5     60.4     44.0     43.6     67.7     63.1     51.5     74.3     56.3

DANN [18]     45.6     59.3     70.1     47.0     58.5     60.9     46.1     43.7     68.5     63.2     51.8     76.8     57.6

MCD [45]    48.9	   68.3	   74.6	   61.3	   67.6	   68.8	   57	   47.1	   75.1	   69.1	   52.2	   79.6	   64.1

EntMin [52]     43.2	   68.4	   78.4	   61.4	   69.9	   71.4	   58.5	   44.2	   78.2	   71.1	   47.6	   81.8	   64.5

CDAN [33]     50.7     70.6     76.0     57.6     70.0     70.0     57.4     50.9     77.3     70.9     56.7     81.6     65.8

Symnets [56]     47.7     72.9     78.5     64.2     71.3     74.2     63.6      47.6     79.4     73.8     50.8     82.6     67.2

SAFN [54]    52.0   	71.7   76.3   64.2   69.9   71.9   63.7   51.4   	77.1   70.9   57.1   81.5   67.3

ATM [29]     52.4    72.6    78.0    61.1    72.0    72.6    59.5    52.0    79.1    73.3    58.9    83.4    67.9

BNM [11]    52.3	   73.9	   80.0	   63.3	   72.9	   74.9	   61.7	   49.5	   79.7	   70.5	   53.6	   82.2	   67.9

MDD [57]     54.9     73.7     77.8     60.0     71.4     71.8     61.2     53.6     78.1     72.5     60.2     82.3     68.1

GVBG [12]     56.5     74.0     79.2     64.2     73.0     74.1     65.2     55.9     81.2     74.2     58.2     84.0     70.0

CADA [27]    56.9    76.4    80.7    61.3    75.2    75.2    63.2    54.5    80.7    73.9    61.5    84.1    70.2

HDAN    56.8 	   75.2 	   79.8 	   65.1 	   73.9 	   75.2 	   66.3 	   56.7 	   81.8 	   75.4 	   59.7 	   84.7 	   70.9

Parameter $M$ and ablation study on Office-Home for HDAN.

Method    A$\rightarrow$C     A$\rightarrow$P     A$\rightarrow$R     C$\rightarrow$A     C$\rightarrow$P     C$\rightarrow$R     P$\rightarrow$A     P$\rightarrow$C     P$\rightarrow$R     R$\rightarrow$A     R$\rightarrow$C     R$\rightarrow$P     Avg

HDAN (M=1)    57.2 	   74.7 	   79.7 	   65.1	   73.6 	   75.4 	   64.8 	   55.9 	   81.6 	   74.4 	   59.1 	   84.4 	   70.5

HDAN (M=2)    57.4 	   75.3 	   79.5 	   65.1 	   74.0 	   74.9 	   65.9 	   56.4 	   81.8 	   74.9 	   59.7 	   84.4 	   70.8

HDAN (M=3)    56.8 	   75.2 	   79.8 	   65.1 	   73.9 	   75.2 	   66.3 	   56.7 	   81.8 	   75.4 	   59.7 	   84.7 	   70.9

HDAN (M=4)    56.3 	   75.4 	   79.4 	   64.4 	   74.8 	   74.8 	   66.5 	   56.8 	   82.0 	   75.3 	   59.7 	   84.6 	   70.8

HDAN (M=5)    56.2 	   74.9 	   79.6 	   64.9 	   73.8 	   74.7 	   66.9 	   56.2 	   81.4 	   75.3 	   59.4 	   84.5 	   70.6

HDAN $_{w.o. range}$    53.8 	   69.9 	   78.3 	   59.9 	   72.1 	   73.0 	   60.0 	   51.8 	   79.7 	   70.8 	   56.6 	   83.0 	   67.4

HDAN $_{w.o. init}$    56.2 	   74.3 	   79.4 	   64.9 	   73.3 	   74.9 	   65.6 	   55.5 	   81.3 	   74.3 	   59.3 	   84.5 	   70.3

HDAN $_{L_2-norm}$    56.6 	   74.9 	   79.9 	   64.6 	   73.7 	   74.3 	   65.8 	   56.1 	   81.4 	   74.9 	   59.4 	   84.4 	   70.5

We evaluate our method on three challenging domain adaptation tasks, i.e., unsupervised domain adaptation, multi-source domain adaptation and semi-supervised domain adaptation. For the three tasks, we construct similar frameworks pre-trained on ImageNet [13], implemented with PyTorch. The heuristic network and the fundament network share the same feature extractor to reduce the number of parameters. To perform adversarial learning, we apply gradient reversal layer (GRL) [17] to the network. At the early stages of the training procedure, the progressive training strategy in GRL could avoid noisy signal from the domain classifier. Besides, we utilize entropy conditioning [33] for alignment between easy-to-classify samples. We employ Stochastic Gradient Descent (SGD) with a momentum of 0.9 and a weight decay of $0.0005$ to train our model. We repeat the experiments of our proposed method over 3 times to report the average of the top-1 classification accuracy. For fair comparisons, the results are directly reported from their original papers if available.

§.§ Unsupervised Domain Adaptation

Unsupervised domain adaptation (UDA) is the classical domain adaptation task performed on labeled source domain and unlabeled target domain. In UDA, we use a standard dataset
Office-Home [51] with 15,500 images in 65 categories. It has four significantly different domains: Artistic images (A), Clip art (C), Product images (P), and Real world (R). The four domains build up 12 challenging UDA scenarios in total.

The experiment results based on ResNet50 [24] are shown in Table <ref>. Experiments show that HDAN could achieve state-of-the-art results on Office-Home. On average, HDAN could achieve $0.9\%$ improvements compared with the method of GVBG [12]. The improvement shows the importance of heuristic similarity and multiple heuristic subnetworks on Office-Home. Besides, HDAN outperforms competitors by a large margin, which shows HDAN is a more powerful framework in UDA.

We also show the results under different settings of $M$ and ablation study in Table <ref>. When $M=1$, there is only one heuristic sub-network. HDAN achieves the best performance when $M=3$ on Office-Home. This validates that more heuristic sub-networks could enhance the ability of modeling heuristics. If $M$ becomes too large, the heuristic function is difficult to optimize and the whole model suffers from over-fitting. Thus we fix $M=3$ for consequent experiments. In ablation study, results of HDAN become worse without heuristic termination or heuristic similarity, which validates the positive effect of the constraints. HDAN with $L_2$-norm in Eqn. <ref> performs worse than $L_1$-norm.

To show more domain-specific properties in $H(x)$, we construct two extra domain classifier to calculate the domain belongings in Figure <ref>, and the details are discussed in Section A.3 in Supplementary. Results show that the domain classification accuracy tends to be higher in $H(x)$ compared to $G(x)$. This result shows that more domain properties exist in $H(x)$. Meanwhile, each $H^k$ is initialized with random weights generated from Gaussian with different variance, resulting in different output ranges in the training process, as shown in Figure <ref>. It could also ensure that distinguished local or cluster-sensitive domain-specific properties can be captured by different $H^k$, as shown in Figure <ref> and <ref>. The ensemble of $H^k(x)$ can capture rich domain-specific properties more  effectively, evidenced by the low similarity between $H^k(x)$ and $G(x)$, i.e., $\cos(G(x), H^k(x))\approx 0$, as shown in Figure <ref>. Moreover, different $H^k(x)$ contains distinguished local property, since $H^k(x)$ and $H^{k'}(x) \ (k \neq k')$ is orthogonal or negatively correlated, i.e., $\cos(H^k(x), H^{k'}(x))\leq 0$, as shown in Figure <ref>. Therefore, the aggregation of $H^k(x)$ could express more diversified domain-specific properties encoded by each sub-network.

We also visualize the loss functions in the training process of HDAN on A$\rightarrow$C in Figure <ref>. As expected, the cosine similarity reduces rapidly. The nongaussianity gap between $G_n(x)$ and $H_n(x)$ is also progressively reduced. This validates that heuristic termination could ensure the independence between the domain-specific and domain-invariant representations. We also visualize the T-SNE results in Figure <ref>, with source domain in blue and target domain in red. The visualization shows that HDAN could achieve surprisingly good distribution alignment at the category level on Office-Home.

Domain accuracy (%) on C$\rightarrow$R.

[$G(x)$ and $H^k(x)$]

[$H^k(x)$and $H^{k'}(x)$]

Relationship among different domain parts on A$\rightarrow$C.

[Cosine similarity]

Losses of HDAN on A$\rightarrow$C.

T-SNE results of HDAN on A$\rightarrow$C.

§.§ Multi-source Domain Adaptation

Accuracies (%) on DomainNet for ResNet101-based MSDA methods.

Models   Clipart      Infograph     Painting    Quickdraw       Real     Sketch     Avg

ResNet101 [24]     47.6$\pm$0.52      13.0$\pm$0.41     38.1$\pm$0.45       13.3$\pm$0.39     51.9$\pm$0.85     33.7$\pm$0.54     32.9$\pm$0.54

DAN [32]    45.4$\pm$0.49   	12.8$\pm$0.86   	36.2$\pm$0.58   	15.3$\pm$0.37   	48.6$\pm$0.72   	34.0$\pm$0.54   	32.1$\pm$0.59

RTN [34]     44.2$\pm$0.57   	12.6$\pm$0.73   	35.3$\pm$0.59   	14.6$\pm$0.76   	48.4$\pm$0.67   	31.7$\pm$0.73   	31.1$\pm$0.68

JAN [35]    40.9$\pm$0.43   	11.1$\pm$0.61   	35.4$\pm$0.50   	12.1$\pm$0.67   	45.8$\pm$0.59   	32.3$\pm$0.63   	29.6$\pm$0.57

DANN [18]    45.5$\pm$0.59   	13.1$\pm$0.72   	37.0$\pm$0.69   	13.2$\pm$0.77   	48.9$\pm$0.65   	31.8$\pm$0.62   	32.6$\pm$0.68

ADDA [50]    47.5$\pm$0.76   	11.4$\pm$0.67   	36.7$\pm$0.53   	14.7$\pm$0.50   	49.1$\pm$0.82   	33.5$\pm$0.49   	32.2$\pm$0.63

SE [16]    24.7$\pm$0.32   	3.9$\pm$0.47   	12.7$\pm$0.35   	 7.1$\pm$0.46   	22.8$\pm$0.51   	9.1$\pm$0.49   	16.1$\pm$0.43

MCD [45]    54.3$\pm$0.64   	22.1$\pm$0.70   	45.7$\pm$0.63   	7.6$\pm$0.49   	58.4$\pm$0.65   	43.5$\pm$0.57   	38.5$\pm$0.61

DCTN [53]    48.6$\pm$0.73      23.5$\pm$0.59     48.8$\pm$0.63     7.2$\pm$0.46    53.5$\pm$0.56     47.3$\pm$0.47     38.2$\pm$0.57

M$^{3}$SDA [39]   57.2$\pm$0.98      24.2$\pm$1.21     51.6$\pm$0.44    5.2$\pm$0.45      61.6$\pm$0.89    49.6$\pm$0.56      41.5$\pm$0.74

M$^{3}$SDA-$\beta$ [39]   58.6$\pm$0.53    26.0$\pm$ 0.89    52.3$\pm$0.55    6.3$\pm$0.58    62.7$\pm$0.51    49.5$\pm$0.76    42.6$\pm$0.64

ML-MSDA [30]    61.4$\pm$0.79    26.2$\pm$0.41    51.9$\pm$0.20    19.1$\pm$0.31    57.0$\pm$1.04    50.3$\pm$0.67    44.3$\pm$0.57

GVBG [12]    61.5$\pm$0.44	   23.9$\pm$ 0.71	   54.2$\pm$0.46 	   16.4$\pm$ 0.57	   67.8$\pm$0.98 	   52.5$\pm$0.62 	   46.0$\pm$0.63

HDAN    63.6$\pm$0.35 	   25.9$\pm$0.16	   56.1$\pm$0.38 	   16.6$\pm$ 0.54	   69.1$\pm$0.42	   54.3$\pm$ 0.26	   47.6$\pm$0.40

Multi-source domain adaptation (MSDA) is a challenging task, since the multiple source domains bring great difficulties to the domain adaptability. For evaluation on MSDA, we utilize a challenging dataset DomainNet [39], which contains about 600,000 images in 345 categories. DomainNet consists of six significantly different domains: Clipart, Infograph, Painting, Quickdraw, Real and Sketch. We follow the same data split scheme of training and validation images in the paper. In each domain adaptation setting, we choose only one domain as target domain, with others as the source domains. Thus the six domains formulate six challenging MSDA scenarios in total. To simplify, we only choose the target domain label as the label of different setting. For example, the label Clipart refers to the setting of transferring from Inforgraph, Painting, Quickdraw, Real and Sketch to Clipart.

Different from the setting of UDA,  multiple source domains are labeled as different domains in adversarial training. Thus we make predictions on the domain labels at the discriminator side. On the generator, we keep fixing the number of heuristic network $M$ to 3, according to previous ablation study. We choose ResNet101 [24] as the basic backbone, and set the initial learning rate as $0.0003$.
Since methods trained under MSDA setting demonstrate large performance gap over methods trained under the single best setting as reported in [39], we only report results under the setting of combined source and multi-source in this paper.
We reproduce GVBG [12] in the same settings with our method for fair comparison.

The experimental results of multi-source domain adaptation are shown in Table <ref>. Our method HDAN achieves state-of-the-art results on DomainNet. More specifically, HDAN outperforms other methods on all the settings except two challenging ones, i.e., Quickdraw and Infograph. The results indicate that with the global guidance of heuristic information, HDAN is quite suitable for normal multi-source domain adaptation. HDAN also outperforms GVBG [12], which shows the effectiveness of our constraints. Besides, the standard deviations of our results are relatively small, which means the training process of HDAN performs smoothly.

§.§ Semi-supervised Domain Adaptation

Semi-supervised domain adaptation (SSDA) is a more realistic setting, where the domain adaptation is performed from labeled source domain to partially labeled target domain. In SSDA, we utilize the standard dataset proposed by MME [43], which is selected from DomainNet [39]. We take the same protocol as  [43] where the four domains including Real (R), Clipart (C), Painting (P) and Sketch (S) in total 126 categories. In this paper, same as <cit.>, seven domain adaptation settings are formulated on the four domains, showing different levels of domain gap.

We use the ResNet34 [24] as the backbones of the generator and fix the number of sub-networks to be 3, and with the initial learning rate as $0.001$. On the discriminator, the labeled target domain is labeled as the third domain in the minmax game. All the methods are evaluated under the one-shot and three-shot settings following [43] where there are only one or three labeled samples per class in the target domain. We also reproduce the results of GVBG [12] in the same settings as our method, by fitting the adversarial process into semi-supervised domain adaptation.

The quantitative results on the selected DomainNet subset are summarized in Table <ref>. It is easily observed that HDAN outperforms all the competitors, and achieves state-of-the-art results on average. For example, compared with CANN, our method improves significantly on the difficult tasks, such as ${R} \rightarrow {S}$. HDAN also achieves significant improvement on one-shot SSDA, which shows the superiority of HDAN. Besides, the improvement compared with GVBG [12] shows the effectiveness of our heuristic constraints.

Accuracies (%) on DomainNet subset of ResNet34-based SSDA methods.

2*Methods     2cR$\rightarrow$C     2cR$\rightarrow$P     2cP$\rightarrow$C     2cC$\rightarrow$S     2cS$\rightarrow$P     2cR$\rightarrow$S     2cP$\rightarrow$R     2cAvg

1$_{shot}$    3$_{shot}$    1$_{shot}$    3$_{shot}$    1$_{shot}$    3$_{shot}$    1$_{shot}$    3$_{shot}$    1$_{shot}$    3$_{shot}$    1$_{shot}$    3-shot    1$_{shot}$    3$_{shot}$    1$_{shot}$    1c3$_{shot}$

1cResNet34 [24]    55.6   60.0     60.6   62.2      56.8   59.4     50.8   55.0     56.0   59.5     46.3   50.1    71.8   73.9    56.9   1c60.0

1cDANN [18]    58.2   59.8     61.4   62.8      56.3   59.6     52.8   55.4     57.4   59.9     52.2   54.9    70.3   72.2    58.4   1c60.7

1cADR [44]    57.1   60.7     61.3   61.9      57.0   60.7     51.0   54.4     56.0   59.9     49.0   51.1    72.0   74.2    57.6   1c60.4

1cCDAN [33]    65.0   69.0     64.9   67.3      63.7   68.4     53.1   57.8     63.4   65.3     54.5   59.0    73.2   78.5    62.5   1c66.5

1cENT [22]    65.2   71.0     65.9   69.2      65.4   71.1     54.6   60.0     59.7   62.1     52.1   61.1    75.0   78.6    62.6   1c67.6

1cMME [43]    70.0   72.2     67.7   69.7      69.0   71.7     56.3   61.8     64.8   66.8     61.0   61.9    76.1   78.5    66.4   1c68.9

1cGVBG [12]   70.8	   	73.3   65.9   	68.7	   71.1    	72.9	   62.4   	65.3 	   65.1    		66.6	   67.1    	68.5	   76.8	   79.2	   68.4	   	70.6

1cCANN [41]    72.7   75.4     70.3   71.5    69.8   73.2     60.5   64.1    66.4   69.4     62.7   64.2    77.3   80.8     68.5   1c71.2

1cHDAN    71.7    73.9	   67.1   69.1	   72.8   73.0   	63.7   	66.3	   65.7    67.5   69.2   		69.5   	76.6    79.7     69.5    1c71.3

§ CONCLUSION

We consider to explicitly separate domain-specific characteristics from the domain-invariant representations for domain adaptation. Inspired by heuristic search, we propose a new framework HDA which achieves lower error bound by theoretical analysis. We design deep network HDAN to perform HDA, which contains multiple subnetworks in the heuristic network $H$ to ensure accurate modeling of domain-specific properties.
Three constraints on initial parameter similarity, network output independence and the termination of heuristics are designed to train HDAN, which ensures more comprehensive reduction of domain properties.
Experiments show state-of-the-art results achieved by HDAN on three domain adaptation tasks.

§ BROADER IMPACT

Domain adaptation aims to obtain the domain-invariant representations across training and testing distributions, which is a general problem in machine learning. Our method stresses the importance of explicitly constructing domain-specific representations as the heuristics towards effective domain adaptation. The guidance of heuristics formulate the principled framework of HDA, similar to heuristic search. Despite of the similarity in working mechanism, remarkable difference still exist between heuristic search and our method from the task setting aspect.

In our method, heuristics are constructed in a way similar to blind signal separation. The separation process could also be applied to a broader range of tasks, such as signal reconstruction, representation learning or visual tracking. However, the separation could not achieve ideal separation without extra knowledge or priors. In common situations, there is no general method constructing the heuristic knowledge, which is another limitation of the method.

The framework of HDA could also be further improved from two perspectives. First, some simply constructed extra domain-specific knowledge could be introduced to enhance the separation. Moreover, the structure of our constructed heuristic networks is an additive ensemble of sub-networks, which could be replaced by Neural Architecture Search (NAS).

Acknowledgement. This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102003, in part by National Natural Science Foundation of China: 61672497, 61620106009, 61836002, 61931008 and U1636214, and in part by Key Research Program of Frontier Sciences, CAS: QYZDJ-SSW-SYS013.

[1]
Hyvärinen A and Oja E.
Independent component analysis: algorithms and applications.
Neural networks, 13(4-5):411–430, 2000.

[2]
Rie Kubota Ando and Tong Zhang.
A framework for learning predictive structures from multiple tasks
and unlabeled data.
Journal of Machine Learning Research, 6(Nov):1817–1853, 2005.

[3]
Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil.
Multi-task feature learning.
In Advances in neural information processing systems, pages
41–48, 2007.

[4]
Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil.
Convex multi-task feature learning.
Machine learning, 73(3):243–272, 2008.

[5]
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and
Jennifer Wortman Vaughan.
A theory of learning from different domains.
Machine learning, 79(1-2):151–175, 2010.

[6]
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan,
and Dumitru Erhan.
Domain separation networks.
In Advances in Neural Information Processing Systems, pages
343–351, 2016.

[7]
Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Michael I. Jordan.
Partial transfer learning with selective adversarial networks.
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2018.

[8]
Rich Caruana.
Multitask learning.
Machine learning, 28(1):41–75, 1997.

[9]
Jr L A Celiberto, J P Matsuura, and R L De Mantaras.
Using cases as heuristics in reinforcement learning: a transfer
learning application.
In Twenty-Second International Joint Conference on Artificial
Intelligence, 2011.

[10]
Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang.
Transferability vs. discriminability: Batch spectral penalization for
adversarial domain adaptation.
In International Conference on Machine Learning, pages
1081–1090, 2019.

[11]
Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qingming Huang, and Qi Tian.
Towards discriminability and diversity: Batch nuclear-norm
maximization under label insufficient situations.
In IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), June 2020.

[12]
Shuhao Cui, Shuhui Wang, Junbao Zhuo, Chi Su, Qingming Huang, and Tian Qi.
Gradually vanishing bridge for adversarial domain adaptation.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, 2020.

[13]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 248–255, 2009.

[14]
Theodoros Evgeniou, Charles A Micchelli, and Massimiliano Pontil.
Learning multiple tasks with kernel methods.
Journal of machine learning research, 6(Apr):615–637, 2005.

[15]
Theodoros Evgeniou and Massimiliano Pontil.
Regularized multi–task learning.
In Proceedings of the tenth ACM SIGKDD international conference
on Knowledge discovery and data mining, pages 109–117, 2004.

[16]
Geoff French, Michal Mackiewicz, and Mark Fisher.
Self-ensembling for visual domain adaptation.
In International Conference on Learning Representations, 2018.

[17]
Yaroslav Ganin and Victor Lempitsky.
Unsupervised domain adaptation by backpropagation.
In International conference on machine learning, pages
1180–1189, 2015.

[18]
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo
Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.
Domain-adversarial training of neural networks.
The Journal of Machine Learning Research, 17(1):2096–2030,

[19]
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman.
Geodesic flow kernel for unsupervised domain adaptation.
In 2012 IEEE Conference on Computer Vision and Pattern
Recognition, pages 2066–2073. IEEE, 2012.

[20]
Rui Gong, Wen Li, Yuhua Chen, and Luc Van Gool.
Dlow: Domain flow for adaptation and generalization.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 2477–2486, 2019.

[21]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial nets.
In Advances in neural information processing systems, pages
2672–2680, 2014.

[22]
Yves Grandvalet and Yoshua Bengio.
Semi-supervised learning by entropy minimization.
In Advances in neural information processing systems, pages
529–536, 2005.

[23]
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and
Alexander Smola.
A kernel two-sample test.
Journal of Machine Learning Research, 13(Mar):723–773, 2012.

[24]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 770–778, 2016.

[25]
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate
Saenko, Alexei A Efros, and Trevor Darrell.
Cycada: Cycle-consistent adversarial domain adaptation.
In International conference on machine learning, pages
1989–1998, 2018.

[26]
Guoliang Kang, Liang Zheng, Yan Yan, and Yi Yang.
Deep adversarial attention alignment for unsupervised domain
adaptation: the benefit of target expectation maximization.
In Proceedings of the European Conference on Computer Vision
(ECCV), pages 401–416, 2018.

[27]
Vinod Kumar Kurmi, Shanu Kumar, and Vinay P. Namboodiri.
Attending to discriminative certainty for domain adaptation.
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2019.

[28]
Chen-Yu Lee, Tanmay Batra, Mohammad Haris Baig, and Daniel Ulbricht.
Sliced wasserstein discrepancy for unsupervised domain adaptation.
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2019.

[29]
J. Li, E. Chen, Z. Ding, L. Zhu, K. Lu, and H. T. Shen.
Maximum density divergence for domain adaptation.
IEEE Transactions on Pattern Analysis and Machine Intelligence,
pages 1–1, 2020.

[30]
Z Li, Z Zhao, Y Guo, H Shen, and J Ye.
Mutual learning network for multi-source domain adaptation.
arXiv preprint arXiv:2003.12944, 2019.

[31]
W Liu, YM Zhang, X Li, Z Yu, B Dai, T Zhao, and Le Song.
Deep hyperspherical learning.
In Advances in neural information processing systems, pages
3950–3960, 2017.

[32]
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I Jordan.
Learning transferable features with deep adaptation networks.
In International conference on machine learning, pages 97–105,

[33]
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan.
Conditional adversarial domain adaptation.
In Advances in Neural Information Processing Systems, pages
1647–1657, 2018.

[34]
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.
Unsupervised domain adaptation with residual transfer networks.
In Advances in Neural Information Processing Systems, pages
136–144, 2016.

[35]
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.
Deep transfer learning with joint adaptation networks.
In Proceedings of the 34th International Conference on Machine
Learning-Volume 70, pages 2208–2217. JMLR. org, 2017.

[36]
Sinno Jialin Pan and Qiang Yang.
A survey on transfer learning.
IEEE Transactions on knowledge and data engineering,
22(10):1345–1359, 2009.

[37]
Shibin Parameswaran and Kilian Q Weinberger.
Large margin multi-task metric learning.
In Advances in neural information processing systems, pages
1867–1875, 2010.

[38]
J. Pearl.
Intelligent search strategies for computer problem solving.
Addision Wesley, 1984.

[39]
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang.
Moment matching for multi-source domain adaptation.
In Proceedings of the IEEE International Conference on Computer
Vision, pages 1406–1415, 2019.

[40]
I. Pohl.
First results on the effect of error in heuristic search.
Machine Intelligence, 5:219–236, 1970.

[41]
C Qin, L Wang, Q Ma, Y Yin, H Wang, and Y Fu.
Opposite structure learning for semi-supervised domain adaptation.
arXiv preprint arXiv:2002.02545, 2019.

[42]
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell.
Adapting visual category models to new domains.
In European conference on computer vision, pages 213–226.
Springer, 2010.

[43]
Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko.
Semi-supervised domain adaptation via minimax entropy.
In Proceedings of the IEEE International Conference on Computer
Vision, pages 8050–8058, 2019.

[44]
Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko.
Adversarial dropout regularization.
International Conference on Learning Representations, 2018.

[45]
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada.
Maximum classifier discrepancy for unsupervised domain adaptation.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 3723–3732, 2018.

[46]
Christos Sakaridis, Dengxin Dai, Simon Hecker, and Luc Van Gool.
Model adaptation with synthetic and real data for semantic dense
foggy scene understanding.
In European Conference on Computer Vision (ECCV), pages
707–724, 2018.

[47]
A. Singhal.
Modern information retrieval: A brief overview.
IEEE Data Eng. Bull., 24(4):35–43, 2001.

[48]
Baochen Sun and Kate Saenko.
Deep coral: Correlation alignment for deep domain adaptation.
In European Conference on Computer Vision, pages 443–450.
Springer, 2016.

[49]
R. Tibshirani.
Regression shrinkage and selection via the lasso.
Journal of the Royal Statistical Society: Series B
(Methodological), 58(1):267–288, 1996.

[50]
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.
Adversarial discriminative domain adaptation.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 7167–7176, 2017.

[51]
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman
Deep hashing network for unsupervised domain adaptation.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 5018–5027, 2017.

[52]
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick
Advent: Adversarial entropy minimization for domain adaptation in
semantic segmentation.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 2517–2526, 2019.

[53]
Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin.
Deep cocktail network: Multi-source unsupervised domain adaptation
with category shift.
In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), June 2018.

[54]
Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin.
Larger norm more transferable: An adaptive feature norm approach for
unsupervised domain adaptation.
In The IEEE International Conference on Computer Vision (ICCV),
October 2019.

[55]
Hongliang Yan, Yukang Ding, Peihua Li, Qilong Wang, Yong Xu, and Wangmeng Zuo.
Mind the class weight bias: Weighted maximum mean discrepancy for
unsupervised domain adaptation.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 2272–2281, 2017.

[56]
Yabin Zhang, Hui Tang, Kui Jia, and Mingkui Tan.
Domain-symmetric networks for adversarial domain adaptation.
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 5031–5040, 2019.

[57]
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan.
Bridging theory and algorithm for domain adaptation.
In International Conference on Machine Learning, pages
7404–7413, 2019.

[58]
Junbao Zhuo, Shuhui Wang, Shuhao Cui, and Qingming Huang.
Unsupervised open domain recognition by semantic discrepancy
In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 750–759, 2019.

[59]
Junbao Zhuo, Shuhui Wang, Weigang Zhang, and Qingming Huang.
Deep unsupervised convolutional domain adaptation.
In Proceedings of the 25th ACM international conference on
Multimedia, pages 261–269. ACM, 2017.

[60]
Yang Zou, Zhiding Yu, Xiaofeng Liu, B.V.K. Vijaya Kumar, and Jinsong Wang.
Confidence regularized self-training.
In The IEEE International Conference on Computer Vision (ICCV),
October 2019.


# Archiving Body Movements: Collective Generation of Chinese Calligraphy

Aven Le ZHOU 1, Jiayi YE2, Tiancheng LIU1, Kang ZHANG1
1 The Hong Kong University of Science and Technology (Guangzhou) 2 Shanghai
Jiao Tong University
<EMAIL_ADDRESS>
###### Abstract

As a communication channel, body movements have been widely explored in
behavioral studies and kinesics. Performing and visual arts share the same
interests but focus on documenting and representing human body movements, such
as for dance notation and visual work creation. This paper investigates body
movements in oriental calligraphy and how to apply calligraphy principles to
stimulate and archive body movements.

Through an artwork(Wushu), the authors experiment with an interactive and
generative approach to engage the audience’s bodily participation and archive
the body movements as a compendium of generated calligraphy. The audience
assumes the role of both writers and readers; creating (“writing”) and
appreciating (“reading”) the generated calligraphy becomes a cyclical process
within this infinite “Book,” which can motivate further attention and
discussions concerning Chinese characters and calligraphy.

## Keywords

Body Movement, Bodily Engagement, Chinese Calligraphy, Chinese Painterly
Content, Interactive Art, Generative Art, AI-Generated Painterly Content.

## Introduction and Motivation

Calligraphy creation is a two-part process, “(body) movements in the air” and
“interaction between brush and canvas” [9]. The body movements initialize the
dynamics and generate forces to drive the brush with ink and start the
creative journey. The contact of brush and canvas, following the driving force
and body movement, creates strokes and ink effects. Once the writing starts,
these two actions alternate as cause and effect of each other. From a
temporal-spatial perspective, Chinese calligraphy is the representation of
sequential strokes: the temporal body movements and the generated moving
brushwork and strokes, and the spatial structure of the Chinese characters and
their arrangement compositions on the canvas. The brushwork’s trajectory is
temporal (like the musical rhythm and choreography) and spatial (like painting
compositions and sculpture structures). Calligraphy emphasizes using the
body’s energy to achieve the artistic creation of “unity of the body and
mind.” The body movements generate the calligraphy. Meanwhile, the calligraphy
and the strokes on the canvas visualize and document the body movement through
its own principles.

Through empirical analysis, experienced calligraphers and researchers can
identify the “interaction between brush and canvas” behind the strokes. This
is because they are identical since “the movement is the Calligraphy” [9]. But
it requires trained expertise and in-depth knowledge of calligraphy, yet not a
complete reflection of body movements. Experts can observe the shape of the
strokes, the space between them, and the ink effects to identify the
“interaction between brush and canvas” but not the “movements in the air” that
generate them. Unfortunately, calligraphy is only presented in its static
format, and the audience cannot easily imagine the body movements of the
calligrapher [23].

To archive body movements, calligraphy could provide new insights into
representing body movements through its principles; meanwhile, the art could
benefit from these explorations. This work investigates how to apply
calligraphy and its principles to represent and archive body movements. To
achieve the proposed goals, we experiment with an interactive artwork, an
infinite and collectively-written “book” (i.e., Wushu). It first encourages
the audience’s participatory bodily engagement through interactive
calligraphic visuals (i.e., interactive ink and stroke effects). It then
transforms the participants’ active participation (i.e., body movements) to
Chinese characters using multiple neural networks and then to calligraphic
characters, layout, etc. Further, Wushu accumulates participants-generated
calligraphy and presents them as a compendium and digital archive, which keeps
expanding with the audience’s collective participation.

We highlight our essential contributions to this research investigation as (1)
an interactive and generative approach to represent and archive body
movements; and (2) an artwork and onsite experiments on engaging the
audience’s bodily participation and archiving the body movements as a
compendium of generated calligraphy.

## Background and Contextualization

Behavioral studies that are rooted in the conjunctive fields of sociology and
anthropology have a long interest in human body movement (or motion) 111We can
use “movement” and “motion” interchangeably in the scope of this manuscript,
but will continue with “movement.” and its meaning. Kinesics, a
multidisciplinary nonverbal communication subfield, deals with interpreting
body movements, gestures, and facial expressions[4]. According to the
anthropologist Birdwhistell, who coined the term “kinesics” in the 1950s, all
body movements have meanings; this nonverbal behavior has a grammar that can
be analyzed similarly to spoken language and can vary across different
cultures[5].

Creative professionals in performing arts, such as choreographers and dancers,
have thoroughly studied how to visualize and document body movements through
various explorations since the 18th century [18]. Meanwhile, visual artists
are more interested in how to interpret and represent the human movement in
various forms of artworks. Similarly, in oriental art and tradition, body
movement is involved in various ways in the origin, development, and modern
evolutions of calligraphy.

### Documentating Body Movement in Performing Arts

Pursuing a tooling function to notate body movements, several systems are
invented by performing artists and are widely used for choreography.
Labanotation is structured for analyzing and recording body movement with
symbols [15]; similarly, Laban Movement Analysis (LMA) is a method and
language for describing, visualizing, interpreting, and documenting movements
of the human body [14]. Benesh Movement Notation (BMN) records dance movements
using symbols representing different body parts [3]. Eshkol-Wachman Movement
Notation (EWMN) treats the body as a stick figure and uses a spherical
coordinate system to describe the relationship of different body parts in
three-dimensional space [28].

More scientific and artistic explorations on recording and documenting body
movements are not limited to these functional choreographic notations.
Stroboscopy, invented by Harold Edgerton, is a technique that uses strobe
lights to capture images of moving objects. Edgerton used this technique to
record a person jumping rope [2]. William Forsythe, in his dancing work “Point
Point Line” and “Dropping Curves”, explores the relationship between the human
body and space by drawing lines and curves through body motion [27]. Oskar
Schlemmer develops the analytic Stelzenläufer (Slat Dance), consisting of
dancers wearing stilts and performing movements based on geometric shapes
[28].

### Representating Body Movement in Visual Arts

The interest in body movements is not limited to performing arts; it is more
generally explored as visual art practices and not only as documentation and
archives but also as representation. The term Chronophotography is coined by
Étienne-Jules Marey, and he uses a “motion capture suit” (i.e., a skeletal
marker cloth) and multiple cameras to capture images of moving humans as
“Chronophotography of a Walking Man” in the 1890s [7].

Eadweard Muybridge’s photographic study of motion, including the “Woman
Walking Downstairs, Picking up Pitcher, and Turning”, is a series of
instantaneous photographs that document the movement of humans and animals
[25]. Frank and Lillian Gilbreth develop a different technique for conducting
chronophotographic time-motion studies. They use motion picture cameras to
record and analyze workers’ movements in repetitive tasks and develop a system
of 17 behavioral units to describe motion [24].

Similar to Chronophotography, Marcel Duchamp’s “Nu descendant un escalier
(1912)” depicts motion by successive superimposed images [13]. “Unique Forms
of Continuity in Space (1913)” by Umberto Boccioni is a sculpture that
explores the relationship between the human body and space [10]. The sculpture
consists of a figure that appears to be moving through space, with its limbs
and torso elongated to suggest motion [10]. In “Forms (2012),” Memo Akten and
Davide Quayola explore the relationship between the human body and geometry
via algorithm-generated abstract shapes from motion-captured data from
athletes [29].

Figure 1: Participant vs. Viewer: the Creative Process. ©Authors.

### Body Movement in Oriental Calligraphpy

Figure 2: An example of Zhang Xu’s Calligraphy: Four Poems. ©Authors.

Calligraphy in Eastern Asia refers to writing Chinese characters with ink and
brush following different principles, which originated in China and also
became popular in both Japan (shodō) and Korea (seoye) [11]. The Chinese
calligraphy chronologically evolves through different levels of body
participation that go beyond the common understanding of “writing” and are
performative with body involvement [19]. The varied bodily engagements with
calligraphy result from the changing lifestyle (e.g., furnishing and interior
design) and the iterative formats of calligraphy in terms of the canvas (i.e.,
hand scroll, vertical scroll, and multiple-panel) in different materials,
sizes, and presentation methods.

Accordingly, Chinese Calligraphy starts with writing on a handheld scroll or
on the table, then writing on the wall or on the ground (with vertical
scrolls) [19]. In the late Ming Dynasty, the conventional form of Chinese
Calligraphy changes from the original horizontal-based hand scrolls to meters-
long vertical scrolls. It is no longer appropriate for calligraphers to write
either holding the scroll or sitting in front of a table, but they begin to
stand up and move against the wall or ground [21], which is accompanied by
increasing involvement of the body.

Bodily engagement plays an important role in calligraphy creation and
evolution and is a source of inspiration. According to a legend, Zhang Xu
creates cursive calligraphy inspired by the Sword Dancing of GongSun; he
usually gets drunk and writes like dancing. He even uses his hair as a brush
with dramatic body movements [1]. Huai-su writes on the wall and moves his
body along with the wall, where writing becomes dancing with whole-body
participation [26]. In these various calligraphy creation processes, the
kinesthetic system is essential, and calligraphers coordinate their body
movement in writing for the purpose of artistic expression [9].

## Method

Figure 3: Wushu (testing): Installed with a Curved Screen and Open to Internal
Audience. ©Authors.

### Participant vs. Viewer

By combining the concepts of Interactive Art and Generative Art, we propose a
new perspective to examine the typical process of creating and appreciating
painterly content (e.g., a painter creates an artwork that is later
appreciated by the audience). In Interactive Art, the Audience can be more
than a Viewer; they can also be a Participant who engages with the interactive
artwork. When the interactive artwork incorporates algorithmic generative
capability as part of the interactive system, especially those algorithms that
generate painterly content conditioned on the participant’s real-time inputs,
the participant co-creates the artifact (generated painterly content) with the
interactive artwork.

Within this paradigm (see Figure 2), we distinguish the difference between the
Participant and the Viewer among the Audience; the Participants gain
interactive experiences by engaging with the interactive artwork and become
the participant-creator (differs from artist-creator) in the co-creation
process, while the Viewers do not. During this process, we propose to focus on
the mechanisms and design that enhance their participation, ideally in a
feedback loop of engagement (i.e., the $\alpha$ loop in Figure 2).
Furthermore, we emphasize that the roles of the Participant and the Viewer
should be carefully considered individually; meanwhile, they are
interchangeable, and the transformation and transition between them can be a
cyclic process (i.e., the $\beta$ loop in Figure 2) further to improve the
Audience’s experience with the artwork.

### Essential Principles in Calligraphy

#### Ink Effect and Brush Stroke

In calligraphy, ink effects depend on the control of the brush and the ink
inside and their contact with the canvas through various writing speeds and
brush techniques. Strokes in calligraphy are about shaping them into diverse
expressive forms through brush manipulation. Brush techniques, such as moving
forward or backward, fast or slow, lifting or pressing down, turning
directions, pausing, etc., produce strokes in different forms that differ in
length, roundness, continuity, thickness, and curvature.

#### Character Composition

The essence of character composition is the manipulation and combination of
strokes based on the structure of the original Chinese character. It involves
the position and direction of each stroke and the organization of strokes,
such as how to cluster them, balance the relationship between them, and
control the density of their composition. There are two character composition
structures (i.e., Fat vs. Slim) among all calligraphers in different styles,
which originate from and refer to the human body description.

#### Layout Composition

Chinese calligraphy is written from top to bottom and right to left, one
column after another. The layout composition principle designs the arrangement
at the macro and micro levels. The major design concerns are the distance
between the characters and the columns, how to cluster them and leave white
spaces, and where to place the seals. The characters should not be written to
the very edge, and a margin should be left on the four sides. White spaces are
expected among columns to allow the energy to flow [9].

## Experiments

We experiment with Wushu to encourage and archive participants’ body movements
as a collective compendium. The interactive artwork first engages the audience
and generates ink effects and strokes from their motion input. Two customized
neural networks generate bilingual characters from the audience’s engagements
through a “Motion Caption” process. Procedural algorithms derived from
calligraphy research based on its essential principles manipulate the
previously generated characters in custom layout compositions.

Through the work, the audience participates and automatically “writes” their
body movements in the genre of Chinese calligraphy. Each participant does not
create independent work but collectively contributes and works on a shared
canvas. Only when multiple participatory-generated calligraphic characters
fully occupy the current canvas, it will be automatically archived, and a
blank canvas will be initialized. On the other hand, Wushu keeps expanding
with the generated calligraphy, which serves as a compendium of the audience’s
collective creation and a digital archive of their body movements.

### Audience Engagement and Interaction

When the audience presents themselves in front of the artwork, Wushu
establishes their skeletal body pose and uses the chosen joint (e.g., hand
position) and its (recorded) trajectory to automatically and simultaneously
create interactive ink effects and calligraphic stroke. The real-time
generated visual responds to and encourages the participant’s bodily
engagements and serves as the input for calligraphy generation.

Figure 4: Body Movements (Left) to Ink Effects (Middle) and Strokes (Right).
©Authors.

#### Body Pose (and Trajectory) Estimation

We use Google MediaPipe [22] for the human body pose estimation, which
provides a set of body joint temporal positions $J_{t}$ at a fixed time
period. In equation (1), $i$ denotes the joint index, and $t$ represents the
time.

$J_{t}=\\{j_{t}^{i}\\},$ (1)

Equation (2) shows the joint position (e.g., hand position) $h_{t}$ can be
obtained by selection of index. We use the current $h_{t}$ for the realtime
control of the ink diffusion effects so that the chosen joint becomes the
brush that draws ink effects on the canvas, which responds to the body
movements and interaction.

$h_{t}=\arg\max_{j_{i}^{t}\in\text{hand}}||j_{t}^{i}||,$ (2)

$H_{t}=\\{h_{1},h_{2},...,h_{t}\\},$ (3)

The joint trajectory $H_{t}$ in equation (3) is a sequence that includes all
the joint positions from the beginning of the observation. Each $h_{i}$ in
$H_{t}$ represents the hand position at time $i$ out of $t$. We use $H_{t}$
for the ink stroke simulation to create the brush strokes that follow the body
movement and trajectory. Through the two real-time visual effects (i.e., ink
and stroke simulation) feeding back and looping, the artwork attracts the
audience to motivate and engage their bodily engagements for further
processing.

Figure 5: Ink Diffusion Effects (on Left) and Stroke Simulation (on Right) in
Wushu. ©Authors.

#### Ink Effects and Stroke Simulation

The temporal evolution of the ink-diffusing texture is simulated based on a
standard optical flow algorithm through a partial differential equation (PDE);
we include the technical implementation details (i.e., formula and equation)
in Appendix 01. Based on the input trajectory, we select the joints’ convex
hull points $P^{a}=\\{P_{1}^{a},P_{2}^{a},...,P_{t}^{a}\\}=\\{j_{t}^{a}\\}$
and $P^{b}=\\{P_{1}^{b},P_{2}^{b},...,P_{t}^{b}\\}=\\{j_{t}^{b}\\}$ to be the
two endpoints of lines $L_{t}^{ab}$.

For all the set of points $P^{ab}$, $\forall P_{t}^{ab}\in
L^{ab}|P_{t}^{ab}=0\lor 255$ on the line, the color of the individual point in
$P^{ab}$ is $Color(p_{t})$ denoted as equation (5). A threshold applies to the
color and determines whether it is black or white based on the return value of
the noise function. If the noise value is less than or equal to the threshold,
the color is black; otherwise, it’s white.

$L^{ab}=\\{k*P^{a}+(1-k)*P^{b},0<=k<=1\\},$ (4)

$Color(p_{t})=\begin{cases}\text{Black}&\text{if
}N(p_{x},p_{y},p_{z})\leq\text{threshold}\\\ \text{White}&\text{if
}N(p_{x},p_{y},p_{z})>\text{threshold}\end{cases}$ (5)

In expression of equation (5):

  * •

$Color(p_{t})$ represents the color of point $p$ in the point set $L^{ab}$ at
time $t$.

  * •

$N(p_{x},p_{y},p_{z})$ is the return value from the Simple3D noise function at
the coordinates $(p_{x},p_{y},p_{z})$ of point $p$.

  * •

Please note: the noise offset dynamically changes based on signal
transformations: $O_{t}$.

Figure 6: One Participant at Our Ars Electronica Campus Exhibition Interacted
with Wushu Like Performing Martial Art. ©Authors.

#### Interacion and Input

The body poses estimation and the visual effects enable multiple joints to act
as virtual brushes simultaneously. In the iterations of Wushu’s interaction
design, we have tried various behaviors as input and control. Using the hand
seems most intuitive for most audiences, but it can also be confusing if there
is no mechanism to distinguish the start and end of the simulation clearly.
Applying multiple joints as input works better with dancers or other
professionals trained with kinesthetic-relevant skills. In the open-to-public
exhibition, Wushu asks the audience to close their two wrists to start the
simulation (and then open their wrists to stop the simulation), which makes
the participant look like they are performing martial arts when interacting
with the work (see Figure 7).

Figure 7: Sample Pages of the Compendium. ©Authors.

### Participatory Generation of Calligraphy

Through real-time generated visual effects, Wushu can successfully stimulate
participants to engage with more dynamic body movements when interacting with
the work. Wushu then automatically “writes” the participant’s body movements
in the Chinese calligraphy genre by generating characters from the live images
and manipulating and arranging them as calligraphic characters and layouts.

#### Motion Caption (to Generate Characters)

We integrate two image caption (image-to-text) algorithms, BLIP and vit-gpt2,
for English and Chinese captions to generate characters (i.e., textual
descriptions) from the body movements. These two models are based on visual
language pre-training (VLP) and translate captured body movement images into
textual descriptions. The BLIP model specializes in visual language
understanding and generating tasks and can process visual data to produce
descriptive notes in English. The ViT-GPT2 model, for Chinese language
processing, combines image encoding and GPT-2 decoding using Google’s Visual
Transformer (ViT) to generate the corresponding Chinese captions.

The two image-to-text models provide semantic descriptions of the body
movement images in English and Chinese. We especially incorporate the
bilingual caption to broaden the context and make the work accessible to
audiences who do not understand Chinese. With the real-time captured images as
input, this process can be denoted as:

$T=f(I)$

Where $T$ is the set of textual descriptions corresponding to the images in
set $I$, processed by the function $f$, which encompasses the image-to-text
algorithms.

#### Characters Composition

We work with a professional calligrapher to conduct an empirical analysis to
develop a custom algorithm to manipulate and simulate character composition
and layout (see the results in Figure 8). The custom algorithm controls the
dimensions of the characters in terms of height and width, which alters the
aspect ratio of the character and results in various appearances, e.g., tall
or short, fat or slim.

The character and layout composition simulation starts with applying a
calligraphic font to the image caption texts and obtaining the calligraphic
characters, denoted as $C=\\{C_{1},C_{2},...,C_{n}\\}$. The algorithm alters
each character’s aspect ratio by function $f$, which reflects on the nuances
of the captured body movements.

It takes the recorded trajectory of the body joint as input and calculates the
bounding box to determine the ratio of the character(s). The calligraphic
presentation of character composition based on the body movements and its
textual caption can be summarized as

$M(C)=\\{f(C_{1}),f(C_{2}),...,f(C_{n})\\}$

#### Layout Composition

We use randomized grouping to generate white space for layout composition.
When initializing the canvas, the algorithm generates a new layout by grouping
the columns with calligraphic characters and inserting the white space (i.e.,
empty columns) between the groups. The pre-determined range of the grouping
and the spacing between groups ensure the calligraphic aesthetic and keep
consistency when introducing randomness (see Figure 8).

Both the grouping and white spaces are variational within a dynamic range to
increase the visual rhythm of the composition and let the ”energy” flow.
Consider a set of columns $L=\\{L_{1},L_{2},...,L_{n}\\}$. These columns are
grouped into $G=\\{G_{1},G_{2},...,G_{m}\\}$. Intra-group spacing is fixed at
$s_{intra}$, while inter-group spacing varies within
$s_{inter}\in[S_{min},S_{max}]$. In our work, the layout composition based on
the calligraphic rules can be shown as follows:

$L(G,s_{intra},s_{inter})=\bigcup_{i=1}^{m}\left(G_{i},s_{intra},s_{inter}\right)$

### The Compendium, Archive, and Installation

When interacting with Wushu, the participant generates calligraphy and
contributes to the collective efforts by working on a shared canvas with other
participants asynchronously. In this way, the current participant collaborates
with previous and future ones to complete the calligraphy piece. When the
participatory-generated calligraphic contents fully fill the canvas, Wushu
automatically archives them (i.e., saves them to the compendium). Wushu keeps
expanding with the generated calligraphy and becomes a compendium of the
audience’s collective creation, which is also the archive of all their body
movements (see examples in Figure 8).

The installation, dimension, and interface design of Wushu are flexible.
Figure 4 shows our first test installation with a big curved screen open to
the internal audience for participation. Figure 7 illustrates our (simplified)
installation setup at Ars Electronica Campus Exhibition that used a portable
4K projector as the interactive screen. Both worked smoothly; we conclude the
essence of the installation is, hardware-wise, to keep the screen larger or
identical to human scale so that a light touch of immersion could enhance the
interactive experience.

## Related Work

In the LABEL:sec:_Contextualization Section, we have reviewed and discussed
the interests of documenting and representing body movements in performance
and visual art. Many researchers and artists investigate the imaginative and
unseen relationships between body movements and visual forms through various
analytic techniques and creative practices. We discover that pioneer
calligraphers have focused on better transforming the body’s energy and
movements to calligraphy practice and pushing the boundary of the art
tradition. However, there is a lack of work on linking body movements with
calligraphy characters and archiving them.

### From Body Movements to Forms

Several visual artworks (on the internet) explore the relationship between
body movements and visual forms. For example, Tobias Gremmler, in his “Kung Fu
Motion Visualization Series (2016),” explores how “velocity transforms into
matter [12].” He displays particles whose opacity is proportional to the speed
of the performer’s body movements, while stationary body parts remain
invisible [12]. In his series of daily computational sketches, Zachary
Lieberman interprets human body movements with simple yet elegant visual forms
[20]. However, these works focus only on the relevance of body movements to
abstract visuals and do not consider the connection between body movements and
calligraphy characters, which is the main topic of our research.

### Modern Calligraphy Practices

Modern pioneer calligraphers in Eastern Asia have recently focused on large-
scale works with a single or a few characters on the canvas. To write the
huge-size characters, calligraphers need to work through full-body
engagements. Japanese artist Yuichi Inoue, “the ghost of calligraphy,” often
stands on the canvas barefoot and writes to transform the full body movements
and energy to the work [6].

Similarly, Chinese contemporary calligrapher Dongling Wang features huge-size
characters and wild cursive calligraphy. He summarizes that the whole-body
writing practice relies on “the strength of waist, endurance, agile body, and
its movements, as well as the overall balance of them [16].” In the
advancement of calligraphy art practice, exploring body movements is still
avant-garde.

### Interactions with (Calligraphic) Characters

Other interactive artworks investigate the relationship between body movement
and characters and calligraphy. For example, Google Creative Lab collaborates
with Bill Jones to design interactions based on body movements. The experiment
invites users to explore their body movements and connect with Bill Jones’
iconic dance, using the PoseNet machine learning model [17] to interact with
the speech-to-text characters.

Bryan Chung, in his “Movement in Time, Part 2 (2016),” applies computer vision
techniques to speculate the transformation of body movements in martial arts
to cursive calligraphy [8]. The body movements are mapped to a given thousand
characters through machine-learning clustering algorithms [8]. These works
create real-time experiences and connect body movements to calligraphic
characters but do not present them as calligraphy or archive them, which
identifies the research gap for our work.

## Conclusion and Discussion

On archiving body movements, we experiment with Wushu – an interactive artwork
that generates calligraphy from body movements. We invite the audience to
participate as the co-creator of the “Book,” a digital compendium of the
collective calligraphic works and an archive of the body movements. Our
approach establishes a new calligraphy creation process accessible to
individuals regardless of their prior knowledge of the traditional art form.

Wushu also serves as a virtual platform to review and appreciate the audience-
generated works; the audience assumes the role of both writers and readers in
the context of the “Book.” Creating (“Writing”) and appreciating (“Reading”)
the generated calligraphy becomes a cyclical process within this infinite
“Book.” Wushu can stimulate further attention and discussions concerning
Chinese characters and calligraphy, such as their visual forms, linguistic
meanings, and cultural essence.

## Limitations and Future Work

However, Wushu has its limitations, challenges, and opportunities that we
would like to address in the future. The compendium is an essential component
of Wushu’s concept and the archive. The audience finds it intriguing that
their collective efforts will form an actual “book.” However, we have not yet
figured out the best format to present the archive, except for the real-time
showcasing as part of Wushu’s interactive experiences. We currently save the
digital pages as image files but are considering an online gallery that
updates in real-time and publishes the archive publicly online or a physical
book in print that mimics the reading experience. We will further explore and
investigate how to present the archive in the future better.

During the exhibition of Wushu, we notice a significant difference in the
performance of the English and Chinese caption models. The English caption
model, which we acquired from a public online source, generated more accurate
and fluent captions than the Chinese caption model, which we obtained from a
different source. We attributed this difference not to the lack of AI research
on Chinese captioning but to the scarcity and quality of the data sources. The
large models that we used for captioning were trained with semantic
information from various online sources, but the predominant source of the
Chinese data was taobao.com, which had lower quality and diversity than the
English data. Although this manuscript and research do not address the data
and AI bias issues, we intend to explore them further.

## References

  * Barrass [2002] Barrass, G. S.  2002\.  The art of calligraphy in modern China.  Univ of California Press.
  * Belden-Adams [2011] Belden-Adams, K.  2011\.  Harold edgerton and complications of the’photographic instant.’.  Frame (2160-5114) 1(1).
  * Benesh [1955] Benesh, R.  1955\.  Benesh Movement Notation.  Institute of Choreology.
  * Birdwhistell [1952] Birdwhistell, R. L.  1952\.  Introduction to kinesics:(An annotation system for analysis of body motion and gesture).  Department of State, Foreign Service Institute.
  * Birdwhistell [2010] Birdwhistell, R. L.  2010\.  Kinesics and context: Essays on body motion communication.  University of Pennsylvania press.
  * Bogdanova-Kummer [2020] Bogdanova-Kummer, E.  2020\.  Bokujinkai: Japanese Calligraphy and the Postwar Avant-Garde, volume 19.  Brill.
  * Braun [1992] Braun, M.  1992\.  Picturing time: the work of Etienne-Jules Marey (1830-1904).  University of Chicago Press.
  * [8] Bryan, Chung.  Movement in time, part 2, 2022 version (2022).
  * Chiang [1973] Chiang, Y.  1973\.  Chinese calligraphy: An introduction to its aesthetic and technique, volume 60.  Harvard University Press.
  * Colbert [2018] Colbert, E. A.  2018\.  Umberto Boccioni’s Unique Forms of Continuity in Space: How materiality obfuscates originality, intent, and ethics.  University of California, Irvine.
  * Da-Wei [2012] Da-Wei, K.  2012\.  Chinese Brushwork in Calligraphy and Painting: its history, aesthetics, and techniques.  Courier Corporation.
  * [12] Diezmann, T., and Gremmler, T.  Grids for the Dynamic Image.  Bloomsbury Academic.  Google-Books-ID: e5KYPwAACAAJ.
  * Duchamp and Schwarz [1965] Duchamp, M., and Schwarz, A.  1965\.  Marcel Duchamp.  Hachette; Fabbri.
  * Groff [1995] Groff, E.  1995\.  Laban movement analysis: Charting the ineffable domain of human movement.  Journal of Physical Education, Recreation & Dance 66(2):27–30.
  * Guest [2013] Guest, A. H.  2013\.  Labanotation: the system of analyzing and recording movement.  Routledge.
  * Hertel [2016] Hertel, S.-L.  2016\.  Lines in translation: Cross-cultural encounters in modernist calligraphy, early 1980s–early 1990s.  Yishu 15(4):6–28.
  * [17] Jones, B., and GoogleCreativeLab.  Body, movement, language: AI sketches.
  * Jones [2009] Jones, E. J.  2009\.  Nama Marks and Etchings: Employing movement analysis techniques to interpret the Nama Stap.  University of London, School of Oriental and African Studies (United Kingdom).
  * Li [2010] Li, W.  2010\.  Chinese writing and calligraphy.  University of Hawaii Press.
  * [20] Liberman, Z.  Zach lieberman’s instagram daily sketches.
  * Lin and others [2015] Lin, H., et al.  2015\.  Printed as handwritten: The importance of calligraphy in printing in late ming china.  The Polish Journal of the Arts and Culture. New Series 1(1):51–76.
  * Lugaresi et al. [2019] Lugaresi, C.; Tang, J.; Nash, H.; McClanahan, C.; Uboweja, E.; Hays, M.; Zhang, F.; Chang, C.-L.; Yong, M. G.; Lee, J.; et al.  2019\.  Mediapipe: A framework for building perception pipelines.  arXiv preprint arXiv:1906.08172.
  * Marks [2011] Marks, L. U.  2011\.  Calligraphic animation: Documenting the invisible.  Animation 6(3):307–323.
  * Price [2003] Price, M. P.  2003\.  Frank and lillian gilbreth and the motion study controversy, 1907-1930.  Frank and Lillian Gilbreth: Critical evaluations in business and management 2:455.
  * Rossell [2013] Rossell, D.  2013\.  Chronophotography in the context of moving pictures.  Early Popular Visual Culture 11(1):10–27.
  * Schlombs [1998] Schlombs, A.  1998\.  Huai-su and the beginnings of wild cursive script in Chinese calligraphy, volume 75.  Franz Steiner Verlag.
  * Spier [2011] Spier, S.  2011\.  William Forsythe and the practice of choreography: it starts from any point.  Routledge.
  * Trimingham [2017] Trimingham, M.  2017\.  The Theatre of the Bauhaus: The Modern and Postmodern Stage of Oskar Schlemmer.  Routledge.
  * Vattano [2017] Vattano, S.  2017\.  Dynamic perception of plastic movements: Biomechanics and digital artifacts.  In Proceedings, volume 1, 871.  MDPI.

## Appendix 01

The optical flow algorithm and partial differential equation (PDE) formula
used for the ink diffusion simulation effects:

$\frac{\partial T_{t}}{\partial
t}=\alpha\nabla^{2}T_{t}+\beta\nabla\cdot\left(N(x,y,z,t)+O_{t}\right)\cdot(1-T_{t})$
(6)

Where $(\frac{\partial T_{t}}{\partial t})$ is the ink texture changing rate;
$(\alpha\,\beta)$ are the speed parameters control ink spreading and
absorption respectively; $(\nabla^{2}\,\nabla)$ are the laplacian and gradient
operators for diffusion simulation; $(N(x,y,z,t))$ represents Simple3D noise
with values in the range [0, 1]; $(O_{t})$ is the noise offset at time $(t)$;
and $((1-T_{t}))$ represents the remaining ink portion for simulating
absorption.