* [182] D. Hendrycks, M. Mazeika, D. Wilson, and K. Gimpel, “Using trusted data to train deep networks on labels corrupted by severe noise,” in _Advances in neural information processing systems_ , vol. 31, 2018.
  * [183] T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang, “Learning from massive noisy labeled data for image classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 2691–2699.

##  Appendix A Proof Details in Section 3

###  A.1 Proof of Theorem 1

We firstly present some necessary assumptions and lemma for our proof.

######  Assumption 1 (Regularity of marginal distribution, [159]).

Let $x\in P(x)$ be zero-mean $\mathbb{E}[x]=0$, with the covairance
$\mathbb{E}[xx^{T}]=\Sigma_{x}\succ 0$. Assume that $\Sigma_{x}^{-1/2}x$ is
$\rho^{2}$-subgaussian, and there exist constants $C\geq c=\mathcal{O}(1)$,
such that $c\mathbf{I}_{d}\preccurlyeq\Sigma_{x}\preccurlyeq C\mathbf{I}_{d}$.

######  Assumption 2 (Sufficient labeled data, [159]).

We assume that $MN\gg\rho^{4}(d-d_{dac})$.

######  Lemma 1.

Let $x\in P(x)$ be zero-mean $\mathbb{E}[x]=0$, with the covariance
$\mathbb{E}[xx^{T}]=\Sigma_{x}\succ 0$, and $\Sigma_{x}^{-1/2}x$ is
$\rho^{2}$-subgaussian. Given an i.i.d. sample of $x$,
$\mathbf{X}=[x_{11},\cdots,x_{ij},\cdots,x_{Mn}]$, for any $\delta\in(0,1)$,
if $Mn\gg\rho^{4}d$, then
$0.9\Sigma_{x}\preccurlyeq\frac{1}{Mn}\mathbf{X}^{T}\mathbf{X}\preccurlyeq
1.1\Sigma_{x}$ with high probability.

The proof of Lemma 1 can refer to Lemma 5 in [159]. Formally, we could define
the following DAC-MR operator $\mathcal{T}^{mr}$ over $\mathcal{H}$:

######  Definition 4 (DAC-MR Operator).

$\displaystyle\mathcal{T}^{mr}_{\mathcal{A},\mathbf{X}}(\mathcal{H})\triangleq\left\\{h|h\in\mathcal{H},f_{\theta^{*}_{i}(h_{\phi})}(x_{ij})=f_{\theta^{*}_{i}(h_{\phi})}(A(x_{ij})),f_{\theta^{*}_{i}}\right.$
$\displaystyle\left.\in\mathcal{F},A\in\mathcal{A},\mathbf{X}=[x_{11},\\!\cdots\\!,x_{ij},\\!\cdots\\!,x_{Mn}],\forall
i\in[M],j\in[n]\right\\}.$

We use $d_{dac}$ to quantify the strength of data augmentation $\mathcal{A}$
defined by

$\displaystyle
d_{dac}\triangleq\mathrm{rank}(A\mathbf{X}-\mathbf{X}),\text{for fixed}\
A\in\mathcal{A}.$ (14)

Our theoretical results are built upon the results of [16]. Specifically, we
instantiate our framework for one of the most frequently used classification
methods — logistic regression with
$\mathcal{X}=\\{x\in\mathbb{R}^{d}|\|x\|_{2}\leq G\\}$,
$\mathcal{Y}=\\{0,1\\}$. We follow the setting in [16], and consider the
function class

$\displaystyle\mathcal{F}=$
$\displaystyle\\{f|f(z)=\alpha^{T}z,\alpha\in\mathbb{R}^{r},\|\alpha\|\leq
c\\},$ $\displaystyle\mathcal{H}=$
$\displaystyle\\{h|h(x)=B^{T}x,B=(b_{1},\cdots,b_{r})\in\mathbb{R}^{d\times
r},$ $\displaystyle B\ \text{is a matrix with orthonormal columns}\\},$ (15)

where task-specific functions $f$s are linear maps, and the underlying meta-
representation $h$ is a projection onto a low-dimensional subspace. Such meta-
level representation learning would provide a statistical guarantee for
several importantly meta-learning scenarios [15, 16, 105, 106, 107, 108],
e.g., transfering learning, few-shot learning.

We assume that $P(y=1|f\circ h(x))=\sigma(\alpha^{T}A^{T}x)$, where
$\sigma(\cdot)$ is the sigmoid function with $\sigma(z)=1/(1+\exp(-z))$. We
use the logistic loss $\ell(z,y)=-y\log(\sigma(z))-(1-y)\log(1-\sigma(z))$ for
$\mathcal{L}^{train}$ and $\mathcal{L}^{meta}$. For the instantiation in
Eq.(15), [16] recently has theoretically proved that meta-level error bound
with respect to meta-model scales as $C(\mathcal{H})+MC(\mathcal{F})$, where
$C(\cdot)$ captures the complexity of function class, and $M$ denotes some
coefficients independent of model complexity.

To demonstrate that our additional DAC-MR objective brings better meta-level
generalization than purely meta-data-driven meta-objective, we just need to
illustrate that the complexity of
$\mathcal{T}^{mr}_{\mathcal{A},\mathbf{X}}(\mathcal{H})$ is smaller than
$\mathcal{H}$. Here we use the Gaussian complexity [160] to measure the
complexity of a function class. The following theorem shows the complexities
of $\mathcal{T}^{mr}_{\mathcal{A},\mathbf{X}}(\mathcal{H})$ and $\mathcal{H}$.

######  Theorem 4 (Formal restatement of Theorem 1 in Section 3.5).

Considering the setting in Eq.(15), if Assumptions 1, 2 and the conditions in
Lemma 1 hold, then the complexity of
$\mathcal{T}^{mr}_{\mathcal{A},\mathbf{X}}(\mathcal{H})$ and $\mathcal{H}$
satisfy

$\displaystyle
C(\mathcal{T}^{mr}_{\mathcal{A},\mathbf{X}}(\mathcal{H}))\lesssim\sqrt{\frac{(d-d_{dac})r^{2}}{Mn}},C(\mathcal{H})\lesssim\sqrt{\frac{dr^{2}}{Mn}},$
(16)

where $d_{dac}$ is defined in Eq.(14).

###### Proof.

We firstly calculate the Gaussian complexity of $\mathcal{H}$ as follows:

$\displaystyle\hat{\mathcal{G}}_{\mathbf{X}}(\mathcal{H})$
$\displaystyle=\frac{1}{Mn}\mathbb{E}\left[\sup_{B}\sum_{i=1}^{M}\sum_{j=1}^{n}\sum_{k=1}^{r}g_{kij}b_{k}^{T}x_{ij}\right]=\frac{1}{Mn}\mathbb{E}\left[\sup_{(b_{1},\cdots,b_{r})\in\mathcal{H}}\sum_{k=1}^{r}b_{k}^{T}\left(\sum_{i=1}^{M}\sum_{j=1}^{n}g_{kij}x_{ij}\right)\right]$
$\displaystyle\leq\frac{1}{Mn}\sum_{k=1}^{r}\mathbb{E}\left[\left\|\sum_{i=1}^{M}\sum_{j=1}^{n}g_{kij}x_{ij}\right\|\right]\leq\frac{1}{Mn}\sum_{k=1}^{r}\sqrt{\mathbb{E}\left[\left\|\sum_{i=1}^{M}\sum_{j=1}^{n}g_{kij}x_{ij}\right\|^{2}\right]}$
$\displaystyle\leq\frac{1}{Mn}\sum_{k=1}^{r}\sqrt{\sum_{i=1}^{M}\sum_{j=1}^{n}\left\|x_{ij}\right\|^{2}}=\frac{r}{\sqrt{Mn}}\sqrt{\mathrm{tr}\left(\frac{1}{Mn}\mathbf{X}\mathbf{X}^{T}\right)},$

where $\mathbf{X}=[x_{11},\\!\cdots\\!,x_{ij},\\!\cdots\\!,x_{Mn}]$, and the
first and second inequalities hold by Jensen inequality; and the third
inequality holds by Cauchy-Schwarz inequality. Furthermore, we take
expectation over $\mathbf{X}$ to obtain the population Gaussian complexity as
below:

$\displaystyle\mathcal{G}_{Mn}(\mathcal{H})\leq\frac{r}{\sqrt{Mn}}\mathbb{E}\left[\sqrt{\mathrm{tr}\left(\frac{1}{Mn}\mathbf{X}\mathbf{X}^{T}\right)}\right]\lesssim\sqrt{\frac{dr^{2}}{Mn}},$

where the last inequality is obtained by Theorem 4 in [16].

We then calculate the Gaussian complexity of
$\mathcal{T}^{mr}_{\mathcal{A},\mathcal{X}}(\mathcal{H})$. DAC-MR can be
rewritten as the following constraints:

$\displaystyle\alpha^{T}B^{T}(\mathcal{A}(\mathbf{X})-\mathbf{X})=0.$ (17)

Without loss of generality, Eq.(17) can be written as

$\displaystyle B^{T}(\mathcal{A}(\mathbf{X})-\mathbf{X})=\mathbf{0}.$

i.e.,

$\displaystyle(\mathcal{A}(\mathbf{X})-\mathbf{X})b_{k}=0,k\in[r].$

Let $\Delta=\mathcal{A}(\mathbf{X})-\mathbf{X}$, and then we have

$\displaystyle\mathcal{T}^{mr}_{\mathcal{A},\mathcal{X}}(\mathcal{H})\triangleq$
$\displaystyle\left\\{h(x)=B^{T}x|b_{k}\in\mathrm{Null}(\Delta),k\in[r]\right\\}.$

Denote $\mathbf{P}_{\Delta}$ as the orthogonal projector onto $\Delta$,
$\mathbf{P}^{\perp}_{\Delta}\triangleq\mathbf{I}_{d}-\mathbf{P}_{\Delta}$ as
the orthogonal complement of $\mathbf{P}_{\Delta}$, and we are ready to bound
the Gaussian complexity of
$\mathcal{T}^{mr}_{\mathcal{A},\mathcal{X}}(\mathcal{H})$ as follows:

$\displaystyle\hat{\mathcal{G}}_{\mathbf{X}}(\mathcal{T}^{mr}_{\mathcal{A},\mathcal{X}}(\mathcal{H}))=$
$\displaystyle\frac{1}{Mn}\mathbb{E}\left[\sup_{B\in\mathcal{T}^{mr}_{\mathcal{A},\mathcal{X}}(\mathcal{H})}\sum_{i=1}^{M}\sum_{j=1}^{n}\sum_{k=1}^{r}g_{kij}b_{k}^{T}x_{ij}\right]=\frac{1}{Mn}\mathbb{E}\left[\sup_{(b_{1},\cdots,b_{r})\in\mathcal{H}}\sum_{k=1}^{r}b_{k}^{T}\left(\sum_{i=1}^{M}\sum_{j=1}^{n}g_{kij}\mathbf{P}^{\perp}_{\Delta}x_{ij}\right)\right]$
$\displaystyle\leq$
$\displaystyle\frac{1}{Mn}\sum_{k=1}^{r}\mathbb{E}\left[\left\|\sum_{i=1}^{M}\sum_{j=1}^{n}g_{kij}\mathbf{P}^{\perp}_{\Delta}x_{ij}\right\|\right]\leq\frac{1}{Mn}\sum_{k=1}^{r}\sqrt{\mathbb{E}\left[\left\|\sum_{i=1}^{M}\sum_{j=1}^{n}g_{kij}\mathbf{P}^{\perp}_{\Delta}x_{ij}\right\|^{2}\right]}$
$\displaystyle\leq$
$\displaystyle\frac{1}{Mn}\sum_{k=1}^{r}\sqrt{\sum_{i=1}^{M}\sum_{j=1}^{n}\left\|\mathbf{P}^{\perp}_{\Delta}x_{ij}\right\|^{2}}=\frac{r}{\sqrt{Mn}}\sqrt{\mathrm{tr}\left(\frac{1}{Mn}\mathbf{P}^{\perp}_{\Delta}\mathbf{X}^{T}\mathbf{X}\mathbf{P}^{\perp}_{\Delta}\right)}.$

Taking expectation over $\mathbf{X}$, we can obtain that the population
Gaussian complexity is satisfied with:

$\displaystyle\mathcal{G}_{Mn}(\mathcal{T}^{mr}_{\mathcal{A},\mathcal{X}}(\mathcal{H}))\leq\frac{r}{\sqrt{Mn}}\mathbb{E}\left[\sqrt{\mathrm{tr}\left(\frac{1}{Mn}\mathbf{P}^{\perp}_{\Delta}\mathbf{X}^{T}\mathbf{X}\mathbf{P}^{\perp}_{\Delta}\right)}\right]\leq\frac{r}{\sqrt{Mn}}\sqrt{\mathbb{E}\left[\mathrm{tr}\left(\frac{1}{Mn}\mathbf{P}^{\perp}_{\Delta}\mathbf{X}^{T}\mathbf{X}\mathbf{P}^{\perp}_{\Delta}\right)\right]}\triangleq\frac{rC_{\mathcal{A}}}{\sqrt{Mn}}.$

Under the Assumptions 1, 2, and Lemma 1, we have

$\displaystyle
C_{\mathcal{A}}\leq\sqrt{d-d_{dac}}\left\|\frac{1}{Mn}\mathbf{P}^{\perp}_{\Delta}\mathbf{X}^{T}\mathbf{X}\mathbf{P}^{\perp}_{\Delta}\right\|_{2}\leq\sqrt{d-d_{dac}}\left\|\frac{1}{Mn}\mathbf{X}^{T}\mathbf{X}\right\|_{2}\leq
1.1C\sqrt{d-d_{dac}}\lesssim\sqrt{d-d_{dac}},$

and thus we have

$\displaystyle\mathcal{G}_{Mn}(\mathcal{T}^{mr}_{\mathcal{A},\mathcal{X}}(\mathcal{H}))\lesssim\sqrt{\frac{(d-d_{dac})r^{2}}{Mn}}.$

∎

###  A.2 Proof of Theorem 2 & 3

Firstly, we introduce some additional basic notations and definitions. For an
arbitrary set $\mathcal{X}$, let $\mathcal{Y}=[K]$, and
$g^{*}:\mathcal{X}\rightarrow[K]$ be the ground truth classifier that
partition $\mathcal{X}$: for each $k\in[K]$, let
$\mathcal{X}_{k}\triangleq\\{x\in\mathcal{X}|g^{*}(x)=k\\}$, with
$\mathcal{X}_{i}\cap\mathcal{X}_{j}=\emptyset,\forall i\neq j$. In addition,
for an arbitrary classifier $g:\mathcal{X}\rightarrow[K]$, we denote the
majority label with respect to $g$ for each class,

$\displaystyle\hat{y}_{k}\triangleq\mathop{\arg\max}_{y\in[K]}P(g(x)=y|x\in\mathcal{X}_{k}),\forall
k\in[K],$

and the class-wise and global minority sets

$\displaystyle
M_{k}\triangleq\\{x\in\mathcal{X}_{k}|g(x)\neq\hat{y}_{k}\\},\forall k\in[K],\
M\triangleq\bigcup_{k=1}^{K}M_{k}.$

To capture the connectivity of the data distribution, we further introduce the
expansion propety [118, 117] on the mixed distribution $S\cup Q$ as below:

######  Definition 5 (Constant Expansion).

We say that the distribution $S\cup Q$ satisfies $(q,\xi)$-constant expansion
for some constant $q,\xi\in(0,1)$, if for any $V\subset S\cup Q$ with
$P(V)\geq q$ and $P(V\cap(S_{k}\cup Q_{k}))\leq 1/2$ for any $k\in[K]$, we
have $P((\mathcal{N}(V)\setminus V)\cap(S_{k}\cup
Q_{k}))\geq\min\\{P(V\cap(S_{k}\cup Q_{k})),\xi\\}$.

######  Definition 6 (Multiplicative Expansion).

We say that the distribution $S\cup Q$ satisfies $(a,c)$-multiplicative
expansion for some constant $a\in(0,1),c>1$, if for any $k\in[K]$ and
$V\subset S\cup Q$ with $P(V\cap(S_{k}\cup Q_{k}))\leq a$, we have
$P(\mathcal{N}(V)\cap(S_{k}\cup Q_{k}))\geq\min\\{c\cdot P(V\cap(S_{k}\cup
Q_{k})),1\\}$.

This expansion property lower bounds the neighborhood size of low probability
sets, and the parameters $(q,\xi)$ or $(a,c)$ quantify the augmentation
strength of $\mathcal{A}$. Specifically, the strength of expansion-based data
augmentations is characterized by expansion capability of $\mathcal{A}$: for a
neighborhood $V\subset\mathcal{X}$ of proper size (characterized by $q$ or $a$
under measure $P$), the stronger augmentation $A$ leads to more expansion in
$\mathcal{N}(S)$, and therefore larger $\xi$ or $c$. The following proposition
builds a bridge between two expansions.

######  Proposition 2 (Lemma C.6 in [118]).

Suppose that the distribution $S\cup Q$ satisfies $(1/2,c)$-multiplicative
expansion on $\mathcal{X}$. Then for any choice of $\xi>0$, $S\cup Q$
satisfies $(\frac{\xi}{c-1},\xi)$-constant expansion.

To establish the relatinship between the expected meta-loss
$\epsilon^{Q}(g)=\mathbb{E}_{Q}\mathbf{1}[g(x)\neq g^{*}(x)]$ and the expected
DAC-MR $R_{Q}^{\mathcal{B}}(g)=\mathbb{E}_{Q}[\mathbf{1}(\exists
x^{\prime}\in\mathcal{B}(x),\text{s.t.},g(x^{\prime})\neq g(x))]$, we present
some necessary assumption and lemmas as below.

######  Assumption 2.

Assume that the task training and meta data distributions have the following
structures: $supp(S)=\cup_{k=1}^{K}S_{k}$, $supp(Q)=\cup_{k=1}^{K}Q_{k}$, and
$S_{i}\cap S_{j}=Q_{i}\cap Q_{j}=\emptyset,\forall i\neq j$. We further assume
that the ground truth class $g^{*}(x)$ for $x\in S_{k}\cup Q_{k}$ is
consistent, which is denoted as $y_{k}\in[K]$. Additionally, suppose that
there exists a constant $\kappa\geq 1$, for all $i\in[K]$, such that

$\displaystyle P(Q_{i}\cap A)\leq\kappa P\left(A\cap\frac{1}{2}\left(S_{i}\cup
Q_{i}\right)\right),\forall A\subset\mathcal{S\cup Q}.$

######  Lemma 1 (Robustness on Sub-Populations with Constant Expansion).

Suppose that $Q$ satisfies $(q,\xi)$-constant expansion, and we divide $[K]$
into two partitions $S_{1}$ and $S_{2}$, where for every $i\in S_{1}$,
$\mathbb{E}_{Q_{i}}\mathbf{1}[\exists x^{\prime}\in\mathcal{N}(x),g(x)\neq
g(x^{\prime})]\leq\min\\{q,\xi\\}$, and for every $i\in S_{2}$,
$\mathbb{E}_{Q_{i}}\mathbf{1}[\exists x^{\prime}\in\mathcal{N}(x),g(x)\neq
g(x^{\prime})]\geq\min\\{q,\xi\\}$. Under such partition, we then have

$\displaystyle\sum_{k\in S_{1}}P(Q_{k}\cap Q)\geq
1-\frac{R_{Q}^{\mathcal{B}}(g)}{\min\\{q,\xi\\}}.$

###### Proof.

Suppose $\sum_{k\in S_{1}}P(Q_{k}\cap
Q)<1-\frac{R_{Q}^{\mathcal{B}}(g)}{\min\\{q,\xi\\}}$. Then we have $\sum_{k\in
S_{2}}P(Q_{k}\cap Q)>\frac{R_{Q}^{\mathcal{B}}(g)}{\min\\{q,\xi\\}}$, which
implies:

$\displaystyle\mathbb{E}_{Q}\mathbf{1}[\exists
x^{\prime}\in\mathcal{N}(x),g(x)\neq g(x^{\prime})]$
$\displaystyle=\sum_{k\in[K]}\mathbb{E}_{Q_{k}}\mathbf{1}[\exists
x^{\prime}\in\mathcal{N}(x),g(x)\neq g(x^{\prime})]P(Q_{k}\cap Q)$
$\displaystyle\geq\sum_{k\in S_{2}}\mathbb{E}_{Q_{k}}\mathbf{1}[\exists
x^{\prime}\in\mathcal{N}(x),g(x)\neq g(x^{\prime})]P(Q_{k}\cap Q)$
$\displaystyle>\min\\{q,\xi\\}\sum_{k\in S_{2}}P(Q_{k}\cap Q)$
$\displaystyle=R_{Q}^{\mathcal{B}}(g).$

This leads to $R_{Q}^{\mathcal{B}}(g)<\mathbb{E}_{Q}\mathbf{1}[\exists
x^{\prime}\in\mathcal{N}(x),g(x)\neq g(x^{\prime})]$, which induces a
contradiction. ∎

######  Lemma 2 (Robustness on Sub-Populations with Multiplicative Expansion).

Suppose that $Q$ satisfies $(1/2,c)$-multiplicative expansion, and we divide
$[K]$ into two partitions $S_{1}$ and $S_{2}$, where for every $i\in S_{1}$,
$\mathbb{E}_{Q_{i}}\mathbf{1}[\exists x^{\prime}\in\mathcal{N}(x),g(x)\neq
g(x^{\prime})]\leq\min\\{\frac{\xi}{c-1},\xi\\}$, and for every $i\in S_{2}$,
$\mathbb{E}_{Q_{i}}\mathbf{1}[\exists x^{\prime}\in\mathcal{N}(x),g(x)\neq
g(x^{\prime})]\geq\min\\{\frac{\xi}{c-1},\xi\\}$. Under such partition, we
then have

$\displaystyle\sum_{k\in S_{1}}P(Q_{k}\cap Q)\geq
1-\frac{R_{Q}^{\mathcal{B}}(g)}{\min\\{\frac{\xi}{c-1},\xi\\}}.$

Based on Proposition 2, Lemma 2 can be obtained from Lemma 1 by plugging in
$q=\frac{\xi}{c-1}$.

######  Lemma 3 (Accuracy propagates on Sub-Populations with Constant
Expansion).

Supposed that $Q$ satisfies $(q,\xi)$-constant expansion, and the
subpopulation $Q_{k}$ satisfies $R_{Q_{k}}^{\mathcal{B}}(g)<\min\\{q,\xi\\}$,
and we then have

$\displaystyle\mathbb{E}_{Q_{k}}\mathbf{1}[g(x)\neq\hat{y}_{k}]\geq 1-q.$

###### Proof.

We claim that $\mathbb{E}_{Q_{k}}\mathbf{1}[g(x)\neq\hat{y}_{k}]\geq 1-q$.
Suppose $\mathbb{E}_{Q_{k}}\mathbf{1}[g(x)\neq\hat{y}_{k}]<1-q$. Let
$A=\\{x|g(x)=\hat{y}_{k}\\}$, and then we have $P(A\cap Q_{k})>q$. By the
$(q,\xi)$-constant expansion property, $P(\mathcal{N}(A)\setminus A\cap
Q_{k})<\min\\{q,\xi\\}$. Observing that for $x$ in $\mathcal{N}(A)\setminus
A$, we have $g(x)\neq\hat{y}_{k}$. Therefore, for $x$ in
$\mathcal{N}(A)\setminus A$, there exists $x^{\prime}\in\mathcal{N}(x)$, such
that $\hat{y}_{k}=g(x^{\prime})\neq g(x)$. Note that

$\displaystyle R_{Q}^{\mathcal{B}}(g)$
$\displaystyle=\mathbb{E}_{Q}[\mathbf{1}(\exists
x^{\prime}\in\mathcal{B}(x),\text{s.t.},g(x^{\prime})\neq g(x))]$
$\displaystyle\geq\mathbb{E}_{Q}[\mathbf{1}(\exists
x^{\prime}\in\mathcal{B}(x),\text{s.t.},g(x^{\prime})\neq
g(x))]\mathbf{1}[x\in\mathcal{N}(A)\setminus A]$
$\displaystyle=P((\mathcal{N}(A)\setminus A)\cap Q)>\min\\{q,\xi\\},$

which contradicts the condition that
$R_{Q}^{\mathcal{B}}(g)\leq\min\\{q,\xi\\}$. ∎

Based on Proposition 2, we can easily obtain the following Lemma under the
multiplicative expansion assumption.

######  Lemma 4 (Accuracy propagation on Sub-Populations with Multiplicative
Expansion).

Supposed that $Q$ satisfies $(1/2,c)$-multiplicative expansion, and the
subpopulation $Q_{k}$ satisfies
$R_{Q_{k}}^{\mathcal{B}}(g)<\min\\{\frac{\xi}{c-1},\xi\\}$, and then we have

$\displaystyle\mathbb{E}_{Q_{k}}\mathbf{1}[g(x)\neq\hat{y}_{k}]\geq
1-\frac{\xi}{c-1}.$

######  Lemma 5 (Upper Bound of Minority Set [117], Lemma A.1).

Under the Assumption 2, $P(M\cap\frac{1}{2}(S+Q))$ can be bounded as follows:

(a) Under $(\frac{1}{2},c)$-multiplicative expansion, we have
$P(M\cap\frac{1}{2}(S+Q))\leq\max\left(\frac{c+1}{c-1},3\right)R_{Q}^{\mathcal{B}}(g)$;

(b) Under $(q,\xi)$–constant expansion, we have $P(M\cap\frac{1}{2}(S+Q))\leq
2\max(q,R_{Q}^{\mathcal{B}}(g))+R_{Q}^{\mathcal{B}}(g)$.

######  Theorem 5 (Bounding the Meta Error with Constant Expansion).

Suppose that Assumption 2 holds and $\frac{1}{2}(S+Q)$ satisfies
$(q,\xi)$-constant expansion, and then we have

$\displaystyle\begin{split}\epsilon^{Q}(g)\leq&\left\\{\frac{\kappa}{1-q}+\frac{1}{\min\\{q,\xi\\}}\right\\}R_{Q}^{\mathcal{B}}(g)\\\
&+\frac{2\kappa}{1-q}\max(q,R_{Q}^{\mathcal{B}}(g)).\end{split}$

###### Proof.

Suppose that $Q$ satisfies $(q,\xi)$-constant expansion, and we divide $[K]$
into two partitions $S_{1}$ and $S_{2}$, where for every $i\in S_{1}$,
$\mathbb{E}_{Q_{i}}\mathbf{1}[\exists x^{\prime}\in\mathcal{N}(x),g(x)\neq
g(x^{\prime})]\leq\min\\{q,\xi\\}$, and for every $i\in S_{2}$,
$\mathbb{E}_{Q_{i}}\mathbf{1}[\exists x^{\prime}\in\mathcal{N}(x),g(x)\neq
g(x^{\prime})]\geq\min\\{q,\xi\\}$. Then we have

$\displaystyle\epsilon^{Q}(g)$
$\displaystyle=\sum_{k=1}^{K}\epsilon_{k}^{Q}(g)P(Q_{k}\cap Q)$
$\displaystyle=\sum_{k\in[S_{1}]}\epsilon_{k}^{Q}(g)P(Q_{k}\cap
Q)+\sum_{k\in[S_{2}]}\epsilon_{k}^{Q}(g)P(Q_{k}\cap Q)$
$\displaystyle\leq\sum_{k\in[S_{1}]}\epsilon_{k}^{Q}(g)P(Q_{k}\cap
Q)+\sum_{k\in[S_{2}]}P(Q_{k}\cap Q).$

(1) For $k\in S_{1}$, we consider the following two cases:

(a) If $y_{k}=\hat{y}_{k}$, we have

$\displaystyle\epsilon_{k}^{Q}(g)=P(M_{k}\cap Q)\leq\kappa
P(M_{k}\cap\frac{1}{2}(S+Q)).$

(b) If $y_{k}\neq\hat{y}_{k}$, according to Lemma 3, we have

$\displaystyle\frac{P(M_{k}\cap Q)}{P(Q_{k}\cap Q)}=P(M_{k}\cap
Q_{k})=\mathbb{E}_{Q_{k}}\mathbf{1}[g(x)\neq\hat{y}_{k}]\geq 1-q.$

Then we have

$\displaystyle\epsilon_{k}^{Q}(g)$ $\displaystyle\leq P(Q_{k}\cap
Q)\leq\frac{P(M_{k}\cap Q)}{1-q}$
$\displaystyle\leq\frac{\kappa}{1-q}P(M_{k}\cap\frac{1}{2}(S+Q)).$

Combining the two cases (a) and (b), we always have

$\displaystyle\epsilon_{k}^{Q}(g)\leq\frac{\kappa}{1-q}P(M_{k}\cap\frac{1}{2}(S+Q)),$

and thus

$\displaystyle\sum_{k\in[S_{1}]}\epsilon_{k}^{Q}(g)P(Q_{k}\cap Q)$
$\displaystyle\leq\sum_{k\in[S_{1}]}\epsilon_{k}^{Q}(g)$
$\displaystyle\leq\sum_{k\in[S_{1}]}\frac{\kappa}{1-q}P(M_{k}\cap\frac{1}{2}(S+Q))$
$\displaystyle\leq\frac{\kappa}{1-q}P(M\cap\frac{1}{2}(S+Q))$
$\displaystyle=\frac{\kappa}{1-q}\left(2\max(q,R_{Q}^{\mathcal{B}}(g))+R_{Q}^{\mathcal{B}}(g)\right),$

where the last equality holds based on Lemma 5(b).

(2) For $k\in S_{2}$, according to Lemma 2, we have

$\displaystyle\sum_{k\in[S_{2}]}P(Q_{k}\cap
Q)\leq\frac{R_{Q}^{\mathcal{B}}(g)}{\min\\{q,\xi\\}}.$

Therefore,

$\displaystyle\epsilon^{Q}(g)$
$\displaystyle\leq\frac{\kappa}{1-q}\left(2\max(q,R_{Q}^{\mathcal{B}}(g))+R_{Q}^{\mathcal{B}}(g)\right)+\frac{R_{Q}^{\mathcal{B}}(g)}{\min\\{q,\xi\\}}$
$\displaystyle=\left\\{\frac{\kappa}{1-q}+\frac{1}{\min\\{q,\xi\\}}\right\\}R_{Q}^{\mathcal{B}}(g)+\frac{2\kappa}{1-q}\max(q,R_{Q}^{\mathcal{B}}(g)).$

∎

We can easily obtain the following Theorem 6 by plugging in
$q=\frac{\xi}{c-1}$ in Theorem 5.

######  Theorem 6 (Bounding the Meta Error with Multiplicative Expansion).

Suppose that Assumption 2 holds and $\frac{1}{2}(S+Q)$ satisfies
$(q,\xi)$-multiplicative expansion, and then we have

$\displaystyle\epsilon^{Q}(g)\leq\left\\{\max\left(\frac{\kappa(c+1)}{c-1-\xi},3(c-1)\right)+\frac{1}{\min(\frac{\xi}{c-1},\xi)}\right\\}R_{Q}^{\mathcal{B}}(g).$

##  Appendix B More Experimental Setting details in Section 4

###  B.1 Inductive Few-Shot Learning

Datasets. We adopt two typical few-shot image classification benchmarks. The
miniImageNet dataset [19] consists of 100 randomly chosen classes from
ImageNet [161]. The meta-training, meta-validation, and meta-testing sets
contain 64, 16 and 20 classes randomly split from 100 classes, respectively.
Each class contains 600 images of size $84\times 84$. We use the commonly-used
split proposed by [47]. The CIFAR-FS dataset [109] consists of all 100 classes
from CIFAR-100. The classes are randomly split into 64, 16 and 20 for meta-
training, meta-validation, and meta-testing, respectively. Each class contains
600 images of size $32\times 32$.

Experimental setup. We identically follow the practice in [43] for fair
comparison. We uses a ResNet-12 backbone as meta-model to achieve better
performance as suggested in [43]. We also follow the regularization tricks
such as DropBlock [162] to avoid the overfitting risk. We use SGD with a
Nesterov momentum 0.9 and a weight decay 0.0005. Each mini-batch consists of 8
episodes. The model was meta-trained for 60 epochs, with each epoch consisting
of 1000 episodes. The learning rate was initially set to 0.1, and then changed
to 0.006, 0.0012, and 0.00024 at epochs 20, 40 and 50, respectively. We adopt
horizontal flip, random crop, and color (brightness, contrast, and saturation)
jitter data augmentation techniques. We use 5-way classification in both meta-
training and meta-test stages. Each class contains 6 query samples during
meta-training and 15 test samples during meta-testing. Our meta-trained model
was chosen based on 5-way 5-shot test accuracy on the meta-validation set.
Meanwhile, we set training shot to 15 for miniImageNet and 5 for CIFAR-FS. We
set $\gamma=\lambda=1$ in Eq.(4) of the main paper.

###  B.2 Cross-Domain Few-Shot Learning

Datasets. We use the Meta-Dataset [136] to evaluate our method, which is the
standard benchmark for FSL. It contains images from 13 diverse datasets and we
follow the standard protocol in [136], and see [136] for details.

Baselines. We identically follow the baselines in [136] for fair comparison,
and see [136] for details.

Experimental setup. We use ResNet-18 to train a multi-domain feature extractor
over eight training subdatasets by following [116] with the same
hyperparameters in our experiments. We adopt horizontal flip, random crop, and
color (brightness, contrast and saturation) jitter data augmentation
techniques. To finetune the feature extrator with DAC-MR, we use SGD with
Nesterov momentum 0.9, a learning rate 0.001 and a weight decay 0.0005 on the
support samples in meta-test stage. For learning task-specific weights,
including the pre-classifier transformation and the adapter parameters, we
directly attach them to the task-agnostic weights and learn them on the
support samples in meta-test by using Adadelta optimizer following [116] with
the same hyperparameters. We report the few-shot classification accuracy in
previously seen domains and unseen domains along with their average accuracy.
We also report average accuracy over all domains and the average rank.

###  B.3 Transductive / Semi-Supervised Few-Shot Learning

Datasets. We use four common few-shot claudication benchmark datasets,
miniImageNet [19], tieredImageNet [163], CUB [164] and CIFAR-FS [109]. More
details see iLPC [122].

Baselines. We identically follow the baselines in [122] for fair comparison,
including LR+ICI [120], PT+MAP [121] for semi-supervised FSL and EP [123], SIB
[124], LaplacianShot [125], PT+MAP [121] for transductive FSL.

Experimental setup. We use pre-trained weights of a WRN28-10 for
transductive/semi-supervised FSL, which are the same to those used by [121,
122, 125, 165]. The experimental setting for producing pseudo-labels on query
set is same as iLPC [122], and we fine-tune feature extractor using a SGD with
momentum 0.9, weight decay 0.0005, learning rate 0.0001 for 10 epochs. We
report mean accuracy and 95% confidence interval on the 1000 5-way $K$-shot
test tasks, $K\in\\{1,5\\}$. The query set contains 15 examples per class.

##  Appendix C More Experimental Setting details in Section 5

###  C.1 Unsupervised Domain Adaptation

Datasets. We evaluate DAC-MR over two visual object recognition datasets:
Office-Home [166] has 65 classes from four kinds of environment with large
domain gap: Artistic (Ar), Clip Art (Cl), Product (Pr), and Real-World (Rw);
VisDA-2017 [167] is a large-scale UDA dataset with two domains named Synthetic
and Real. The datasets consist of over 200k images from 12 categories of
objects.

Baselines. We identically follow the baselines in [27] for fair comparison,
including DANN [126], CDAN [127], VAT [69], FixMatch [52], MDD [128], and
SENTRY [129].

Experimental setup. We use ResNet-50 [1] (pretrained on ImageNet [161]) as
feature extractors, and we also provide results of ResNet101 for VisDA-2017 to
include more baselines. We adopt SGD with initial learning rate $2\times
10^{-3}$, and decay the learning rate exponentially until 30 epochs. We add
the DAC-MR after the 10 epochs. We run all the tasks 3 times and report mean
in top-1 accuracy. For VisDA-2017, we report the mean class accuracy. We also
use sharpness-aware regularization [168] to enhance performance following CST
[27]. For the data augmentation techniques we adopt horizontal flip, random
crop, and color (brightness, contrast, and saturation) jitter.

###  C.2 Domain Generalization

Datasets. We just follow benchmarks in ARM [144] to evaluate our DAC-MR,
including four image classification problems: Rotated MNIST, FEMNIST,
CIFAR-10-C, Tiny ImageNet-C, and the WILDS benchmark [169].

Baselines. We identically follow baselines in [144] for fair comparison,
including BN adaptation [131], TTT [132], UW [130], DRNN [130], DANN [126],
MDD [128]. We also compare CORAL [170] and IRM [171] for WILDS benchmark.

Experimental setup. We set $\lambda=1$ for all experiments, and follow the
exact experimental settings as ARM [144]. We add the DAC-MR after the 20
epochs.For the Ratated MNIST dataset we adopt random rotation data
augmentation techniques. For FEMNIST dataset we adopt random crop and blur
data augmentation techniques. For the CIFAR-10-C dataset and Tiny ImageNet-C
dataset we adopt horizontal flip, random crop, and color (brightness,
contrast, and saturation) jitter data augmentation techniques.

###  C.3 Transfer Learning with Fine-tuning

Datasets. We consider several extensively investigated transfer learning
benchmarks, consisting of CUB-200 (11, 788 images for 200 bird species) [172],
Stanford Cars (16, 185 images for 196 car categories) [173], and FGVC Aircraft
(10, 000 images for 100 aircraft variants) [174].

Baselines. We compared against several state-of-the-art fine-tuning methods:
Fine-tuning [175], $L^{2}$-SP [133], DELTA [134] and Co-Tuning [135]. The
implementation of this paper is adapted from the transfer learning library
[176, 177].

Experimental setup. We use ResNet-50 [1] (supervised pretrained or self-
supervised MoCo [178] pre-trained on ImageNet [161]) as the source model. We
optimize all models by SGD with a momentum 0.9, and learning rate for task-
specific classifier is ten times of the learning rate for pre-trained
parameters, following the common fine-tuning practice [175]. We set batch size
as 48. To explore the impact of negative transfer with different numbers of
training examples, we create four configurations for each dataset, which
respectively have 15%, 30%, 50%, and 100% randomly sampled training examples
for each category. Each experiment is repeated three times with different
random seeds to collect mean and standard deviation of the performance. We
finetune the feature shared representation function by SGD with a momentum
0.9, and the learning rate is the same as the learning rate for pre-trained
parameter. And for the target data we dynamicly use the data whose maximum
predicted probability is higher than 0.99 invovling in computing DAC-MR. we
adopt horizontal flip, random crop, and color (brightness, contrast, and
saturation) jitter for the data augmentations.

##  Appendix D More Experimental Setting details in Section 6

###  D.1 Task-Incremental Learning

DAC-MR amelioration manner for the task. Following the La-MAML [137] method,
we consider a setting where a sequence of $T$ tasks
$[\tau_{1},\tau_{2},\cdots,\tau_{T}]$ is learned by observing their training
data $[\mathcal{D}_{1},\mathcal{D}_{2},\cdots,\mathcal{D}_{T}]$ sequentially.
We define $(X^{i},Y^{i})=\\{(x_{n}^{i},y_{n}^{i})\\}_{n=0}^{N_{i}}$ as the set
of $N_{i}$ input-label pairs randomly drawn from $\mathcal{D}_{i}$. For any
time-step $j$ during online learning, we aim to minimize the empirical risk of
the model on all the $t$ tasks seen so far $(\tau_{1:t})$, given limited
access to data $(X^{i},Y^{i})$ from previous tasks $\tau_{i}(i<t)$. The
learning objective of La-MAML [137] is defined as:

$\displaystyle\min_{\theta_{0}^{j},\alpha^{j}}\sum_{\mathcal{S}_{k}^{j}\sim\mathcal{D}_{t}}\left[L_{t}(U_{k}(\alpha_{j},\theta_{0}^{j},\mathcal{S}_{k}^{j}))\right],$
(18)

where meta loss $L_{t}\\!=\\!\sum_{i=1}^{t}\\!\ell_{i}$ backpropagates
gradients with respect to the weights $\theta_{0}^{j}$ and learning rate
$\alpha^{j}$, which is evaluated on
$\theta_{k}^{j}=U_{k}(\alpha_{j},\theta_{0}^{j})$.
$U_{k}(\alpha_{j},\theta_{0}^{j})$ denotes $k$ steps of gradient descent with
learning rate $\alpha_{j}$ on the inner level task-specific loss function. The
meta loss is computed on the samples from replay-buffer indicating the
performance of parameters $\theta_{k}^{j}$ on all the tasks $\tau_{1:t}$ seen
till time $j$. Details please see La-MAML [137]. We additionally introduce
DAC-MR into Eq.(18) as supplemental meta-knowledge to produce better continual
learning performance of La-MAML [137], i.e.,

$\displaystyle\min_{\theta_{0}^{j},\alpha^{j}}\sum_{\mathcal{S}_{k}^{j}\sim\mathcal{D}_{t}}\left[L_{t}(U_{k}(\alpha_{j},\theta_{0}^{j},\mathcal{S}_{k}^{j}))+\mathcal{MR}^{dac}(\mathcal{S}_{k}^{j};U_{k}(\alpha_{j},\theta_{0}^{j}),A)\right],A\in\mathcal{A}.$

Baselines. We identically follow the baselines in La-MAML [137] for fair
comparision, including MER [55], iCaRL [138], GEM [139], AGEM [140]. We also
compare C-MAML (base algorithm of LA-MAML) and SYNC (without meta-updating
learning rate in LA-MAML).

Datasets. We conduct experiments on the CIFAR-100 dataset [179] in a task-
incremental manner where 20 tasks comprising of disjoint 5-way classification
problems are streamed. We also evaluate on the TinyImagenet-200 dataset by
partitioning its 200 classes into 40 5-way classification tasks.

Experimental setup. Following La-MAML [137], we conduct experiments in both
the Single-Pass and Multiple-Pass settings. Each method is allowed a replay-
buffer, containing upto 200 and 400 samples for CIFAR-100 and TinyImagenet
respectively. We report the retained accuracy (RA) metric and backward-
transfer and interference (BTI) value, which computes the average accuracy of
the model across tasks at the end of training and the average change in
accuracy of each task from when it was learnt to the end of the last task. We
adopt horizontal flip, random crop, and color (brightness, contrast, and
saturation) jitter for the data augmentations.

###  D.2 Few-Shot Class-Incremental Learning

DAC-MR amelioration manner for the task. Different from FSL, few-shot class-
incremental learning (FSCIL) learns training sessions in sequence. Let
$\\{\mathcal{D}_{0},\mathcal{D}_{1},\cdots,\mathcal{D}_{T}\\}$ denote the
training sets of different training sessions, and the corresponding label
space of $\mathcal{D}_{i}$ is denoted by $\mathcal{Y}_{i}$. Different sessions
have no overlapped classes, i.e.,
$\mathcal{Y}_{i}\cap\mathcal{Y}_{j}=\emptyset,\forall i\neq j$. FSCIL aims to
develop an algorithm that can sequentially train a model from all tasks to
possibly avoid the catastrophic forgetting issue, i.e., when the model is
trained on the $i$-th task it should still provide possibly accurate
predictions for all tasks $j<i$ seen in the past. In this paper, we study the
CEC [143] algorithm due to its SOTA FSCIL performance. It mainly contains
three learning stages: feature pre-training, pseudo incremental learning and
classifier learning. To ensure the classifier learning incorporates the global
context information of all individual tasks in previous sessions, CEC [143]
proposes a continually evolved classifier as shown in Algorithm 1 which
includes a classifier adaptation module $\mathcal{G}_{\theta}$ to update the
classifier weights learned on each individual session based on the global
context of previous sessions. We introduce DAC-MR into CEC as a meta-
regularizer to provide supplemental meta-knowledge information to help improve
the performance of CEC. The modification of CEC Algorithm is shown in red.

Algorithm 1 Pseudo incremental learning [143] incorporated with DAC-MR.

0: Base classes datasets $\mathcal{D}_{0}$, pre-trained model $\mathcal{R}$, a
randomly initialized GAT model $\mathcal{G}_{\theta}$.

0: A trained GAT model $\mathcal{G}_{\theta}$.

1: while not done do

2: $\\{\mathcal{S}_{b},\mathcal{Q}_{b}\\}\leftarrow$ Sample the the support
and query set for pseudo base classes from $\mathcal{D}_{0}$.

3: $\mathbf{W}_{b}\leftarrow$ Learn FC layer upon $\mathcal{R}$ with
$\mathcal{S}_{b}$.

4: $\\{\mathcal{S}_{i},\mathcal{Q}_{i}\\}\leftarrow$ Sample the the support
and query set for pseudo incremental classes from $\mathcal{D}_{0}$.

5: for class $c$ in $C_{i}$ do

6: $\gamma\leftarrow$ Randomly select angle in
$\\{90^{\circ},180^{\circ},270^{\circ}\\}$;

7: $\\{\mathcal{S}^{\prime}_{i},\mathcal{Q}^{\prime}_{i}\\}\leftarrow$ Rotate
$\\{\mathcal{S}_{i},\mathcal{Q}_{i}\\}$ from class $c$ with the selected angle
$\gamma$;

8: end for

9: $\mathbf{W}_{i}\leftarrow$ Learn FC layer upon $\mathcal{R}$ with pseudo
incremental support set $\mathcal{S}^{\prime}_{i}$ after rotation.

10: $\mathbf{W}^{\prime}_{b},\mathbf{W}^{\prime}_{i}\leftarrow$ Update
classifier $\mathbf{W}_{b},\mathbf{W}_{i}$ using $\mathcal{G}_{\theta}$.

11: $\hat{y}^{(q)}\leftarrow$ Predict labels of
$\\{\mathcal{Q}_{b},\mathcal{Q}^{\prime}_{i}\\}$ using
$[\mathcal{R},(\mathbf{W}^{\prime}_{b},\mathbf{W}^{\prime}_{i})]$.

12: loss $\leftarrow$ Compute loss with $\mathcal{L}(y^{(q)},\hat{y}^{(q)})$.

13: MR $\leftarrow$ Compute meta-regularization
$\mathcal{MR}^{dac}(\mathcal{Q}_{b};\mathbf{W}^{\prime}_{b}\circ\mathcal{R},A\in\mathcal{A})+\mathcal{MR}^{dac}(\mathcal{Q}^{\prime}_{i};\mathbf{W}^{\prime}_{i}\circ\mathcal{R},A\in\mathcal{A})$.

14: loss_all $\leftarrow$ loss $+$ MR.

15: Optimize $\mathcal{G}_{\theta}$ with SGD

16: end while

Note: $C_{i}$ is the number of classes in pseudo incremental classes;
$y^{(q)}$ and $\hat{y}^{(q)}$ indicate the ground truth label and the network
prediction, respectively. $\mathcal{L}$ and $\mathcal{MR}^{dac}$ are the
cross-entropy loss and DAC-MR, respectively.

Baselines. We compared with SOTA methods including iCaRL [138], TOPIC [141],
SPPR [142] and CEC [143]. We evaluate the model after each session with test
set and report the Top-1 accuracy and the average of all sessions. We also
include the relative improvement for the final session.

Datasets. We evaluate DAC-MR upon CEC on three popular few-shot incremental
learning benchmark datasets, including CIFAR100 [179], miniImageNet [19] and
CUB-200 [172]. We follow the experimental setting in [141].

Experimental setup. Following [141], we employ ResNet20 as the backbone for
experiments on CIFAR100 and ResNet18 for experiments on miniImageNet and
CUB200. We train the GAT model $\mathcal{G}_{\theta}$ for 5000 iterations with
the learning rate of 0.0002, and decay it by 0.5 every 1000 iteration. Random
crop, random scale, and random horizontal flip are used for data augmentation
at training time. We add the DAC-MR after the 20 epochs.

##  Appendix E More Experimental Setting details in Section 7

###  E.1 Sample Weighting Learning

####  E.1.1 Comparison with MW-Net

This subsection compares against MW-Net meta-learned with clean meta dataset
to show our novel meta-objective is capable of learning proper weighting
strategy.

Datasets. We use CIFAR-10 and CIFAR-100 [179] for comparing robust learning
methods. We identically follow MW-Net [26], and apply the symmetric and
asymmetric noise models. We randomly select different 1000 images at each
epoch from training set as $D$ to compute DAC-MR.

Experimental setup. We use ResNet-32 as the classifier network, and the
weighting network is a single layer MLP with 100 hidden nodes and ReLU
activations. All classifier networks were trained using SGD with a momentum
0.9, a weight decay $5\times 10^{-4}$ and an initial learning rate 0.1. The
learning rate is divided by 10 after 80 and 100 epochs (for a total 120
epochs). We use Adam optimizer to train MW-Net with learning rate 0.001. We
use a batch size of 100 for both the training samples and the meta ones. We
repeat experiments with three different seeds for corrupting samples with
label noise and initializing the classifier networks. We adopt horizontal
flip, random crop, and color (brightness, contrast, and saturation) jitter for
the data augmentations.

####  E.1.2 Comparison with SOTA methods.

This subsection compares with SOTA robust learning methods against both
synthetic and real-world noisy labels. To fairly compare with the these SOTA
methods, we use pseudo-labels to correct noisy labels to more sufficiently
make use of samples inspired by DivideMix [150], C2D [151] and AugDesc [146].

Specifically, we use the following novel bi-level optimization objective to
learn MW-Net:

$\displaystyle{\phi}^{*}$
$\displaystyle=\mathop{\arg\min}_{{\phi}}\mathcal{L}^{meta}(\mathcal{D}^{(q)};\mathbf{w}^{*}(\phi)),$
(19) s.t. $\displaystyle\
\mathbf{w}^{*}(\phi)=\mathop{\arg\min}_{{\mathbf{w}}}\frac{1}{m}\sum_{i=1}^{m}\left[h_{\phi}(L_{i}^{tr}(\mathbf{w}))\ell(f_{\mathbf{w}}(x_{i}^{(s)}),y_{i}^{(s)})+(1-h_{\phi}(L_{i}^{tr}(\mathbf{w})))\ell(f_{\mathbf{w}}(x_{i}^{(s)}),z_{i}^{(s)})\right],$
(20)

where
$L_{i}^{tr}(\mathbf{w})\\!=\\!{\ell}(f_{\mathbf{w}}(x_{i}^{(s)}),y_{i}^{(s)})$,
$\ell$ is the cross-entropy loss, and $z$ is pseudo-label. In our experiments,
we apply EMA [71] and temporal ensembling [70] techniques to produce pseudo-
labels in our algorithm, which has been verified to be effective in tasks like
semi-supervised learning [70, 70] and robust learning [148, 145]. And we add a
warm-up self-supervised pre-training step and impose a data augmentation based
consistency regularization as in C2D and AugDesc to boost our method. The
complete algorithm is summarized in the Algorithm 2.

Algorithm 2 Label correction with MW-Net algorithm

0: Training data $\mathcal{D}^{(s)}$, batch size $n$, temporal ensembling
momentum $\alpha\in[0,1)$, weight averaging momentum $\beta\in[0,1)$, mixup
hyperparameter $\gamma>0$, learning rates $\eta_{1},\eta_{2}$, data
augmentations $A$.

0: Classifier parameter $\mathbf{w}^{*}$

1: Initialize classifier network parameter $\mathbf{w}^{(0)}$ with a self-
supervised pre-training model. Initialize averaged predictions with noisy
labels $\mathbf{z}^{(0)}=\mathbf{\hat{y}}_{[N\times C]}$, and averaged weights
(untrainable) $\mathbf{w}_{WA}^{(0)}=\mathbf{w}^{(0)}$.

2: for $t=0$ to $T-1$ do

3: $\\{x,y\\}\leftarrow$ SampleMiniBatch($\mathcal{D}^{(s)},n$).

4: $\\{x^{meta}\\}\leftarrow$ SampleMiniBatch($\mathcal{D}^{(s)},m$).

5: Generate mixing coefficient $\lambda\sim
Beta(\gamma,\gamma),\lambda=\max(\lambda,1-\lambda)$.

6: Calculate weight averaging:
$\mathbf{w}_{WA}^{(t+1)}=\beta\mathbf{w}_{WA}^{(t)}+(1-\beta)\mathbf{w}^{(t)}$.

7: Calculate temporal ensembling:
$\mathbf{z}^{(t+1)}=\alpha\mathbf{z}_{i}^{(t)}+(1-\alpha)f(x;\mathbf{w}_{WA}^{(t+1)})$.

8: Generate new index sequence idx = torch.randperm(n).

9: Generate $\tilde{x}=\lambda^{\prime}x+(1-\lambda^{\prime})x[\text{idx}]$,
and let $\tilde{y}=y[\text{idx}],\tilde{z}^{(t+1)}={z}^{(t+1)}[\text{idx}]$.
Calculate $N_{i}$ and $\tilde{N}_{i}$, representing the numbers of samples
contained in the classes to which $x_{i}$ and $x[\text{idx}]_{i}$ belong,
respectively.

10: Formulate the learning manner of classifier network:
$\displaystyle\hat{\mathbf{w}}^{(t+1)}(\phi)=\mathbf{w}^{(t)}$
$\displaystyle-\eta_{1}\sum_{i=1}^{n}\\{\lambda\left[h({\ell}(f(\tilde{x}_{i};\mathbf{w}_{WA}^{(t)}),y_{i});\phi^{(t)})\nabla_{\mathbf{w}}{\ell}(f(\tilde{x}_{i};\mathbf{w}^{(t)}),y_{i})\Big{|}_{\mathbf{w}^{(t)}}+(1-h({\ell}(f(\tilde{x}_{i};\mathbf{w}_{WA}^{(t)}),y_{i});\phi^{(t)})\nabla_{\mathbf{w}}{\ell}(f(\tilde{x}_{i};\mathbf{w}^{(t)}),\mathbf{z}_{i}^{(t+1)})\Big{|}_{\mathbf{w}^{(t)}}\right]$
$\displaystyle+(1-\lambda)\left[h({\ell}(f(\tilde{x}_{i};\mathbf{w}_{WA}^{(t)}),\tilde{y}_{i});\phi^{(t)})\nabla_{\mathbf{w}}{\ell}(f(\tilde{x}_{i};\mathbf{w}^{(t)}),\tilde{y}_{i})\Big{|}_{\mathbf{w}^{(t)}}+(1-h({\ell}(f(\tilde{x}_{i};\mathbf{w}_{WA}^{(t)}),\tilde{y}_{i}));\phi^{(t)})\nabla_{\mathbf{w}}{\ell}(f(\tilde{x}_{i};\mathbf{w}^{(t)}),\tilde{\mathbf{z}}_{i}^{(t+1)})\Big{|}_{\mathbf{w}^{(t)}}\right]\\}.$

11: Update parameters of MW-Net $\phi^{(t+1)}$ by
$\displaystyle{\phi}^{(t+1)}={\phi}^{(t)}-\eta_{2}\frac{1}{m}\sum_{i=1}^{m}\nabla_{\phi}\mathcal{MR}^{dac}(\\{x^{meta}\\};\hat{\mathbf{w}}^{(t+1)}(\phi),A)\Big{|}_{\phi^{(t)}}.$

12: Update parameters of classifier $\mathbf{w}^{(t+1)}$ by
$\displaystyle{\mathbf{w}}^{(t+1)}=\mathbf{w}^{(t)}$
$\displaystyle-\eta_{1}\sum_{i=1}^{n}\\{\lambda\left[h({\ell}(f(\tilde{x}_{i};\mathbf{w}_{WA}^{(t)}),y_{i});\phi^{(t+1)})\nabla_{\mathbf{w}}{\ell}(f(\tilde{x}_{i};\mathbf{w}^{(t)}),y_{i})\Big{|}_{\mathbf{w}^{(t)}}+(1-h({\ell}(f(\tilde{x}_{i};\mathbf{w}_{WA}^{(t)}),y_{i});\phi^{(t+1)})\nabla_{\mathbf{w}}{\ell}(f(\tilde{x}_{i};\mathbf{w}^{(t)}),\mathbf{z}_{i}^{(t+1)})\Big{|}_{\mathbf{w}^{(t)}}\right]$
$\displaystyle+(1-\lambda)\left[h({\ell}(f(\tilde{x}_{i};\mathbf{w}_{WA}^{(t)}),\tilde{y}_{i});\phi^{(t+1)})\nabla_{\mathbf{w}}{\ell}(f(\tilde{x}_{i};\mathbf{w}^{(t)}),\tilde{y}_{i})\Big{|}_{\mathbf{w}^{(t)}}+(1-h({\ell}(f(\tilde{x}_{i};\mathbf{w}_{WA}^{(t)}),\tilde{y}_{i}));\phi^{(t+1)})\nabla_{\mathbf{w}}{\ell}(f(\tilde{x}_{i};\mathbf{w}^{(t)}),\tilde{\mathbf{z}}_{i}^{(t+1)})\Big{|}_{\mathbf{w}^{(t)}}\right]\\}.$

13: end for

Datasets. For the real-world noisy labels, we employ mini-WebVision dataset,
which contains the top 50 classes from the Google image subset of WebVision
[180]. And We randomly select different 10 images every class at each epochs
from training set as $D$ to compute DAC-MR.

Baselines. The comparison methods include: 1) ERM, 2) Forward [147], 3)
M-correction [148], 4) PENCIL [149], 5) DivideMix [150], 6) ELR+ [145], 7)
AugDesc [146], and 8) C2D [151].

Experimental setup. We adopt horizontal flip, random crop, and color
(brightness, contrast, and saturation) jitter for the data augmentations. For
CIFAR-10 and CIFAR-100, we use an 18-layer PreAct-ResNet and train it using
SGD with a momentum of 0.9, a weight decay of 0.0005, and a batch size of 128.
The network is trained for 300 epochs. We set the initial learning rate as
0.02, and reduce it by a factor of 10 after 150 epochs. For mini-Webvision, we
use ResNet-50 and train it using SGD with a momentum of 0.9, a weight decay of
0.0005, and a batch size of 64. We set the initial learning rate as 0.01 and
reduce it by a factor of 10 after 50 epochs (for a total 90 epochs). We train
MW-Net using Adam with a learning rate of 0.001, a weight decay of 0.001. To
make our results comparable to the existing literature, we introduce pseudo
labels to correct noisy labels to possibly make sufficient use of samples as
done by M-correction, PENCIL and DivideMix, and we add a warm-up self-
supervised pre-training step and impose a data augmentation based consistency
regularization as in C2D and AugDesc to boost our method.

###  E.2 Transition Matrix Estimation

DAC-MR amelioration manner for the task. We consider the label noise
generation process studied in previous works [147, 32]. Specifically, the
clean class-posterior $P(Y|X=x)\triangleq[P(Y=1|X=x),\cdots,(Y=C|X=x)]$ can be
inferred by utilizing the noisy class-posterior $P(\tilde{Y}|X=x)$ and the
transition matrix $\mathbf{T}\in[0,1]^{C\times C}$ where
$\mathbf{T}_{ij}=P(\tilde{Y}=j|Y=i)$, i.e.,

$\displaystyle p(\widetilde{Y}|X=x)=\mathbf{T}^{T}p(Y|X=x).$ (21)

This formulation has facilitated progress to some statistically consistent
robust learning methods [147, 181, 182, 32]. However, they all heavily rely on
the success of estimating transition matrices. Early attempts develop to
estimate the transition matrices under the anchor-point assumption [147, 182].
However, the violation of the assumption in some cases could lead to a poorly
estimated transition matrix and a degenerated classifier. This motivates the
development of algorithms without exploiting anchor points [155, 157]. Here,
we consider the SOTA estimator for the transition matrix, VolMinNet [157],
which requires the volume of the simplex formed by the columns of the
transition matrix to be small. The main idea is that the true simplex has the
minimum volume in geometry. The objective function is

$\displaystyle{\mathbf{T}}^{*}$
$\displaystyle=\mathop{\arg\min}_{\mathbf{T}\in\mathbb{T}}\log\det(\mathbf{T}),$
$\displaystyle\text{s.t.,}\ \mathbf{w}^{*}$
$\displaystyle=\mathop{\arg\min}_{\mathbf{w}}\frac{1}{m}\sum_{i=1}^{m}\ell({\mathbf{T}}^{*}f_{\mathbf{w}}(x_{i}^{(s)}),y_{i}^{(s)}),$

where $\mathbb{T}\\!=\\!\\{\mathbf{T}\\!\in\\![0,1]^{C\times
C}|\sum_{j=1}^{C}\mathbf{T}_{ij}\\!=\\!1,\mathbf{T}_{ii}>\mathbf{T}_{ij},\forall
i\neq j\\}$ is the set of diagonally dominant column stochastic matrices, and
$\det$ denotes the determinant of matrix. The volume regularization is
delicately designed for the estimation of transition matrix. In contrast, we
explore to use the problem-agnostic DAC-MR for estimating transition matrix,
i.e.,

$\displaystyle{\mathbf{T}}^{*}$
$\displaystyle=\mathop{\arg\min}_{\mathbf{T}\in\mathbb{T}}\mathcal{MR}^{dac}(D;\mathbf{w}^{*}(\mathbf{T}),A),A\in\mathcal{A},$
$\displaystyle\text{s.t.,}\ \mathbf{w}^{*}(\mathbf{T})$
$\displaystyle=\mathop{\arg\min}_{\mathbf{w}}\frac{1}{m}\sum_{i=1}^{m}\ell(\mathbf{T}f_{\mathbf{w}}(x_{i}^{(s)}),y_{i}^{(s)}),$

where $D=\\{x_{i}\\}_{i=1}^{n}$ are additionally sampled/divided from
$\mathcal{D}^{(s)}$. Note that we treat transition matrix $\mathbf{T}$ as
meta-representation, and its estimation is obtained by minimizing the DAC-MR
on the collected $D$ in a meta-learning manner [9].

Datasets. We evaluate the proposed method on two synthetic noisy datasets:
CIFAR-10 and CIFAR-100 [179], and one real-world noisy dataset: Clothing1M
[183]. We conduct experiments with two commonly used types of noise: symmetry
flipping [147] and pair flipping [153]. For CIFAR-10 and CIFAR-100, we
randomly select different 1000 images at each epochs from training set as $D$
to compute DAC-MR. For Clothing1M, we randomly select different 10 images
every class at each epochs from training set as $D$ to compute DAC-MR.

Baselines. We compared with several SOTA transition matrix estimation methods,
including Forward [147], T-Revision [155], Dual-T [156] and VolMinNet [157].

Experimental setup. For a fair comparison, we identically follow the practice
in [157]. We adopt horizontal flip, random crop, and color (brightness,
contrast, and saturation) jitter for the data augmentations. We train a
ResNet-18 network for CIFAR10, and a ResNet-34 network for CIFAR-100. We train
them using SGD with a batch size 128, a momentum 0.9, a weight decay 0.001 and
an initial learning rate 0.01. The learning rate is divided by 10 after the
30th and 60th epochs for a total 150 epoch. We adopt Adam with learning rate
$10^{-3}$ and weight decay $10^{-4}$ to update the transition matices. For
Clothing1M, we train a ResNet-50 pre-trained on ImageNet using SGD with a
momentum 0.9, a weight decay 0.001, a batch size 32 and a learning rates
$0.002$. We set the learning rates as $0.00002$ after 5-th epoch for a total
10 epoch. We adopt Adam with learning rate $10^{-4}$ and weight decay
$10^{-5}$ to update the transition matices.


11institutetext: University of Mannheim
11email<EMAIL_ADDRESS>
# A Reference Data Model for Process-Related User Interaction Logs

Luka Abb  0000-0002-4263-8438 Jana-Rebecca Rehse  0000-0001-5707-6944

###### Abstract

User interaction (UI) logs are high-resolution event logs that record low-
level activities performed by a user during the execution of a task in an
information system. Each event in a UI log corresponds to a single interaction
between the user and the interface, such as clicking a button or entering a
string into a text field. UI logs are used for purposes like task mining or
robotic process automation (RPA), but each study and tool relies on a
different conceptualization and implementation of the elements and attributes
that constitute user interactions. This lack of standardization makes it
difficult to integrate UI logs from different sources and to combine tools for
UI data collection with downstream analytics or automation solutions. To
address this, we propose a universally applicable reference data model for
process-related UI logs. Based on a review of scientific literature and
industry solutions, this model includes the core attributes of UI logs, but
remains flexible with regard to the scope, level of abstraction, and case
notion. We provide an implementation of the model as an extension to the XES
interchange standard for event logs and demonstrate its practical
applicability in a real-life RPA scenario.

###### Keywords:

User Behavior Mining UI Log Data Model Robotic Process Automation Task Mining

##  1 Introduction

_User interaction (UI) logs_ are high-resolution event logs that record low-
level, manual activities performed by a user during the execution of a task in
an information system (IS) [1]. Each event in a UI log corresponds to a single
interaction between the user and the graphical user interface (GUI) of a
software application. Examples include clicking a button, entering a string
into a text field, ticking a checkbox, or selecting an item from a dropdown
[26]. Multiple recent research streams use this type of data, for example to
analyze usage patterns in software applications [12, 28, 6], to identify
candidate routines for robotic process automation (RPA) [26, 7, 35], or to
derive RPA automation and test scripts [3, 8]. In addition, companies like
Celonis and UiPath offer tools that record and process UI data for inspecting
and automating task executions [5].

The UI logs currently used in research differ substantially. The data
collected in a specific research context is usually limited in scope and
tailored to the proposed analysis technique or automation approach. This
results in considerable variation regarding the number, type, and granularity
of recorded events and corresponding attributes. Even when researchers record
the same attributes at a similar level of detail, there is no common
definition of UI log attributes to which they can adhere. Instead, they often
rely on ad-hoc conceptualizations of elementary notions like activities and UI
components. The situation is similar in industry, where each vendor has
developed their own UI log format tailored to the capabilities of their
recording software [27].

This lack of standardization makes it difficult to integrate UI logs from
different sources [26, 30]. It also poses a challenge for the interoperability
of data collection and downstream processing tools: logs recorded by one tool
are usually only compatible with the associated analytics or automation
approach. Combining data collection and processing tools requires considerable
preprocessing effort or is entirely infeasible if the necessary attributes
cannot be recorded [27].

In this paper, we address these challenges by proposing a reference data model
for process-related UI logs. This model provides a data structure and an
accompanying interchange format that others can reuse to conceptualize and
capture UI logs in a process context. To ensure widespread applicability, the
model is designed such that it subsumes and integrates the commonalities of
existing process-related UI logs, but remains flexible with regard to the
their differences. To identify those commonalities and differences, we conduct
a literature review in section 3 and a review of industry solutions in section
4. The reference data model, along with its underlying design principles and
an accompanying interchange format, is presented in section 5. In section 6,
we demonstrate how the data model can be instantiated in practice by applying
it in a real-life RPA scenario. Finally, we conclude the paper with a
discussion in section 7.

##  2 Background and Related Work

Event Logs. Process mining extracts information from _event logs_ , i.e.,
collections of _events_ recorded in an IS [32]. An event log consists of
_cases_ that each correspond to one process instance. Each case contains a
trace of events that occurred during the execution of the process instance and
can have additional attributes, for example, the size of an order in an order-
to-cash process. Events are related to a particular step in a process with an
_activity_ label (e.g., create invoice) and can also have additional
attributes.

Data Formats. To enable the exchange of event data between different ISs, the
business process management (BPM) community has developed interchange formats
that define the structure and general contents of event logs. The current main
format is XES (eXtensible Event Stream), which was introduced in 2010 to
replace the older MXML format and was accepted as the official IEEE standard
for event data in 2016 [38]. In XES, an event log consists of a three-level
hierarchy of log, trace, and event objects. The format is designed to be
highly generic, with a minimal set of explicitly defined attributes on each of
the three levels. Additional attributes, with a commonly understood semantic
meaning, can be introduced by XES extensions. For example, the concept
extension introduces the “name” attribute, which stores names for event logs,
traces, and events. Although researchers have recently pointed out
shortcomings of XES and proposed more flexible, object-centric alternatives
such as OCEL [17], XES remains the most common event log format and is
supported by many process mining tools.

UI Logs. UI logs are a particular type of event log in which events correspond
to low-level interactions of a user with a GUI. They can be recorded either
internally by adding logging capabilities to an application, or externally by
dedicated logging tools. These tools record screen coordinates for each action
and map them to parts of the GUI using optical character recognition
technology.

User Behavior Mining. UI logs essentially record how users behave while they
are engaged with an application. They can be analyzed by means of data or
process mining techniques to gain data-driven insights into user behavior. We
refer to this analysis of UI logs as _user behavior mining (UBM)_ [1]. UBM can
serve different purposes, including the analysis of software usage patterns,
the design of new user assistance components, or the automation of tasks.

Task Mining. One application of UBM is to enhance traditional process mining
by providing a more detailed view of execution steps. Event logs gathered from
ERP systems like SAP or Oracle capture the main tasks in a process, like
creating an order, but they do not provide insights into how employees
actually perform these tasks. Recording and analyzing detailed task executions
is referred to as _task mining_ [32] or _desktop activity mining_ [29]. These
techniques can give companies deeper insights into their processes than
traditional process mining alone, and they can also help software vendors to
optimize their products, for example, by identifying common usability issues.

Robotic Process Automation. UI logs can be used to automate tasks and entire
processes by having bots emulate the recorded user interactions. This approach
to automation is called _robotic process automation (RPA)_ [22] and has lately
received considerable attention in research and practice. Within RPA, UBM
techniques can be used to derive automation scripts, but also for _robotic
process mining_ [26], which for example encompasses the identification of
suitable tasks for automation from UI logs [24].

Web Usage Mining. Another field that is concerned with the analysis of user
behavior is _web usage mining_ [34], i.e., the analysis of clickstream user
data recorded during interactions with websites. Web usage mining is often
process-agnostic; its main purpose is to optimize websites, for example, by
adapting their content and structure to users’ browsing behavior [19, 13]. The
primary data source for web usage mining are server UI logs that are generated
in a standardized logging format like the Extended Log Format [37] and have a
fixed set of attributes. These include the URL of the current and previous
page request, the resource accessed, timestamps, identifying data like the
user’s IP address, and technical data about the user’s web browser and
operating system.

UBM in other domains. In addition to the research areas mentioned above,
interaction logs have been used as a source of data-driven insights into user
behavior in several other domains, such as human-computer-interaction [14,
15], information retrieval [21], and visualization [18]. The logs in these
domains can take various forms, but they generally record user interactions at
a much lower level of detail than the process-related UI logs that we focus on
in this paper.

##  3 Literature Review

This paper’s goal is to develop a reference data model that subsumes and
integrates the commonalities of current approaches for capturing process-
related UI logs, but stays flexible with regard to their differences. In this
section, we review UI logs from scientific literature to identify those
commonalities and differences.

###  3.1 Research Method

We conducted a structured literature review [23] in SpringerLink, IEEE Xplore,
and ACM Digital Library. As search terms, we used “log” combined with  (1)
“user interact*” and “user interface”  (2) “task mining” and “desktop activity
mining” as common terms for high-resolution process mining, and  (3) “robotic
process automation” and “robotic process mining” as important applications of
UI logs.  We limited our search to papers written in English and published
after 2015 because we focus on the current state of the art. The relevance of
the initial search results was assessed based on their title and abstract.
This yielded a set of potentially relevant papers, on which we performed a
forward-backward-search to also cover papers that our search terms might have
missed.

To ascertain the relevance of the identified papers, we scanned their full
text for passages on UI logs or recording approaches for them. Papers were
considered as relevant if (1) they contained a concrete UI log or (2) they
described the UI log collection process in enough detail to infer the captured
attributes.

Table 1: The papers found in the database search Search Term |  IEEE Xplore |  ACM DL |  SpringerLink
---|---|---|---
log AND “user interact*” |  [24] |  [12] |  [2, 35, 7, 3]
log AND “user interface” |  |  |
log AND “task mining” |  |  |  [26]
log AND “desktop activity mining” |  |  |
log AND “robotic process automation” |  |  |  [5, 22]
log AND “robotic process mining” |  |  |

As listed in Table 1, we found 9 relevant publications in the initial search.
Several papers appeared in more than one query, but are only listed under the
search term that we first found them with. The forward-backward search
returned another 10 publications. Although we did not explicitly search for
web usage mining logs, our search returned papers about server-side and also
client-side web usage mining (recording web activities by adding tracking
software to a browser), but none of these met the above-listed criteria. In
our review, we therefore only included one exemplary clickstream log from a
process mining context: the BPI Challenge 2016 [11], in which the Dutch
Employee Insurance Agency recorded eight months of user activities on their
website. Our final result was hence a set of 20 relevant publications.

###  3.2 Results

Some of the 20 relevant publications covered the same use case and data
collection approach and were therefore treated as duplicates, resulting in 12
unique approaches. The majority of papers cover RPA [2, 3, 4, 5, 9, 20, 8, 22,
7, 24, 26, 27, 25, 35]. Four publications [6, 10, 12, 28] focus on software
process mining [33]. The remaining two are general approaches to analyzing
low-level user interactions with broader applications [11, 29].

Commonalities. Although the reviewed UI logs were fairly heterogeneous, we
found a set of six core attributes that are recorded in more than half of
them. Table 2 indicates which of the 12 approaches include which attributes
($\bullet$).

Table 2: UI log attributes as found in the literature review Source |  Action type |  Target element |  UI Hierarchy |  Appli-cation |  Input value |  Timestamp
---|---|---|---|---|---|---
[2, 3, 4, 5] |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$
[6] |  $\bullet$ |  $\bullet$ |  |  single |  $\bullet$ |  $\bullet$
[9] |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$
[10] |  $\bullet$ |  $\bullet$ |  |  single |  |  $\bullet$
[11] |  |  $\bullet$ |  $\bullet$ |  single |  $\bullet$ |  $\bullet$
[12] |  $\bullet$ |  |  |  single |  |  $\bullet$
[20] |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  |  $\bullet$
[8, 22] |  $\bullet$ |  |  |  $\bullet$ |  $\bullet$ |  $\bullet$
[7, 25, 24, 26, 27] |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$
[28] |  $\bullet$ |  $\bullet$ |  $\bullet$ |  single |  |  $\bullet$
[29] |  $\bullet$ |  $\bullet$ |  |  $\bullet$ |  |  $\bullet$
[35] |  |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$

  1. 1.

An _action type_ , which describes the action a user takes. Actions are most
often divided into mouse and keyboard inputs, but some logs further
distinguish between different mouse buttons, string inputs, and hotkeys. Only
two logs do not record action types: the BPIC 2016 clickstream log [11] and
Urabe et al. [35], who only record that an interaction has taken place.

  2. 2.

The atomic _target UI element_ , on which the user action is executed. This
attribute is recorded in most UI logs, except for two: Dev et al. [12] only
record the usage of specific functions, such as crop in a graphics editor, and
Jimenez-Ramirez et al. [22, 8] record click coordinates and screenshots, but
only use them to match similar user actions and do not map them to target
elements.

  3. 3.

The _software application_ that the user interacts with. This could be a web
browser, an ERP system, or an office application. This attribute is always
recorded when researchers track user actions across multiple applications, but
is not captured when the tracking is limited to a single application.

  4. 4.

One or multiple attributes that specify the location of the target element in
the application’s _UI hierarchy_. For example, an Excel cell is located in a
worksheet (hierarchy level 1), which belongs to a workbook (hierarchy level 2)
[27]. UI hierarchy attributes are included in about half the reviewed logs.

  5. 5.

The _input value_ that the user writes into a text field. Input values are
included in about half of the reviewed logs.

  6. 6.

A _timestamp_ , which records the exact date and time at which the action
occurred. This is recorded in all logs.

Differences. Most authors characterize user interactions through an action
type, i.e., _what_ the user does, and a target element, i.e., _where_ they do
it. However, the set of possible values for the action type, and hence the
level of detail at which actions are recorded, differs considerably. For
example, Agostinelli et al. [3] record aggregated action types abstracted from
raw hardware input (e.g., clickButton and clickTextField), whereas Jimenez-
Ramirez et al. [22] make the low-level differentiation between left, right,
and middle mouse clicks. Which other attributes are included in a UI log
differs between approaches: whereas timestamps and the application in focus
(where applicable) are recorded in all logs, input values and information on
the location of a target element within the application’s UI hierarchy are
included in about half of them. Examples for other, less common attributes
that are only recorded in few approaches include the current value of a text
field [27, 9, 5], user IDs [35, 5, 27, 10], other resources involved [6, 11],
and associations to higher-level process steps [29, 20].

Another interesting finding was that most of the reviewed UI logs are
initially unlabeled, i.e., they do not have a concrete case notion [16]. In
some publications, events in unlabelled logs are later grouped into cases
based on different attributes. These attributes include external session IDs
created automatically by a system [11] or manually by users [27, 3, 28], user
IDs [10], or case IDs from associated higher-level event logs [20].

##  4 Review of Industry Solutions

To ensure broad applicability of our reference data model, we also review
industry approaches for conceptualizing and capturing UI logs. Because those
approaches are core to the industry solutions’ functionality and business
secrets, the available material for this review may be less specific than
scientific papers. Therefore, we conduct the industry review in this section
as an addition to the literature review, meant to confirm and complement the
established findings.

###  4.1 Research Method

Selection Strategy. An initial analysis indicated that RPA tools are presently
the only industry solutions that collect UI logs on a large scale. Some
vendors also advertise task mining capabilities, but their primary focus is on
recording UI logs for the automation of routines. Because the RPA market is
highly fractured and fast-moving, we could not conduct a complete review.
Instead, we opted to analyze a sample of companies that can be seen as
representative for the market. Therefore, we selected the companies that the
2021 Gartner Magic Quadrant RPA
report111https://www.gartner.com/en/documents/3988021 attributes with a “high
ability to execute” and/or a “high completeness of vision”: UiPath, Automation
Anywhere, Microsoft Power Automate, Blue Prism, NICE, WorkFusion, Pegasystems,
Appian, EdgeVerve Systems, and Servicetrace. We also included Celonis Task
Mining, which is the only major product that uses UI logs primarily for low-
level process mining.

Review Approach. In analyzing those eleven tools, we focused on finding the
commonalities and differences between the industry logs and the scientific
logs. Specifically, we wanted to know whether the industry logs capture the
same set of six core attributes found in the scientific logs (commonalities)
and whether the industry logs capture any other attributes that could be
relevant for a widely applicable reference data model (differences). To answer
those questions, we collected freely available material about the tools.222A
full list of the material that we analyzed can be found at https://gitlab.uni-
mannheim.de/jpmac/ui-log-data-
model/-/blob/46b363dc75b992a43398f501e3a4cb0e755107d0/industry_review_sources.pdf
This included trial or demo versions, documentations, and promotional
material, such as videos showcasing the recording process. After collecting
the material, we had to exclude two companies from our list, Pegasystems and
EdgeVerve Systems, because we could not obtain sufficient information on the
functionalities of their recording software.

###  4.2 Results

Commonalities. For each industry solution, we analyzed whether it also records
the six core attributes found in the literature review. The results are
summarized in Table 3. All reviewed tools record action types, target
elements, input values, applications, and timestamps. Similar to what we found
in the literature review, the recordable action types differ considerably
between tools.

Table 3: UI log attributes as found in the industry review Company |  Action type |  Target element |  UI Hierarchy |  Appli-cation |  Input value |  Time-stamp
---|---|---|---|---|---|---
UiPath |  $\bullet$ |  $\bullet$ |  screenshots |  $\bullet$ |  $\bullet$ |  $\bullet$
MS Power Automate |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$
Automation Anywhere |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$ |  $\bullet$
Celonis |  $\bullet$ |  $\bullet$ |  screenshots |  $\bullet$ |  $\bullet$ |  $\bullet$
Blue Prism |  $\bullet$ |  $\bullet$ |  screenshots |  $\bullet$ |  $\bullet$ |  $\bullet$
Workfusion |  $\bullet$ |  $\bullet$ |  screenshots |  $\bullet$ |  $\bullet$ |  $\bullet$
NICE |  $\bullet$ |  $\bullet$ |  screenshots |  $\bullet$ |  $\bullet$ |  $\bullet$
Appian |  $\bullet$ |  $\bullet$ |  screenshots |  $\bullet$ |  $\bullet$ |  $\bullet$
ServiceTrace |  $\bullet$ |  $\bullet$ |  screenshots |  $\bullet$ |  $\bullet$ |  $\bullet$

Differences. We also examined whether the industry solutions systematically
record any other attributes, but we did not find any. However, we did find a
significant difference between industry logs and scientific logs in how they
capture information on the location of elements within the UI hierarchy. In
research, this information is explicitly recorded in UI log attributes, but
most industry tools instead store it as screenshots outside of the log. Some
tools also use the UI hierarchy to construct selectors that uniquely identify
an element within an application’s GUI, similar to file paths. Another
difference between industry logs and scientific logs concerns the case notion.
In the industry solutions, the case ID is always a task or process label that
is manually added to the log. Additional business context attributes also need
to be added by users and are not recorded by the tool.

##  5 Reference Data Model

In this section, we introduce our reference data model for user interactions.
We consider a reference model to be a conceptual model that serves to be
reused for the design of other conceptual models [31]. Under such a reuse-
oriented conceptualization, (universal) applicability of the model is not a
defining property. However, maximizing the model’s application scope increases
its reuse potential and therefore its value to the community. Therefore, we
designed the model in an inductive or bottom-up fashion [31]: based on the
commonalities and differences between existing UI logs that we found in the
literature and industry reviews, we constructed a model that subsumes those
commonalities, but remains flexible with regard to their differences. In the
following, we first elaborate on the principles that guided our design process
in subsection 5.1. The reference model is presented in detail in subsection
5.2. In subsection 5.3, we provide a data interchange format for UI log data
as a supplement to the reference model.

###  5.1 Design Principles

In the literature and industry reviews, we found that the main commonality
between existing UI logs are the six core attributes. The main differences
between them concerned the scope, the level of abstraction, and the case
notion. Based on these findings, our data model follows four fundamental
design principles:

  1. 1.

Minimal set of core components: The essential characteristics of user
interactions, as found in the reviews, are modeled as the components and
standard attributes of the data model. Because the model is intended to be
non-specific and universally applicable, we include no other elements, thus
keeping the number of components and standard attributes to a minimum.

  2. 2.

Flexible scope: To ensure flexibility in scope, the data model can be extended
with any number of additional components and all components can have an
arbitrary number of attributes. Also, nearly all components and standard
attributes are optional. The only non-optional component and attribute that
ensure the existence of a UI log are the activity and its name.

  3. 3.

Flexible level of abstraction: To enable user interactions to be modeled in
various application contexts and at various levels of abstraction, the domain
of the standard attributes in the data model, such as the action type, is left
unspecified and can be determined at the point of instantiation. Furthermore,
all components are modeled as classes and can be subclassed. Explicit
subclasses are only defined for the target object, because they are inherent
to the structure of user interfaces and the way they are embedded in ISs.

  4. 4.

No explicit case notion: Whereas the case notion of a business processes is
tied to its instances, UI logs are not inherently structured along any data
dimension. The reviews have shown that they can have many possible case
identifiers. The data model therefore does not include an explicit case
notion. Instead, the case notion needs to be defined at the point of
instantiation.

###  5.2 Reference Model Components

The reference data model is depicted as a UML diagram in Figure 1. It consists
of nine components, modeled as classes, and their interrelations, modeled as
associations. Each class has an ID and can have any number of attributes. Some
components have standard attributes that have a particular significance for
user interactions. In the following, we define and explain the individual
components.

Figure 1: User interaction data model

Components that define the user interaction. In our model, user interactions
have two parts. First, the _action_ component with its _action type_ standard
attribute that describes what the user does. Common action types, as observed
in the reviews, correspond to the functionalities of standard peripheral input
devices, such as left or right mouse clicks, single keystrokes, or keystroke
combinations for shortcuts. Higher-level distinctions are also possible. For
example, when collecting data in an ERP system, actions can be divided into
input actions, which make changes to a business object, and navigation
actions, which only serve to navigate the GUI.

The second part of an interaction is the _target object_ that the action is
executed on. It is instantiated as one of four object types in the UI
hierarchy, as explained below. The action type and target object together
determine the central model component: the _activity_. It is uniquely defined
as a combination of an action and a target object and acts as the event label,
like in a traditional event log. An activity has three standard attributes:
the _activity name_ , an optional _input value_ that denotes, e.g., the string
that is entered into a text field, and a _timestamp_ to indicate its execution
time. The activity name is determined as a function of the action type of the
corresponding action and the identifier of the corresponding target object,
for example, a concatenation. The timestamp is a very common attribute in
traditional event logs as well as UI logs. It is, however, not a strictly
required attribute in the data model, since there are alternative ways to
introduce a notion of order into an event log [36].

Components that define the UI hierarchy. The UI hierarchy integrates the
various types of UI element context data into a general structure. It consists
of four components, which form a tree-shaped composition hierarchy: UI
element, UI group, application, and system. The UI element and UI group levels
mirror the hierarchical structure of virtually all GUIs (e.g., the document
object model of a website). The application and system levels go beyond the
actual GUI and position it within an IS, which makes it possible to record
application- and system-level user interactions and allows the UI log to be
compatible with cross-application and even cross-system UI tracking.

Most actions are executed on the atomic _UI elements_ , which form the lowest
level. Examples include buttons, text boxes, dropdowns, checkboxes, or
sliders. Elements can be stateful, such as a non-empty text box or a greyed-
out button. Capturing this state is necessary, for example, to track the
effects of copy/paste actions or to differentiate between activity outcomes.
The state of a UI element is therefore recorded in its _current state_
standard attribute.

UI elements are combined into _UI groups_ , which can be nested within other
UI groups. In many cases, these UI groups are explicit design elements of the
user interface, but our model does not impose grouping criteria and allows UI
groups to be formed from arbitrary sets of UI elements. A simple example that
we saw in the literature review is an Excel cell (UI element), which is part
of a worksheet (UI group), which is again part of a workbook (UI group).
Modeling UI groups has two main advantages. First, it allows to uniquely
identify functionally identical UI elements. For the example above, recording
information about UI groups allows us to distinguish between the cell A1 in
separate Excel worksheets. This idea is used in many industry solutions to
generate element selectors from screen captures. Second, UI groups can be
useful for event abstraction, i.e., mapping user interactions to higher-level
conceptual tasks, if these tasks are closely tied to particular UI groups. For
example, all interactions with elements in a login mask (enter username, enter
password, click login) can directly be abstracted to the “login” task.

UI elements and UI groups belong to an _application_ , i.e., a single program
instance. Some actions are directly executed on the application and are not
tied to lower-level elements, such as “undo” or application-specific hotkeys.

The root node of the UI hierarchy is the _system_ , on which the applications
run and actions are recorded. Similar to application-level actions, it is also
possible to capture system-level actions, such as the Ctrl-Alt-Del key
combination to open the Task Manager on a Windows system.

Components that define the context. Finally, the data model includes two
components that put UIs in a conceptual context: _user_ and _task_. These
exist in some form for all UI logs, which is why they are included in the
model. In contrast, other potential context components, such as organizational
or resource attributes, are use-case-specific and can be considered by
extending the model.

The user is the entity that initiates any interaction. Each action is
associated with a single user. Because user IDs and attributes depend on the
data collection environment (e.g., device IDs in mobile applications or IP
addresses on websites), the model does not specify any attributes for users.
This also means that the user component is not necessarily restricted to
humans and can model computer-initiated interactions, for example when
recording partially automated processes.

The task component associates the recorded user interactions with conceptual
tasks or routines, which makes it possible to map low-level GUI interactions
to higher-level user activities. This abstraction is an essential prerequisite
for being able to perform meaningful analysis on UI logs or to use them for
automation.

###  5.3 Exchange Format

To further increase the applicability and reuse potential of our data model,
we implemented it as an extension to the XES standard for event logs.333The
XML specification for the _UILog_ extension is available at
https://gitlab.uni-mannheim.de/jpmac/ui-log-data-
model/-/raw/main/UILog_extension. This _UIlog_ extension provides a
standardized exchange format for UI logs as a supplement to the data model.
The implementation considers the activity equivalent to the event label and
does not include the activity name or timestamp standard attributes because
those are already provided by the concept and time extensions. The other
components and standard attributes are defined at event level, i.e., as
attributes of an activity instance. The generic target object is not directly
implemented, but can instead be specified through attributes that correspond
to its four UI hierarchy subclasses: the target object is the lowest-level UI
hierarchy component that exists for this event. For example, for an event with
a UI element attribute, the target object is always this UI element, whereas
for an event with no UI element or UI group attributes, the target object is
the application.

##  6 Working Example

To demonstrate the practical utility of the reference data model, we describe
in this section how it can be instantiated in a real-life scenario. The
scenario is based on an RPA project, which we are currently conducting in
cooperation with an ERP system vendor. The project is set in the medical
technology industry, where companies are required to regularly validate their
ISs to ensure that they are in compliance with external quality regulations.
The validation of an IS involves manually executing a number of predefined
workflows step-by-step according to a rigid execution plan, checking the
result of each step against a set of acceptance criteria, and documenting the
result. Manually executing a well-defined validation workflow is a repetitive
and time-consuming task. The goal of our project is to automate this task
using RPA. We want to record how process experts interact with the UI of the
ERP system during validation and then train bots to emulate their actions.

In the following, we use the example of a keyword creation workflow to show
how an artificial UI log that captures one execution of this workflow may
instantiate the data model. The keyword creation workflow consists of five
consecutive steps, which are executed on the GUI parts shown in Figure 2. The
user  (1) logs in (a),  (2) selects the right client and profile (b),  (3)
navigates through the dashboard (c) to reach the explorer tree (d, left),  (4)
creates a new keyword (d, right), and  (5) logs out.  The main acceptance
criterion is that the newly created keyword shows up in the explorer tree
after refreshing.

a) Login

b) Profile selection

c) Dashboard

d) Explorer tree with ”Create information object” form panel

Figure 2: The user interface of the application described in the example

Table 4 shows a UI log for one case, i.e., one execution of the keyword
creation workflow. It includes the action type, target UI element, and one
level of UI groups, plus input value and current state where applicable. The
captured action types are left and right clicks, text input, selected keyboard
shortcuts, and none. The activity label of an event (in most cases) consists
of the concatenated action type and target object identifier.

Table 4: UI log for one execution of the keyword creation workflow Activity | Action type |  UI element | UI group |  Input value |  Current state
---|---|---|---|---|---
A_Login | none |  | login mask |  {username: pren, password: dts123} |
A_Profile Selection | none |  | user select client |  {client: base, profile: author} |
click content | left click |  content | dashboard ov |  |
click masterdata | left click |  masterdata | explorer tree |  |
click masterdata node expand | left click |  masterdata node expand | explorer tree |  |
click keywords node expand | left click |  keywords node expand | explorer tree |  |
rclick keywords | right click |  keywords | explorer tree |  |
click ppanel new | left click |  ppanel new | explorer tree |  |
click new information object | left click |  new information object | explorer tree |  |
click name | left click |  name | fpanel keyword |  |
input name | input |  name | fpanel keyword |  MyKeyword |
click dd type | left click |  dd type | fpanel keyword |  |  [keyword, keywords folder]
click dd type | left click |  dd type | fpanel keyword |  keyword |  [keyword, keywords folder]
click dd linksto | left click |  dd linksto | fpanel keyword |  |  [linksto]
click dd linksto | left click |  dd linksto | fpanel keyword |  linksto |  [linksto]
click confirm | left click |  confirm | fpanel keyword |  |
click keywords node expand | left click |  keywords node expand | explorer tree |  |
KEY_F5 explorer tree | KEY_F5 |  | explorer tree |  |
click logout | left click |  logout | explorer tree |  |
click confirm | left click |  confirm | dialog logout |  |

The first two events in the log do not correspond to single user interactions,
but instead take advantage of the UI group concept to directly abstract to
higher-level tasks. Instead of recording each event in the login and client
selection masks separately, the task is tracked only at completion and the
content of the text fields is read out when the user presses the “Login” and
“Set Profile” buttons. For these abstracted activities (marked with an “A_”
prefix), the action type is “none”; they are defined only through the target
UI group, independent of the performed actions. This approach can be used for
simple tasks with the same execution pattern in all workflows. Its main upside
is that it reduces noise, which is a common problem in UI logs [26]. In our
scenario, activities like initially entering a wrong password do not affect
the outcome of the workflow and are therefore not relevant for automating it.
By abstracting during data collection, those activities are automatically
disregarded. Other advantages of abstraction are reduced implementation effort
and smaller UI logs.

For effective automation, various user inputs need to be tracked. Therefore,
the instantiation of the input value attribute in the log is flexible and
depends on the action type and target object: When a user writes into a
textbox, the input value is the entered string. When an item is selected from
a dropdown, the input value records the label of that item. For abstracted
activities, the input value captures the string values of all relevant UI
group elements as a map. Most state information, however, is not required for
automation. Therefore, the current state attribute only records the values
that can be selected from list and dropdown elements, which is needed for some
more complex workflows in the validation process. For example, if a documents
needs to be approved, the validation must verify that a document’s author
cannot be selected as approver.

This simple example demonstrates how some of the core components of the
reference model can be instantiated in a real-life scenario, and how the
flexibility in abstraction level can be leveraged to record attributes in a
way that matches the requirements of a particular use case. It also shows
that, in practice, components that are not relevant for a use case can simply
be left out. The main advantage of using the reference model here is that,
unlike with an ad-hoc model tailored to the use case, the attributes captured
in the UI log follow a general convention that also applies to other user
interfaces. This makes recording UI logs in the same format straightforward
even in other applications, and makes it possible to develop automation or
task mining solutions that are independent of the recording approach used.

##  7 Discussion and Conclusion

In this paper, we propose a reference data model for UI logs. Based on reviews
of scientific literature and industry solutions, it has a set of core
components to capture essential characteristics of user interactions and is
flexible with regard to scope, abstraction level, and case notion. We
implement the model as an XES extension and exemplarily show how it can be
instantiated in a real-life RPA scenario.

Contribution. Our main objective is to address the issues that arise from the
lack of standardization of UI logs. Therefore, we derive the reference from
existing UI logs. Most of the components are directly adopted from the core
attributes identified in our reviews. Our contribution is their integration
into a unified framework with well-defined relations. For example, we propose
a rigid interpretation of an activity by defining it as a combination of an
action and a target element. We also expand on the location context of UI
elements and explicitly define four distinct types of target objects in an
unambiguous hierarchy.

To this unified framework, we add additional, less frequently collected
components and standard attributes that are particularly relevant for a
complete model of user interactions. For instance, the current state is an
important property of stateful UI elements when the log is intended to be used
for automation. In the UI hierarchy, we add the system on top of the commonly
recorded application to model system-level user interactions. We also
introduce the user and task context components to add (optional) generic
business context to UI logs.

Limitations. One limitation of our work concerns its grounding in existing UI
logs. Despite following a methodical approach, we do not claim that our
reviews or the model are complete or exhaustive. There could be unidentified
UI logs or future UI logs in different use cases, which are not well
represented by the model. For instance, our data model is only intended to
model user interactions with graphical user interfaces, and we did not
consider alternative input types, for example from voice commands or eye-
tracking devices. The model may also be somewhat biased towards automation use
cases because RPA solutions are overrepresented in the two reviews that it is
based on.

Another limitation is that the XES standard is not particularly well-suited
for UI logs. It does not support explicitly defining the relations between
attributes, so all components of the UI hierarchy have to be implemented at
event level. Therefore, even if many events involve the same target object, UI
group, application, system and their attributes need to be included each time,
leading to considerable redundancy. XES also assumes a single case notion,
contrary to the flexible case notion that we intend for the data model.

Conceptually, implementing the model as an extension to the Object-Centric
Event Log format OCEL [17] would be more appealing, because users, tasks, and
UI hierarchy elements could be modeled as objects, reducing the redundancy.
However, OCEL does not support extensions and has two main limitations with
regard to UI logs. First, it does not support dynamic object attributes that
can differ between events, such as the current value of a textbox that may
change between interactions. Second, object attributes cannot be tied to
certain object types, so for example the current state attribute cannot be
limited to UI elements only. Therefore, we decided to implement the model as
an XES extension.

Future Work. Our reference model can contribute to the field by providing a
common, application-independent conceptual framework for user interactions.
However, like any reference model, it needs to prove its utility in practice.
We therefore want to encourage researchers and practitioners to adopt the
model for capturing UI logs in their projects, and to extend it both with
regard to new use cases and with regard to conceptual aspects, such as user
privacy.

## References

  * [1] Abb, L., Bormann, C., van der Aa, H., Rehse, J.R.: Trace clustering for user behavior mining. In: European Conference on Information Systems. AIS (2022)
  * [2] Agostinelli, S., Leotta, F., Marrella, A.: Interactive segmentation of user interface logs. In: Service-Oriented Computing. pp. 65–80. Springer (2021)
  * [3] Agostinelli, S., Lupia, M., Marrella, A., Mecella, M.: Automated generation of executable RPA scripts from user interface logs. In: BPM Forum. pp. 116–131. Springer (2020)
  * [4] Agostinelli, S., Marrella, A., Mecella, M.: Automated segmentation of user interface logs. In: Robotic Process Automation. pp. 201–222. De Gruyter Oldenbourg (2021)
  * [5] Agostinelli, S., Marrella, A., Mecella, M.: Exploring the challenge of automated segmentation in robotic process automation. In: Research Challenges in Information Science. pp. 38–54. Springer (2021)
  * [6] Ardimento, P., Bernardi, M.L., Cimitile, M., Ruvo, G.D.: Learning analytics to improve coding abilities: a fuzzy-based process mining approach. In: International Conference on Fuzzy Systems. pp. 1–7. IEEE (2019)
  * [7] Bosco, A., Augusto, A., Dumas, M., La Rosa, M., Fortino, G.: Discovering automatable routines from user interaction logs. In: BPM Forum. pp. 144–162. Springer (2019)
  * [8] Chacón Montero, J., Jimenez-Ramirez, A., Gonzalez Enríquez, J.: Towards a method for automated testing in robotic process automation projects. In: International Workshop on Automation of Software Test. pp. 42–47 (2019)
  * [9] Choi, D., R’bigui, H., Cho, C.: Candidate digital tasks selection methodology for automation with robotic process automation. Sustainability 13(16), 8980 (2021)
  * [10] Damevski, K., Shepherd, D.C., Schneider, J., Pollock, L.: Mining sequences of developer interactions in visual studio for usage smells. IEEE Transactions on Software Engineering 43(4), 359–371 (2017)
  * [11] Dees, M., van Dongen, B.: BPI challenge 2016 (2016), https://data.4tu.nl/articles/dataset/BPI_Challenge_2016_Clicks_Logged_In/12674816/1
  * [12] Dev, H., Liu, Z.: Identifying frequent user tasks from application logs. In: International Conference on Intelligent User Interfaces. p. 263–273. ACM (2017)
  * [13] Ding, A., Li, S., Chatterjee, P.: Learning user real-time intent for optimal dynamic web page transformation. Information Systems Research 26(2), 339–359 (2015)
  * [14] Dumais, S., Jeffries, R., Russell, D., Tang, D., Teevan, J.: Understanding User Behavior Through Log Data and Analysis, pp. 349–372. Springer (2014)
  * [15] Fern, X., Komireddy, C., Grigoreanu, V., Burnett, M.: Mining problem-solving strategies from HCI data. ACM Transactions on Computer-Human Interaction 17(1) (2010)
  * [16] Ferreira, D.R., Gillblad, D.: Discovering process models from unlabelled event logs. In: Business Process Management. Springer (2009)
  * [17] Ghahfarokhi, A., Park, G., Berti, A., van der Aalst, W.: OCEL: A standard for object-centric event logs. In: Advances in Databases and Information Systems. pp. 169–175. Springer (2021)
  * [18] Guo, H., Gomez, S., Ziemkiewicz, C., Laidlaw, D.: A case study using visualization interaction logs and insight metrics to understand how analysts arrive at insights. IEEE Transactions on Visualization and Computer Graphics 22(1), 51–60 (2016)
  * [19] Ho, S., Bodoff, D., Tam, K.: Timing of adaptive web personalization and its effects on online consumer behavior. Information Systems Research 22(3), 660–679 (2010)
  * [20] Hofmann, A., Prätori, T., Seubert, F., Wanner, J., Fischer, M., Winkelmann, A.: Process selection for RPA projects: A holistic approach. In: Robotic Process Automation. pp. 77–90. De Gruyter Oldenbourg (2021)
  * [21] Islamaj Dogan, R., Murray, G., Névéol, A., Lu, Z.: Understanding PubMed® user search behavior through log analysis. Database 2009 (2009)
  * [22] Jimenez-Ramirez, A., Reijers, H.A., Barba, I., Del Valle, C.: A method to improve the early stages of the robotic process automation lifecycle. In: Advanced Information Systems Engineering. pp. 446–461. Springer (2019)
  * [23] Kitchenham, B.: Procedures for performing systematic reviews. Tech. rep., Keele University (2004)
  * [24] Leno, V., Augusto, A., Dumas, M., La Rosa, M., Maggi, F.M., Polyvyanyy, A.: Identifying candidate routines for robotic process automation from unsegmented UI logs. In: International Conference on Process Mining. pp. 153–160. IEEE (2020)
  * [25] Leno, V., Augusto, A., Dumas, M., La Rosa, M., Maggi, F.M., Polyvyanyy, A.: Discovering data transfer routines from user interaction logs. Information Systems (2021)
  * [26] Leno, V., Polyvyanyy, A., Dumas, M., La Rosa, M., Maggi, F.M.: Robotic process mining: Vision and challenges. Business & Information Systems Engineering 63(3), 301–314 (2021)
  * [27] Leno, V., Polyvyanyy, A., La Rosa, M., Dumas, M., Maggi, F.: Action logger: Enabling process mining for robotic process automation. In: BPM Demos. Springer (2019)
  * [28] Linares-Vásquez, M., White, M., Bernal-Cárdenas, C., Moran, K., Poshyvanyk, D.: Mining android app usages for generating actionable gui-based execution scenarios. In: Working Conference on Mining Software Repositories. p. 111–122. IEEE (2015)
  * [29] Linn, C., Zimmermann, P., Werth, D.: Desktop activity mining - a new level of detail in mining business processes. In: INFORMATIK. pp. 245–258. Köllen (2018)
  * [30] López-Carnicer, J.M., del Valle, C., Enríquez, J.G.: Towards an OpenSource logger for the analysis of RPA projects. In: BPM Forum. pp. 176–184. Springer (2020)
  * [31] Rehse, J.R., Fettke, P.: A procedure model for situational reference model mining. Enterprise Modelling and Information Systems Architectures 14(3) (2019)
  * [32] Reinkemeyer, L.: Process Mining in Action: Principles, Use Cases and Outlook. Springer (2020)
  * [33] Rubin, V.A., Mitsyuk, A.A., Lomazova, I.A., van der Aalst, W.: Process mining can be applied to software too! In: International Symposium on Empirical Software Engineering and Measurement. ACM (2014)
  * [34] Srivastava, J., Cooley, R., Deshpande, M., Tan, P.N.: Web usage mining: Discovery and applications of usage patterns from web data. In: SIGKDD Explorations. vol. 1, pp. 12–23. ACM (2000)
  * [35] Urabe, Y., Yagi, S., Tsuchikawa, K., Oishi, H.: Task clustering method using user interaction logs to plan RPA introduction. In: Business Process Management. pp. 273–288. Springer (2021)
  * [36] van der Aalst, W., Santos, L.: May I take your order? In: BPM Workshops. pp. 99–110. Springer (2022)
  * [37] WWW Consortium: Extended log file format (1995), https://www.w3.org/TR/WD-logfile.html
  * [38] XES Working Group: IEEE standard for eXtensible Event Stream (XES) for achieving interoperability in event logs and event streams. IEEE Std 1849 (2016)