# Adaptive Lasso, Transfer Lasso, and Beyond:
An Asymptotic Perspective

Masaaki Takada
<EMAIL_ADDRESS>Toshiba CorporationThe Institute of Statistical
Mathematics Hironori Fujisawa22footnotemark: 2
<EMAIL_ADDRESS>
###### Abstract

This paper presents a comprehensive exploration of the theoretical properties
inherent in the Adaptive Lasso and the Transfer Lasso. The Adaptive Lasso, a
well-established method, employs regularization divided by initial estimators
and is characterized by asymptotic normality and variable selection
consistency. In contrast, the recently proposed Transfer Lasso employs
regularization subtracted by initial estimators with the demonstrated capacity
to curtail non-asymptotic estimation errors. A pivotal question thus emerges:
Given the distinct ways the Adaptive Lasso and the Transfer Lasso employ
initial estimators, what benefits or drawbacks does this disparity confer upon
each method? This paper conducts a theoretical examination of the asymptotic
properties of the Transfer Lasso, thereby elucidating its differentiation from
the Adaptive Lasso. Informed by the findings of this analysis, we introduce a
novel method, one that amalgamates the strengths and compensates for the
weaknesses of both methods. The paper concludes with validations of our theory
and comparisons of the methods via simulation experiments.

##  1 Introduction

We consider an ordinary high-dimensional regression problem. Let
$X=(\mathbf{x}_{1},\dots,\mathbf{x}_{p})=(x_{1}^{\top},\dots,x_{n}^{\top})^{\top}\in\mathbb{R}^{n\times
p}$ and $y\in\mathbb{R}^{n}$ be a feature matrix and response vector,
respectively. We suppose a true model is linear with independent and
identically distributed (i.i.d.) Gaussian noise, that is,

(1) $\displaystyle y$
$\displaystyle=X\beta^{*}+\varepsilon,\leavevmode\nobreak\
\varepsilon_{i}\overset{i.i.d.}{\sim}\mathcal{N}(0,\sigma^{2}),$

where $\beta^{*}\in\mathbb{R}^{p}$ is a true regression parameter and
$\varepsilon\in\mathbb{R}^{n}$ is a Gaussian noise. We presume that
$\beta^{*}$ is sparse, and designate the active and inactive parameters as $S$
and $S^{c}$, namely $S:=\\{j:\beta^{*}_{j}\neq 0\\}$ and
$S^{c}:=\\{j:\beta^{*}_{j}=0\\}$, respectively.

The Lasso [18] is a classical regression method for high-dimensional data,
defined by

(2)
$\displaystyle\hat{\beta}_{n}^{\mathcal{L}}=\operatorname*{argmin}_{\beta}\left\\{\frac{1}{n}\|y-X\beta\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}|\beta_{j}|\right\\}.$

Owing to $\ell_{1}$ regularization, the solution exhibits sparsity. We denote
$\hat{S}_{n}^{\mathcal{L}}:=\\{j:{\hat{\beta}}_{j}^{\mathcal{L}}\neq 0\\}$.

Numerous theoretical studies have elucidated the strengths and limitations of
the Lasso. According to asymptotic theory, the Lasso estimator is consistent
if $\lambda_{n}=o(n)$ and is $\sqrt{n}$-consistent if
$\lambda_{n}=O(\sqrt{n})$ [5]. However, [21] demonstrates that the Lasso has
inconsistent variable selection if $\lambda_{n}=O(\sqrt{n})$, while it does
not have $\sqrt{n}$-consistency if $\lambda_{n}=o(n)$ and
$\lambda_{n}/\sqrt{n}\to\infty$. Hence, the Lasso cannot achieve both
$\sqrt{n}$-consistency and consistent variable selection simultaneously (see
Figure 1 left).

To improve the asymptotic properties of the Lasso, one of the most well-known
methods is the Adaptive Lasso [21, 10], which is given by

(3)
$\displaystyle\hat{\beta}_{n}^{\mathcal{A}}=\operatorname*{argmin}_{\beta}\left\\{\frac{1}{n}\|y-X\beta\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}w_{j}|\beta_{j}|\right\\},\leavevmode\nobreak\
w_{j}:=\frac{1}{|{\tilde{\beta}}_{j}|^{\gamma}},$

where ${\tilde{\beta}}$ is an initial estimator of the true parameter
$\beta^{*}$ and $\gamma>0$ is a hyperparameter. We denote
$\hat{S}_{n}^{\mathcal{A}}:=\\{j:{\hat{\beta}}_{j}^{\mathcal{A}}\neq 0\\}$. If
${\tilde{\beta}}$ is a $\sqrt{n}$-consistent estimator,
$\lambda_{n}=o(\sqrt{n})$, and $\lambda_{n}n^{(\gamma-1)/2}\rightarrow\infty$,
then the Adaptive Lasso satisfies both $\sqrt{n}$-consistency and consistent
variable selection, as well as asymptotic normality (Figure 1 right). This is
known as the oracle property because it behaves as if the true active
variables were given in advance. The Adaptive Lasso assumes the existence of a
$\sqrt{n}$-consistent initial estimator and uses it as the weight of the
$\ell_{1}$ regularization.

$\lambda_{n}$$n$$\sqrt{n}$(ii)✘ sub-$\sqrt{n}$-consistent✔ consistent variable
selection(under the incoherence condition)(i)✔ $\sqrt{n}$-consistent✘
inconsistent variable selection

$\lambda_{n}$$\sqrt{n/n^{\gamma}}$$\sqrt{n}$✔ $\sqrt{n}$-consistent +
normality✔ consistent variable selection

Figure 1:  Phase diagrams with the order of $\lambda_{n}$ for the Lasso (left)
and the Adaptive Lasso (right). The Lasso does not achieve
$\sqrt{n}$-consistent and consistent variable selection simultaneously, while
the Adaptive Lasso satisfies both.

Recently, a different use of an initial estimator has been proposed [16, 1],
which is given by

(4)
$\displaystyle\hat{\beta}_{n}^{\mathcal{T}}=\operatorname*{argmin}_{\beta}\left\\{\frac{1}{n}\|y-X\beta\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}|\beta_{j}|+\frac{\eta_{n}}{n}\sum_{j}\left|\beta_{j}-{\tilde{\beta}}_{j}\right|\right\\},$

where ${\tilde{\beta}}$ is an initial estimator (“source parameter” in the
field of transfer learning). We denote
$\hat{S}_{n}^{\mathcal{T}}:=\\{j:{\hat{\beta}}_{j}^{\mathcal{T}}\neq 0\\}$.
This method is called Transfer Lasso. The first regularization term in (4)
shrinks the estimator to zero and induces sparsity. The second regularization
term in (4), on the other hand, shrinks the estimator to the initial estimator
and induces the sparsity of changes from the initial estimator. The $\ell_{1}$
regularization of the difference between the initial estimator and the target
estimator plays a key role in sparse updating, in which only a small number of
parameters are changed from the initial estimator. Non-asymptotic analysis
reveals that a small $\Delta:={\tilde{\beta}}-\beta^{*}$ brings advantageous
on its estimation error bounds for the Transfer Lasso over the Lasso [16].

The Adaptive Lasso and the Transfer Lasso have similarities and differences.
They are similar in that they both use an initial estimator in $\ell_{1}$
regularization. However, the way the initial estimator is used is different:
the Adaptive Lasso uses the parameter “divided” by the initial estimator in
the regularization, whereas Transfer Lasso uses the parameter “subtracted” by
the initial estimator in the regularization. In addition, the original
motivations are different: The Adaptive Lasso aims to reduce estimation bias
as well as satisfy consistency in variable selection, whereas the Transfer
Lasso aims to sparsify both the estimator itself and the change from the
initial estimator, leveraging the knowledge of the initial estimator.

These raise major questions: How do these similarities and differences between
Adaptive Lasso and Transfer Lasso affect the theoretical properties and
empirical results of each method? In this paper, we highlight the asymptotic
properties of each method and seek to answer the following research questions.

  1. 1.

Does the Transfer Lasso have the same properties as the Adaptive Lasso?
Specifically, does the Transfer Lasso have the oracle property that the
Adaptive Lasso has?

  2. 2.

Does the Transfer Lasso have different properties from the Adaptive Lasso? If
so, under what conditions of initial estimators, does the Transfer Lasso have
an advantage over the Adaptive Lasso, or vice versa?

  3. 3.

If these two methods have their specific advantages and disadvantages, are
there any ways to compensate for the disadvantages of both and to reconcile
their advantages?

  4. 4.

How does the asymptotic property of the estimator change as the order of the
hyperparameters changes for each method?

Our theoretical analysis led us to the following findings.

  1. 1.

The Transfer Lasso does not have the oracle property in general. This is an
unfavorable property compared to the Adaptive Lasso.

  2. 2.

The Transfer Lasso has an advantage in convergence rate if the initial
estimator is estimated from sufficiently large data. The Adaptive Lasso, in
contrast, does not benefit from such an initial estimator.

  3. 3.

We found that a non-trivial integration of the Adaptive Lasso and the Transfer
Lasso provides a combination of the benefits of both. The superiority of this
integration was shown by asymptotic analysis and empirical simulations.

  4. 4.

We comprehensively analyzed the relation between hyperparameters and
asymptotic properties and drew phase diagrams representing them. Figure 2
illustrates the phase diagram of the Adaptive Lasso and the Transfer Lasso,
and Figure 3 illustrates the phase diagram of the proposed method. These
theoretical results were reproduced empirically by numerical simulations in
Figure 5.

This paper discusses the above research questions in the following
organization. First, we review the asymptotic properties of the Lasso and
Adaptive Lasso (Section 2). Then, we define a setup for our analysis and
theoretically analyze the asymptotic properties of the Adaptive Lasso and the
Transfer Lasso (Section 3). This elucidates the advantages and disadvantages
of each method. Furthermore, to compensate for their disadvantages and to
reconcile their advantages, we propose a novel method, which effectively
integrates both of them (Section 4). We demonstrate its superiority through
theoretical analysis. We then compare the Adaptive Lasso, the Transfer Lasso,
and their integrated method through numerical experiments (Section 5).
Finally, we provide additional discussion and conclusions (Sections 6 and 7).

### Notations

Consider a vector $v\in\mathbb{R}^{p}$. We denote the element-wise absolute
vector by $|v|$, with the $j$-th element given by $|v_{j}|$. The sign vector
is represented as ${\rm sgn}(v)$, with its elements being $1$ for $v_{j}>0$,
$-1$ for $v_{j}<0$, and $0$ for $v_{j}=0$. The support set of $v$ is denoted
as $\operatorname{supp}(v)$ and defined as
$\operatorname{supp}(v):=\\{j\in\\{1,\dots,p\\}|v_{j}\neq 0\\}$. The
$\ell_{q}$-norm of $v$ is expressed as
$\|v\|_{q}=(\sum_{j=1}^{p}|v_{j}|^{q})^{1/q}$.

For a matrix $M\in\mathbb{R}^{p\times p}$, we use $M\succeq O$ for a positive
semi-definite matrix and $M\succ O$ for a positive definite matrix, implying
$v^{\top}Mv\geq 0$ for all $v\in\mathbb{R}^{p}$ and $v^{\top}Mv>0$ for all
non-zero $v\in\mathbb{R}^{p}$, respectively.

Given a subset $S$ of $\\{1,\dots,p\\}$, we denote its cardinality as $|S|$,
and the complement set as $S^{c}=\\{1,\dots,p\\}\backslash S$. The vector
$v_{S}$ represents $v$ restricted to the index set $S$. The matrix
$M_{S_{1}S_{2}}$ denotes the submatrix with row indices in $S_{1}$ and column
indices in $S_{2}$.

For sequences $a_{n}$ and $b_{n}$, we use $a_{n}=O(b_{n})$ to indicate that
$|a_{n}/b_{n}|$ converges to a finite value, and $a_{n}=o(b_{n})$ to signify
$|a_{n}/b_{n}|$ converging to zero as $n\to\infty$.

##  2 Literature Review

We review some asymptotic properties for the Lasso and the Adaptive Lasso
based on [5] and [21], and then present other related studies. All of the
proofs in this section are essentially the same as those in [5] and [21], but
for the sake of readability, we provide them in Appendix B.1.

We make the following assumption throughout this paper as in [5, 21].

######  Assumption 2.1.

(5) $\displaystyle C_{n}:=\frac{1}{n}X^{\top}X\rightarrow C\succ
O\leavevmode\nobreak\ (n\rightarrow\infty),$ (6)
$\displaystyle\frac{1}{n}\max_{i}\|x_{i}\|_{2}^{2}\rightarrow
0\leavevmode\nobreak\ (n\rightarrow\infty).$

Let $W$ be a random variable of a Gaussian distribution with mean $0$ and
covariance $\sigma^{2}C$, that is, $W\sim\mathcal{N}(0,\sigma^{2}C)$.

###  2.1 Asymptotic Properties for the Lasso

The Lasso is given by (2). According to [5, 21], several asymptotic properties
have been obtained for the Lasso: consistency (Lemma 2.2 and Corollary 2.3),
convergence rate (Lemma 2.4, Corollary 2.5, Lemma 2.7, and Corollary 2.8), and
variable selection consistency (Lemma 2.6).

######  Lemma 2.2 (Theorem 1 in [5] and Lemma 1 in [21]).

If $\lambda_{n}/n\rightarrow\lambda_{0}\geq 0$, then

(7)
$\displaystyle{\hat{\beta}}_{n}^{\mathcal{L}}\rightarrow^{p}\operatorname*{argmin}_{\beta}\left\\{(\beta-\beta^{*})^{\top}C(\beta-\beta^{*})+\lambda_{0}\sum_{j}|\beta_{j}|\right\\}.$

######  Corollary 2.3 (Consistency for Lasso).

If $\lambda_{n}=o(n)$, then ${\hat{\beta}}_{n}^{\mathcal{L}}$ is consistent.

######  Lemma 2.4 (Theorem 2 in [5] and Lemma 2 in [21]).

If $\lambda_{n}/\sqrt{n}\rightarrow\lambda_{0}\geq 0$, then

(8) $\displaystyle\sqrt{n}({\hat{\beta}}_{n}^{\mathcal{L}}-\beta^{*})$
$\displaystyle\overset{d}{\rightarrow}$
$\displaystyle\operatorname*{argmin}_{u}\left\\{u^{\top}Cu-2u^{\top}W+\lambda_{0}\sum_{j}\left(u_{j}\operatorname{sgn}(\beta^{*}_{j})I(\beta^{*}_{j}\neq
0)+|u_{j}|I(\beta^{*}_{j}=0)\right)\right\\}.$

######  Corollary 2.5 ($\sqrt{n}$-consistency for Lasso).

If $\lambda_{n}=O(\sqrt{n})$, then ${\hat{\beta}}_{n}^{\mathcal{L}}$ is
$\sqrt{n}$-consistent.

######  Lemma 2.6 (Inconsistent Variable Selection; Proposition 1 in [21]).

Let $\hat{S}_{n}^{\mathcal{L}}:=\\{j:{\hat{\beta}}_{j}^{\mathcal{L}}\neq
0\\}$. If $\lambda_{n}/\sqrt{n}\rightarrow\lambda_{0}\geq 0$, then

(9) $\displaystyle\limsup_{n\to\infty}P(\hat{S}_{n}^{\mathcal{L}}=S)\leq c<1$

where $c$ is a constant.

######  Lemma 2.7 (Lemma 3 in [21]).

If $\lambda_{n}/n\rightarrow 0$ and $\lambda_{n}/\sqrt{n}\rightarrow\infty$,
then

(10)
$\displaystyle\frac{n}{\lambda_{n}}({\hat{\beta}}_{n}^{\mathcal{L}}-\beta^{*})\overset{d}{\rightarrow}\operatorname*{argmin}_{u}\left\\{u^{\top}Cu+\sum_{j=1}^{p}\left(u_{j}\operatorname{sgn}(\beta^{*}_{j})I(\beta^{*}_{j}\neq
0)+|u_{j}|I(\beta^{*}_{j}=0)\right)\right\\}.$

######  Corollary 2.8 (Slower Rate Consistency for Lasso).

If $\lambda_{n}/n\rightarrow 0$ and $\lambda_{n}/\sqrt{n}\rightarrow\infty$,
then the convergence rate of ${\hat{\beta}}_{n}^{\mathcal{L}}$ is slower than
$\sqrt{n}$.

We first obtain a convergence result for $\lambda_{n}=O(n)$ (Lemma 2.2). If
$\lambda_{n}=o(n)$, then we have consistency for the Lasso (Corollary 2.3).
Although $\lambda_{n}=o(n)$ is sufficient for consistency, it is not always
$\sqrt{n}$-consistent. We obtain an asymptotic distribution for
$\lambda_{n}=O(\sqrt{n})$ (Lemma 2.4). This implies $\sqrt{n}$-consistency for
the Lasso (Corollary 2.5). Unfortunately, $\lambda_{n}=O(\sqrt{n})$ leads to
inconsistent variable selection (Lemma 2.6). This implies that
$\lambda_{n}=O(\sqrt{n})$ achieves $\sqrt{n}$-consistency but inconsistent
variable selection for the Lasso. In contrast, if $\lambda_{n}$ is greater
than $O(\sqrt{n})$ and $\lambda_{n}=o(n)$, we obtain an asymptotic
distribution (Lemma 2.7). This implies that the convergence rate is slower
than $\sqrt{n}$ (Corollary 2.8), although it can be a consistent variable
selection under the incoherence conditions [21, 20].

Figure 1 (left) summarizes the asymptotic properties for the Lasso. It cannot
simultaneously achieve both $\sqrt{n}$-consistent estimation and consistent
variable selection. This is a major limitation of the Lasso and is the
motivation to develop the Adaptive Lasso.

###  2.2 Asymptotic Properties for Adaptive Lasso

Adaptive Lasso is given by (3). It is known that the Adaptive Lasso has the
so-called “oracle property” [21].

######  Lemma 2.9 (Oracle Property for Adaptive Lasso; Theorem 2 in [21]).

Suppose that ${\tilde{\beta}}_{n}$ is a $\sqrt{n}$-consistent estimator. If
$\lambda_{n}/\sqrt{n}\rightarrow 0$ and
$\lambda_{n}n^{(\gamma-1)/2}\rightarrow\infty$, then the Adaptive Lasso
estimator (2) satisfies the oracle property, that is, consistent variable
selection and $\sqrt{n}$-consistency with asymptotic normality:

(11) $\displaystyle\lim_{n\rightarrow\infty}P(\hat{S}_{n}^{\mathcal{A}}=S)=1,$
(12)
$\displaystyle\sqrt{n}({\hat{\beta}}_{S}^{\mathcal{A}}-\beta^{*}_{S})\overset{d}{\rightarrow}\mathcal{N}(0,\sigma^{2}C_{SS}^{-1}).$

###### Proof.

The proof is given in B.1.5. ∎

The oracle property demonstrates a clear advantage of the Adaptive Lasso over
the Lasso. With a $\sqrt{n}$-consistent initial estimator, the Adaptive Lasso
can simultaneously achieve both $\sqrt{n}$-consistent estimation and
consistent variable selection (Figure 1 right). Thus, the Adaptive Lasso
performs as well as if the true active variables were given in advance.

###  2.3 Other Related Work

Besides the Adaptive Lasso and the Transfer Lasso, several related methods
have been studied. In this subsection, we review related methods in three
categories: (I) methods with the oracle property similar to the Adaptive
Lasso, (II) methods with two-stage estimation to eliminate bias, similar to
the Adaptive Lasso, and (III) methods using the $\ell_{1}$ norm to transfer
knowledge about the source data, similar to the Transfer Lasso.

(I) The oracle property is known to hold not only for the Adaptive Lasso but
also for the SCAD [4] and MCP [19]. These methods use nonconvex
regularization, instead of using an initial estimator. Because of the
nonconvexity, the algorithm converges to a local minimum and the oracle
property holds only for some local minima or under restricted conditions. The
Adaptive Lasso, on the other hand, uses convex regularization and always
converges to a global minimum, although it requires an appropriate initial
estimator.

(II) The Lasso penalizes the $\ell_{1}$ norm of the parameters and thus
introduces a bias, leading to the failure of the oracle property. Several two-
step estimation methods have been proposed to eliminate the bias [13, 9, 3].
In [13], after the Lasso estimation in the first stage, the second stage is
another Lasso estimation using only the selected variables. In [9], after the
Lasso estimation in the first stage, the second stage is estimated by a linear
combination of the first stage estimator and the OLS estimator of the selected
variables. These methods are called Relaxed Lasso. [3] generalized these
refitting methods as “methods that minimize the loss function with
regularization and then decrease the loss function without regularization”.
Based on this idea, they developed several refitting methods.

(III) Regularization of $\ell_{1}$-norm between target and initial estimators
was proposed by [1, 11, 17] as well as the Transfer Lasso [16]. [1]
corresponds to the case where $\lambda_{n}=0$ in Transfer Lasso [16]. In the
TransLasso [11] and its GLM extension [17], two-stage estimation methods were
proposed for the case of multiple source data, where the initial estimator is
estimated using both the source and target data. The Transfer Lasso [16], in
contrast, is performed on target data using the initial estimator without the
need for source data.

##  3 Asymptotic Properties for Adaptive Lasso and Transfer Lasso

We will perform asymptotic analysis based on the following general settings
throughout this paper.

######  Assumption 3.1.

Let $m\geq 0$ be an integer satisfying $n/m\to r_{0}\geq 0$. The initial
estimator ${\tilde{\beta}}$ is a $\sqrt{m}$-consistent estimator and
$z:=\sqrt{m}({\tilde{\beta}}-\beta^{*})$ converges to some distribution.

Assumption 3.1 implies that the initial estimator is estimated on source data
of size $m$, and then the final estimator is estimated on target data of size
$n$ using the initial estimator. The case $m=n\leavevmode\nobreak\ (r_{0}=1)$
corresponds to the existing results for the Adaptive Lasso, whereas $m\gg
n\leavevmode\nobreak\ (r_{0}=0)$ corresponds to the typical transfer learning
setup. The source and target data are assumed to be independent of each other.
We also make assumption 2.1 in our analysis.

We note that the initial estimator ${\tilde{\beta}}$ is not a fixed
(deterministic) source parameter, but an estimator (random variable). This is
the same as the previous studies. The case where ${\tilde{\beta}}$ is fixed is
discussed in Appendix A.

###  3.1 Asymptotic Properties for Adaptive Lasso

We provide the property of the Adaptive Lasso for an initial estimator with
source data of size $m$. It is straightforward to extend the oracle property
for $\sqrt{n}$-consistent initial estimators (Lemma 2.9) to
$\sqrt{m}$-consistent initial estimators (Lemma 3.2).

######  Lemma 3.2 (Oracle Property for Adaptive Lasso with Different Sample
Size).

Suppose that ${\tilde{\beta}}$ is a $\sqrt{m}$-consistent estimator. If
$\lambda_{n}/\sqrt{n}\rightarrow 0$ and
$\lambda_{n}\sqrt{m^{\gamma}/n}\rightarrow\infty$, then the Adaptive Lasso
estimator (3) satisfies the oracle property, that is, consistent variable
selection and $\sqrt{n}$-consistency with asymptotic normality:

(13) $\displaystyle\lim_{n\rightarrow\infty}P(\hat{S}_{n}^{\mathcal{A}}=S)=1,$
(14)
$\displaystyle\sqrt{n}({\hat{\beta}}_{S}^{\mathcal{A}}-\beta^{*}_{S})\overset{d}{\rightarrow}\mathcal{N}(0,\sigma^{2}C_{SS}^{-1}).$

###### Proof.

The proof is given in B.2.1. ∎

Furthermore, we extensively analyze the convergence rate depending on the
hyperparameter $\lambda_{n}$. We obtain Theorem 3.3 and Corollary 3.4.

######  Theorem 3.3 (Asymptotic Distribution for Adaptive Lasso).

We have the following asymptotic distributions for the Adaptive Lasso
estimator (3).
(i) If $\sqrt{m^{\gamma}/n}\leavevmode\nobreak\ \lambda_{n}\to\lambda_{1}\geq
0$, then

(15)
$\displaystyle\sqrt{n}({\hat{\beta}}_{n}^{\mathcal{A}}-\beta^{*})\overset{d}{\to}\operatorname*{argmin}_{u}\left\\{u^{\top}Cu-2u^{\top}W+\sum_{j\in
S^{c}}\frac{\lambda_{1}}{|z_{j}|^{\gamma}}|u_{j}|\right\\}.$

(ii) If $\sqrt{m^{\gamma}/n}\leavevmode\nobreak\ \lambda_{n}\to\infty$ and
$\lambda_{n}/\sqrt{n}\to\lambda_{0}\geq 0$, then

(16)
$\displaystyle\sqrt{n}({\hat{\beta}}_{n}^{\mathcal{A}}-\beta^{*})\overset{d}{\to}\operatorname*{argmin}_{u\in\mathcal{U}}\left\\{u^{\top}Cu-2u^{\top}W+\sum_{j\in
S}\lambda_{0}\frac{\operatorname{sgn}(\beta^{*}_{j})}{|\beta^{*}_{j}|^{\gamma}}u_{j}\right\\},\leavevmode\nobreak\
\mathcal{U}:=\left\\{u\leavevmode\nobreak\ \middle|\leavevmode\nobreak\
u_{S^{c}}=0\right\\}.$

(iii) If $\lambda_{n}/\sqrt{n}\to\infty$ and $\lambda_{n}/n\to 0$, then

(17)
$\displaystyle\frac{n}{\lambda_{n}}({\hat{\beta}}_{n}^{\mathcal{A}}-\beta^{*})\overset{d}{\to}\operatorname*{argmin}_{u\in\mathcal{U}}\left\\{u^{\top}Cu+\sum_{j\in
S}\frac{\operatorname{sgn}(\beta^{*}_{j})}{|\beta^{*}_{j}|^{\gamma}}u_{j}\right\\},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\mathcal{U}:=\left\\{u\leavevmode\nobreak\ \middle|\leavevmode\nobreak\
u_{S^{c}}=0\right\\}.$

###### Proof.

This is a special case of Theorem 4.1 and the proof is the same as B.3.1. ∎

######  Corollary 3.4 (Convergence Rate for Adaptive Lasso).

We have the following convergence rates for the Adaptive Lasso estimator (3).

  1. $(i)$

If $\sqrt{m^{\gamma}/n}\leavevmode\nobreak\ \lambda_{n}\to\lambda_{1}\geq 0$,
then the convergence rate is $\sqrt{n}$.

  2. $(ii)$

If $\sqrt{m^{\gamma}/n}\leavevmode\nobreak\ \lambda_{n}\to\infty$ and
$\lambda_{n}/\sqrt{n}\to\lambda_{0}\geq 0$, then convergence rate is
$\sqrt{n}$.

  3. $(iii)$

If $\lambda_{n}/\sqrt{n}\to\infty$ and $\lambda_{n}/n\to 0$, then the
convergence rate is $n/\lambda_{n}$, which is slower than $\sqrt{n}$.

Lemma 3.2 shows that the oracle property still holds for $\sqrt{m}$-consistent
estimators, $\lambda_{n}/\sqrt{n}\rightarrow 0$, and
$\lambda_{n}\sqrt{m^{\gamma}/n}\rightarrow\infty$. In addition, Theorem 3.3
and Corollary 3.4 show that the convergence rate of the Adaptive Lasso
estimator is equal to $\sqrt{n}$ in the case (i) and (ii) and is less than
$\sqrt{n}$ in the case (iii). The condition of (ii) in Theorem 3.3 includes
the condition of the oracle property in Lemma 3.2. Figure 2 (left) illustrates
each hyperparameter region in Theorem 3.3 and Corollary 3.4.

These results imply both an advantage and a disadvantage. The advantage is
that the initial estimator does not necessarily require $\sqrt{n}$-consistent.
The Adaptive Lasso has the oracle property even when the source data is small
compared to the target data ($m\lesssim n$) and the initial estimator is less
than $\sqrt{n}$-consistent. The disadvantage of the Adaptive Lasso, however,
is that it does not take full advantage even when the sample size of the
source data is very large ($m\gg n$). This is because the convergence rate is
equal to $\sqrt{n}\leavevmode\nobreak\ (\ll\sqrt{m})$.

$\lambda_{n}$$n$$\sqrt{n/m^{\gamma}}$$\sqrt{n}$(iii)(ii)(i)

$\eta_{n}$$\lambda_{n}$$\sqrt{n}$$\sqrt{n}$$n$(i)(ii)(iii)

Figure 2:  Phase diagrams with $\lambda_{n}$ for the Adaptive Lasso in Lemma
3.2–Theorem 3.4 (left) and $\lambda_{n}$ and $\eta_{n}$ for the Transfer Lasso
in Theorem 3.5–Theorem 3.11 (right). The Adaptive Lasso has
$\sqrt{n}$-consistency in (i) and (ii) and active variable selection
consistency in (ii), but the convergence rate in (iii) is slower than
$\sqrt{n}$. The Transfer Lasso has convergence rates of $\sqrt{m}$,
$\sqrt{n}$, and $n/\lambda_{n}(<\sqrt{n})$ for (i), (ii), and (iii)
respectively. It has invariant variable selection consistency in (i) but does
not have active variable selection consistency in (i) and (ii).

###  3.2 Asymptotic Properties for Transfer Lasso

Now we consider the asymptotic properties of the Transfer Lasso. The Transfer
Lasso has two hyperparameters, $\lambda_{n}$ and $\eta_{n}$, and various
asymptotic properties appear depending on their values. We first obtain
several asymptotic distributions in Theorem 3.5 and convergence rate in
Corollary 3.6. The illustration of the division of cases is shown in Figure 2.

######  Theorem 3.5 (Asymptotic distribution for Transfer Lasso).

We have the following asymptotic distributions for the Transfer Lasso
estimator (4).
(i) If $\eta_{n}/\sqrt{n}\to\infty$ and $\lambda_{n}/\eta_{n}\to\rho_{0}$ with
$0\leq\rho_{0}<1$, then

(18)
$\displaystyle\sqrt{m}({\hat{\beta}}_{n}^{\mathcal{T}}-\beta^{*})\overset{d}{\to}z.$

(ii) If $\lambda_{n}/\sqrt{n}\to\lambda_{0}\geq 0$ and
$\eta_{n}/\sqrt{n}\to\eta_{0}\geq 0$, then

(19)
$\displaystyle\sqrt{n}\left({\hat{\beta}}_{n}^{\mathcal{T}}-\beta^{*}\right)$
$\displaystyle\overset{d}{\to}$
$\displaystyle\operatorname*{argmin}_{u}\left\\{u^{\top}Cu-2u^{\top}W+\lambda_{0}\left(\sum_{j\in
S}u_{j}\operatorname{sgn}(\beta^{*}_{j})+\sum_{j\in
S^{c}}\left|u_{j}\right|\right)+\eta_{0}\sum_{j=1}^{p}\left|u_{j}-\sqrt{r_{0}}z_{j}\right|\right\\}.$

(iii) If $\lambda_{n}/\sqrt{n}\to\infty$, $\lambda_{n}/n\to 0$, and
$\eta_{n}/\lambda_{n}\to\rho_{0}^{\prime}\geq 0$, then

(20)
$\displaystyle\frac{n}{\lambda_{n}}({\hat{\beta}}_{n}^{\mathcal{T}}-\beta^{*})\overset{d}{\to}\operatorname*{argmin}_{u}\left\\{u^{\top}Cu+\sum_{j\in
S}\left(u_{j}\operatorname{sgn}(\beta^{*}_{j})+\rho_{0}^{\prime}\left|u_{j}\right|\right)+\sum_{j\in
S^{c}}(1+\rho_{0}^{\prime})\left|u_{j}\right|\right\\}.$

###### Proof.

The proof is given in B.2.2 ∎

######  Corollary 3.6 (Convergence Rate for Transfer Lasso).

We have the following convergence rates for the Transfer Lasso estimator (4).

  1. $(i)$

If $\eta_{n}/\sqrt{n}\to\infty$ and $\lambda_{n}/\eta_{n}\to\rho_{0}$ with
$0\leq\rho_{0}<1$, then the convergence rate is $\sqrt{m}$.

  2. $(ii)$

If $\lambda_{n}/\sqrt{n}\to\lambda_{0}\geq 0$ and
$\eta_{n}/\sqrt{n}\to\eta_{0}\geq 0$, then the convergence rate is $\sqrt{n}$.

  3. $(iii)$

If $\lambda_{n}/\sqrt{n}\to\infty$, $\lambda_{n}/n\to 0$, and
$\eta_{n}/\lambda_{n}\to\rho_{0}^{\prime}$ with $0\leq\rho_{0}^{\prime}<1$,
then the convergence rate is $n/\lambda_{n}$, which is slower than $\sqrt{n}$.
On the other hand, if $\rho_{0}^{\prime}\geq 1$, then the convergence rate is
faster than $n/\lambda_{n}$.

Theorem 3.5 and Corollary 3.6 show that the Transfer Lasso estimators achieve
a convergence rate of $\sqrt{m}$ in the case (i). This is beneficial when
source data is large ($m\gg n$) and is an advantage for the Transfer Lasso
over the Adaptive Lasso.

Next, we provide the results of variable selection consistency. We first
define two types of variable selection consistency.

######  Definition 3.7 (Active Variable Selection Consistency).

We say that an estimator exhibits consistent active variable selection when it
estimates the true active variable to be nonzero and the true inactive
variable to be zero, that is,

(21) $\displaystyle P(\hat{S}_{n}=S)\to 1.$

Conversely, we say that an estimator is an inconsistent active variable
selection when this is not the case, that is,

(22) $\displaystyle\limsup_{n\to\infty}P(\hat{S}_{n}=S)\leq c<1.$

######  Definition 3.8 (Invariant Variable Selection Consistency).

We say that an estimator exhibits consistent invariant variable selection when
the true active variable remains invariant from the initial estimator, that
is,

(23) $\displaystyle P({\hat{\beta}}_{S}^{\mathcal{T}}={\tilde{\beta}}_{S})\to
1.$

“Active” and “invariant” variable selection consistency are different but
related concepts. The property of “active” variable selection consistency is
identical to ordinary variable selection consistency and is induced by the
sparsity of the estimator. This property gives a certain justification to
decision-making based on zero/non-zero estimators since it guarantees the
correct selection of non-zero variables. In contrast, “invariant” variable
selection consistency is a property unique to the Transfer Lasso and is
induced by the sparsity of the difference from the initial estimator. It
guarantees that the estimator of the true active variable remains unchanged,
while the inactive variable is allowed to change. Therefore, it gives a
justification to decision-making that focuses on change in the estimator. When
both active/invariant variable selection holds, the estimator is zero for the
true inactive variable and is the value of the initial estimator for the
active variable.

Now we give some results of active/invariant variable selection consistency
for the Transfer Lasso in Theorems 3.9, 3.10, and 3.11. We suppose that the
initial estimator ${\tilde{\beta}}$ does not hold consistent active variable
selection in the analyses of variable selection.

######  Theorem 3.9 (Inconsistent Active Variable Selection for Transfer
Lasso).

Suppose that ${\tilde{\beta}}$ is inconsistent with active variable selection.
For the cases (i) and (ii) in Theorem 3.5, the Transfer Lasso estimator (4)
yields inconsistent active variable selection, that is,

(24) $\displaystyle\limsup_{n\to\infty}P(\hat{S}_{n}^{\mathcal{T}}=S)\leq
c<1,$

where $c$ is a constant.

###### Proof.

The proof is given in B.2.3. ∎

######  Theorem 3.10 (Consistent Invariant Variable Selection for Transfer
Lasso).

Suppose that ${\tilde{\beta}}$ is inconsistent with active variable selection.
For the case (i) in Theorem 3.5, the Transfer Lasso estimator (4) yields
consistent invariant variable selection, that is,

(25) $\displaystyle P({\hat{\beta}}_{S}^{\mathcal{T}}={\tilde{\beta}}_{S})\to
1.$

###### Proof.

The proof is given in B.2.4. ∎

######  Theorem 3.11 (Inconsistent Invariant Variable Selection for Transfer
Lasso).

Suppose that ${\tilde{\beta}}$ is inconsistent with active variable selection.
For the case (ii) in Theorem 3.5, the Transfer Lasso estimators (4) yield
inconsistent invariant variable selection, that is,

(26)
$\displaystyle\limsup_{n\to\infty}P({\hat{\beta}}_{S}^{\mathcal{T}}={\tilde{\beta}}_{S})\leq
c<1.$

where $c$ is a constant.

###### Proof.

The proof is given in B.2.5. ∎

Theorems 3.9, 3.10, and 3.11 unveil the benefits and drawbacks of the Transfer
Lasso. Theorem 3.9 implies that the $\sqrt{m}$-consistent region (i) does not
hold active variable selection consistency. The $\sqrt{n}$-consistent region
(ii) does not hold as well. This is a disadvantage for the Transfer Lasso. On
the other hand, Theorem 3.10 indicates that the Transfer Lasso in the case (i)
has a property of consistent invariant variable selection, which the Adaptive
Lasso does not have. Theorem 3.11 implies that the estimators are inconsistent
in terms of invariant variable selection in the case (ii).

As shown in Figure 2, the Transfer Lasso cannot simultaneously achieve
$\sqrt{m}$-consistency and consistent active/invariant variable selection in
the regions (i), (ii), and (iii). This is why we explore a new methodology in
the next section. We note that in regions other than (i), (ii), and (iii)
(e.g., boundary regions), the asymptotic property is unclear. Appendix A.3
contains additional results for boundary regions. At the very least, the above
results imply that $\sqrt{m}$-consistency and consistent active/invariant
variable selection are incompatible in most regions for the Transfer Lasso.

##  4 Beyond Adaptive Lasso and Transfer Lasso

The Adaptive Lasso and the Transfer Lasso have their advantages and
disadvantages, as seen in the previous section. The Adaptive Lasso achieves
both $\sqrt{n}$-consistency and consistent variable selection for $m\leq n$,
but its convergence rate is $\sqrt{n}(\ll\sqrt{m})$ for $m\gg n$. The Transfer
Lasso, on the other hand, achieves a convergence rate of $\sqrt{m}$ for $m\gg
n$, but it results in inconsistent variable selection. Are there any ways to
combine their benefits and compensate for their drawbacks?

###  4.1 Adaptive Transfer Lasso: A Non-Trivial Integration

To exploit their benefits and compensate for their drawbacks, we integrate the
ideas of the Adaptive Lasso and the Transfer Lasso. We propose a novel method
using the initial estimator ${\tilde{\beta}}$ as

(27) $\displaystyle\hat{\beta}_{n}^{\\#}=\operatorname*{argmin}_{\beta}$
$\displaystyle\left\\{\frac{1}{n}\|y-X\beta\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}v_{j}|\beta_{j}|+\frac{\eta_{n}}{n}\sum_{j}w_{j}\left|\beta_{j}-{\tilde{\beta}}_{j}\right|\right\\},$
(28) $\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
v_{j}:=\frac{1}{|{\tilde{\beta}}_{j}|^{\gamma_{1}}},\leavevmode\nobreak\
w_{j}:=|{\tilde{\beta}}_{j}|^{\gamma_{2}},$

where $\gamma_{1}\geq 0$ and $\gamma_{2}\geq 0$ are new hyperparameters. We
denote $\hat{S}_{n}^{\\#}:=\\{j:{\hat{\beta}}_{j}^{\\#}\neq 0\\}$. The weight
$v_{j}=1/|{\tilde{\beta}}_{j}|^{\gamma_{1}}$ is the same as that of the
Adaptive Lasso, whereas the term $w_{j}=|{\tilde{\beta}}_{j}|^{\gamma_{2}}$ is
a new non-trivial part. Because $w_{j}\to 0$ as ${\tilde{\beta}}_{j}\to 0$,
the effect of transfer learning from the initial estimator disappears for
inactive parameters. We call this method Adaptive Transfer Lasso because it is
a generalization of the Adaptive Lasso and the Transfer Lasso. Indeed, if
$\eta_{n}=0$, then it reduces to the Adaptive Lasso, and if
$\gamma_{1}=\gamma_{2}=0$, then it reduces to the Transfer Lasso.

###  4.2 Asymptotic Properties for Adaptive Transfer Lasso

We present the asymptotic properties of the Adaptive Transfer Lasso. The
assumptions are the same as for the Adaptive Lasso and the Transfer Lasso. To
derive the asymptotic distribution and convergence rate, we need a more
detailed case analysis than before. The illustration of the division of cases
is shown in Figure 3.

######  Theorem 4.1 (Asymptotic Distribution for Adaptive Transfer Lasso).

We have the following asymptotic distributions for the Adaptive Transfer Lasso
estimator (27).
(i) If $\eta_{n}/\sqrt{nm^{\gamma_{2}}}\to\infty$ and
$\eta_{n}/\sqrt{m^{\gamma_{1}+\gamma_{2}}}\lambda_{n}\to\infty$, then

(29)
$\displaystyle\sqrt{m}({\hat{\beta}}_{n}^{\\#}-\beta^{*})\overset{d}{\to}z.$

(ii) If $\sqrt{m^{\gamma_{1}}/n}\leavevmode\nobreak\ \lambda_{n}\to\infty$,
$\eta_{n}/\sqrt{n}\to\infty$, $\eta_{n}/\lambda_{n}\to\infty$,
$\eta_{n}/\sqrt{m^{\gamma_{1}+\gamma_{2}}}\lambda_{n}\to 0$, and
$\sqrt{m^{\gamma_{1}}}\lambda_{n}/\eta_{n}\to\rho_{0}\geq 0$, then

(30)
$\displaystyle\sqrt{m}({\hat{\beta}}_{n}^{\\#}-\beta^{*})\overset{d}{\to}\begin{dcases}0&\text{for}\leavevmode\nobreak\
j\in S^{c},\\\ z_{j}&\text{for}\leavevmode\nobreak\ j\in S.\end{dcases}$

(iii) If $\sqrt{m^{\gamma_{1}}/n}\leavevmode\nobreak\
\lambda_{n}\to\lambda_{1}\geq 0$ and $\eta_{n}/\sqrt{n}\to\eta_{0}\geq 0$,
then

(31) $\displaystyle\sqrt{n}({\hat{\beta}}_{n}^{\\#}-\beta^{*})$ (32)
$\displaystyle\overset{d}{\to}$
$\displaystyle\operatorname*{argmin}_{u}\left\\{u^{\top}Cu-2u^{\top}W+\sum_{j\in
S^{c}}\frac{\lambda_{1}}{|z_{j}|^{\gamma_{1}}}|u_{j}|+\sum_{j\in
S}\eta_{0}\left|\beta^{*}_{j}\right|^{\gamma_{2}}\left|u_{j}-\sqrt{r_{0}}z_{j}\right|\right\\}.$

(iv) If $\sqrt{m^{\gamma_{1}}/n}\leavevmode\nobreak\
\lambda_{n}\to\lambda_{1}\geq 0$, $\eta_{n}/\sqrt{n}\to\infty$, and
$\eta_{n}/\sqrt{nm^{\gamma_{2}}}\to\eta_{1}\geq 0$, then

(33) $\displaystyle\sqrt{n}({\hat{\beta}}_{n}^{\\#}-\beta^{*})$ (34)
$\displaystyle\overset{d}{\to}$
$\displaystyle\operatorname*{argmin}_{u\in\mathcal{U}}\left\\{u^{\top}Cu-2u^{\top}W+\sum_{j\in
S^{c}}\left(\frac{\lambda_{1}}{|z_{j}|^{\gamma_{1}}}\left|u_{j}\right|+\eta_{1}|z_{j}|^{\gamma_{2}}|u_{j}-\sqrt{r_{0}}z_{j}|\right)\right\\},$
(35) $\displaystyle\mathcal{U}:=\left\\{u\leavevmode\nobreak\
\middle|\leavevmode\nobreak\ u_{S}=r_{0}z_{S}\right\\}.$

(v) If $\sqrt{m^{\gamma_{1}}/n}\leavevmode\nobreak\ \lambda_{n}\to\infty$,
$\lambda_{n}/\sqrt{n}\to\lambda_{0}\geq 0$, and
$\eta_{n}/\sqrt{n}\to\eta_{0}\geq 0$, then

(36) $\displaystyle\sqrt{n}({\hat{\beta}}_{n}^{\\#}-\beta^{*})$ (37)
$\displaystyle\overset{d}{\to}$
$\displaystyle\operatorname*{argmin}_{u\in\mathcal{U}}\left\\{u^{\top}Cu-2u^{\top}W+\sum_{j\in
S}\left(\lambda_{0}\frac{\operatorname{sgn}(\beta^{*}_{j})}{|\beta^{*}_{j}|^{\gamma_{1}}}u_{j}+\eta_{0}\left|\beta^{*}_{j}\right|^{\gamma_{2}}\left|u_{j}-\sqrt{r_{0}}z_{j}\right|\right)\right\\},$
(38) $\displaystyle\mathcal{U}:=\left\\{u\leavevmode\nobreak\
\middle|\leavevmode\nobreak\ u_{S^{c}}=0\right\\}.$

(vi) If $\lambda_{n}/\sqrt{n}\to\infty$, $\lambda_{n}/n\to 0$, and
$\lambda_{n}/\eta_{n}\to\infty$, then

(39)
$\displaystyle\frac{n}{\lambda_{n}}({\hat{\beta}}_{n}^{\\#}-\beta^{*})\overset{d}{\to}\operatorname*{argmin}_{u\in\mathcal{U}}\left\\{u^{\top}Cu+\sum_{j\in
S}\frac{\operatorname{sgn}(\beta^{*}_{j})}{|\beta^{*}_{j}|^{\gamma_{1}}}u_{j}\right\\},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\mathcal{U}:=\left\\{u\leavevmode\nobreak\ \middle|\leavevmode\nobreak\
u_{S^{c}}=0\right\\}.$

###### Proof.

The proof is given in B.3.1 ∎

######  Corollary 4.2 (Convergence Rate for Adaptive Transfer Lasso).

We have the following convergence rates for the Adaptive Transfer Lasso
estimator (27).

  1. $(i)$

$\eta_{n}/\sqrt{nm^{\gamma_{2}}}\to\infty$ and
$\eta_{n}/\sqrt{m^{\gamma_{1}+\gamma_{2}}}\lambda_{n}\to\infty$, then the
convergence rate is $\sqrt{m}$.

  2. $(ii)$

$\sqrt{m^{\gamma_{1}}/n}\leavevmode\nobreak\ \lambda_{n}\to\infty$,
$\eta_{n}/\sqrt{n}\to\infty$, $\eta_{n}/\lambda_{n}\to\infty$, and
$\eta_{n}/\sqrt{m^{\gamma_{1}+\gamma_{2}}}\lambda_{n}\to 0$, then then the
convergence rate is $\sqrt{m}$.

  3. $(iii)$

If $\sqrt{m^{\gamma_{1}}/n}\leavevmode\nobreak\ \lambda_{n}\to\lambda_{1}\geq
0$ and $\eta_{n}/\sqrt{n}\to\eta_{0}\geq 0$, then the convergence rate is
$\sqrt{n}$.

  4. $(iv)$

If $\sqrt{m^{\gamma_{1}}/n}\leavevmode\nobreak\ \lambda_{n}\to\lambda_{1}\geq
0$, $\eta_{n}/\sqrt{n}\to\infty$, and
$\eta_{n}/\sqrt{nm^{\gamma_{2}}}\to\eta_{1}\geq 0$, then convergence rate is
$\sqrt{n}$.

  5. $(v)$

If $\sqrt{m^{\gamma_{1}}/n}\leavevmode\nobreak\ \lambda_{n}\to\infty$,
$\lambda_{n}/\sqrt{n}\to\lambda_{0}\geq 0$, and
$\eta_{n}/\sqrt{n}\to\eta_{0}\geq 0$, then convergence rate is $\sqrt{n}$.

  6. $(vi)$

If $\lambda_{n}/\sqrt{n}\to\infty$, $\lambda_{n}/n\to 0$, and
$\lambda_{n}/\eta_{n}\to\infty$, then the convergence rate is $n/\lambda_{n}$,
which is slower than $\sqrt{n}$.

$\eta_{n}$$\lambda_{n}$$\sqrt{n}$$\sqrt{nm^{\gamma_{2}}}$$\sqrt{n}$$\sqrt{n/m^{\gamma_{1}}}$$n$(i)(ii)(iii)(iv)(v)(vi)

$\eta_{n}$$\lambda_{n}$$\sqrt{n}$$\sqrt{nm^{\gamma_{2}}}$$\sqrt{n}$$\sqrt{n/m^{\gamma_{1}}}$$n$(ii)(v)(vi)

$\eta_{n}$$\lambda_{n}$$\sqrt{n}$$\sqrt{nm^{\gamma_{2}}}$$\sqrt{n}$$\sqrt{n/m^{\gamma_{1}}}$$n$(i)(ii)(iv)

Figure 3:  Phase diagrams of convergence rate (top) and active/invariant
variable selection (bottom left/right) with $\lambda_{n}$ and $\eta_{n}$ for
the Adaptive Transfer Lasso in Theorems 4.1, 4.3, 4.4, and Corollary 4.2. They
are $\sqrt{m}$-consistent in (i - ii), $\sqrt{n}$-consistent in (iii - v), and
sub-$\sqrt{n}$-consistent in (vi). They yield consistent active variable
selection in (ii), (v), and (vi) (left), while consistent invariant variable
selection in (i), (ii), and (iv) (right). Estimators in (ii) satisfy
$\sqrt{m}$-consistency and active/invariant variable selection consistency.

Theorem 4.1 and Corollary 4.2 show that the Adaptive Transfer Lasso achieves a
convergence rate of $\sqrt{m}$ in the case (i) and (ii). This property is
inherited from the Transfer Lasso. The asymptotic distribution for (i) is the
same as the initial estimator. On the other hand, the asymptotic distribution
for (ii) is remarkable. The distribution is the same as the initial estimator
for the active variables, whereas is zero for the inactive variables. This
implies that inactive parameters shrink to zero quickly.

We also provide the results of active/invariant variable selection consistency
for the Adaptive Transfer Lasso.

######  Theorem 4.3 (Consistent Active Variable Selection for Adaptive
Transfer Lasso).

For the cases (ii), (v), and (vi) in Theorem 4.1, the Adaptive Transfer Lasso
yields consistent active variable selection, that is,

(40) $\displaystyle P(\hat{S}_{n}^{\\#}=S)\to 1.$

###### Proof.

The proof is given in B.3.2. ∎

######  Theorem 4.4 (Consistent Invariant Variable Selection for Adaptive
Transfer Lasso).

For the cases (i), (ii), and (iv) in Theorem 4.1, the Adaptive Transfer Lasso
yields consistent invariant variable selection, that is,

(41) $\displaystyle P({\hat{\beta}}_{S}={\tilde{\beta}}_{S})\to 1.$

###### Proof.

The proof is given in B.3.3. ∎

Theorems 4.3 and 4.4 imply that both active/invariant variable selection
consistency hold in the case (ii). Hence, we have the following corollary.

######  Corollary 4.5 (Oracle Region for Adaptive Transfer Lasso).

For the case (ii) in Theorem 4.1, the Adaptive Transfer Lasso estimator
satisfies

  * •

$\sqrt{m}$-consistent: $\sqrt{m}({\hat{\beta}}_{n}^{\\#}-\beta^{*})$ converges
to some distributions,

  * •

consistent active variable selection: $\hat{S}_{n}^{\\#}=S$ with probability
tending to $1$,

  * •

consistent invariant variable selection:
${\hat{\beta}}_{S}={\tilde{\beta}}_{S}$ with probability tending to $1$.

Corollary 4.5 shows that the Adaptive Transfer Lasso incorporates the
advantages of both the Adaptive Lasso and the Transfer Lasso. The
hyperparameters $\gamma_{1}$ and $\gamma_{2}$ play a crucial role in this
property. If $\gamma_{1}=\gamma_{2}=0$, then the region (ii) disappears and it
reduces to the Transfer Lasso. If either $\gamma_{1}$ or $\gamma_{2}$ is
positive, then the region (ii) appears and it holds $\sqrt{m}$-consistency and
active/invariant variable selection consistency. Both $\gamma_{1}$ and
$\gamma_{2}$ contribute to expanding the region (ii). One possible advantage
of using $\gamma_{2}>0$ compared to $\gamma_{1}>0$ is that it is stable since
there is no division by zero even when the initial estimator is sparse and the
values are exactly zero.

Figure 3 are the phase diagrams that demonstrate the relation between
hyperparameters $(\lambda_{n},\eta_{n})$ and their asymptotic properties for
the Adaptive Transfer Lasso. We see that the region (ii) is the intersection
of the part with $\sqrt{m}$-consistency and the part with active/invariant
variable selection consistency. Such a region exists neither in the Adaptive
Lasso nor in the Transfer Lasso.

##  5 Empirical Results

We first empirically validate the theoretical properties. We then compare the
performance of various methods through extensive simulations. Appendix C
provides additional experimental results111The codes will be available at a
later date..

###  5.1 Empirical Validation of Theory

In this subsection, we empirically validate the theoretical results for the
Transfer Lasso and the Adaptive Transfer Lasso.

We first evaluated the $\ell_{2}$ norm of the estimation error with respect to
sample size. Theoretically, the convergence rate is $\sqrt{m}$, $\sqrt{n}$,
and so on, depending on the hyperparameters. Assuming the convergence rate is
$l(n)$, we have $E[\log\|{\hat{\beta}}-\beta^{*}\|_{2}]=\text{const.}-\log
l(n)$, since $l(n)\|{\hat{\beta}}-\beta^{*}\|_{2}$ converges to some
distribution. Therefore, by drawing a graph with
$E[\log\|{\hat{\beta}}-\beta^{*}\|_{2}]$ on the vertical axis and $\log n$ on
the horizontal axis, the convergence rate can be empirically calculated from
its slope. Assuming $m=n^{2}$, the slope is $-1/2$ when $\sqrt{n}$-consistent,
and $-1$ when $\sqrt{m}$-consistent.

We generated data by
$y_{i}=x_{i}^{\top}\beta^{*}+\varepsilon_{i}\leavevmode\nobreak\
(i=1,\dots,n)$ where
$x_{i}(\in\mathbb{R}^{10})\overset{i.i.d.}{\sim}\mathcal{N}(0,\Sigma)$,
$\Sigma_{jk}=0.5^{|j-k|}$,
$\varepsilon_{i}\overset{i.i.d.}{\sim}\mathcal{N}(0,\sigma^{2})$, $\sigma=1$,
and $\beta^{*}=[3,1.5,0,0,2,0,0,\dots,0]^{\top}(\in\mathbb{R}^{10})$ (as in
[21]). We generated source data of size $m$ and target data of size $n$ with
$m=n^{2}$ and $n=20,50,100,200,500,1000,2000,5000$. The initial estimators
were obtained by the ordinary least squares using source data. The
hyperparameters for each method were determined as follows according to
Figures 1, 2, and 3.

  * •

Lasso: $\lambda_{n}=n^{1/4}$ (i) and $\lambda_{n}=n^{3/4}$ (ii).

  * •

Adaptive Lasso: $\gamma=1$. $\lambda_{n}=n^{-1}$ (i), $n^{1/4}$ (ii), and
$n^{3/4}$ (iii).

  * •

Transfer Lasso: $(\lambda_{n},\eta_{n})=(n^{1/2},n^{3/4})$ (i),
$(n^{1/4},n^{1/4})$ (ii), and $(n^{3/4},n^{1/2})$ (iii).

  * •

Adaptive Transfer Lasso: $\gamma_{1}=\gamma_{2}=1$.
$(\lambda_{n},\eta_{n})=(n^{-1/2},n^{2})$ (i), $(n^{1/2},n^{3/2})$ (ii),
$(n^{-1},n^{1/4})$ (iii), $(n^{-1},n)$ (iv), $(1,n^{1/4})$ (v), and
$(n^{3/4},n^{1/2})$ (vi).

We performed each experiment ten times and evaluated their averages and
standard errors.

Figure 4: $\ell_{2}$ estimation errors for the Lasso (top left), the Adaptive
Lasso (top right), the Transfer Lasso (bottom left), and the Adaptive Transfer
Lasso (bottom right) with respect to sample size. The convergence rates of the
Transfer Lasso in the region (i) and the Adaptive Transfer Lasso in the region
(i) and (ii) are $\sqrt{m}$ (the slopes are $-1$), whereas the others are
$\sqrt{n}$ or less (the slope are $-1/2$ or greater).

Figure 4 shows the $\ell_{2}$ estimation errors for the Lasso, the Adaptive
Lasso, the Transfer Lasso, and the Adaptive Transfer Lasso with respect to
sample size. The slopes of the Transfer Lasso in the region (i) and the
Adaptive Transfer Lasso in the region (i) and (ii) are $-1$, indicating that
the convergence rate was $n=\sqrt{m}$. For the other methods or regions, the
slopes are $-0.5$ or greater, which confirms that the convergence rate is
$\sqrt{n}$ or less. These results were fully consistent with Theorems 3.5,
4.1, and 4.3.

We can observe two potential advantages of Adaptive Transfer Lasso. First,
although the convergence rate (for $n\geq 500$) is $\sqrt{n}$ in regions (v)
and (vi), the estimation error is on the line of the convergence rate
$\sqrt{m}$ for $n<500$. In other words, even in regions where the convergence
rate is $\sqrt{n}$, the estimation error can be reduced when the sample size
is small. Second, the estimation error is consistently smaller in the region
(ii) than in the region (i), although the convergence rates are comparable
between the two regions. This might be because the estimator in (i) is more
likely to be perfectly matched to the initial estimator, whereas the estimator
in (ii) is more likely to be matched to the initial estimator for active
variables, but not for the inactive variables, and is more likely to be zero.

Having found that the convergence rate can be empirically evaluated
accurately, we next empirically drew phase diagrams for the Transfer Lasso and
the Adaptive Transfer Lasso as in Figure 3. The experimental setup was the
same as in the previous subsection and $m=n^{2}$. The hyperparameters
$\lambda_{n}$ and $\eta_{n}$ were set to $n^{\delta}$ with
$\delta=-2,-1.75,-1.5,\dots,1.75,2$, respectively. The convergence rates were
calculated from the slopes of the $\ell_{2}$ errors for $n=1000$ and $n=5000$.
We plotted the exponential parts of $n$ in the convergence rates, taking the
value $1$ if $\sqrt{m}$-consistent and $0.5$ if $\sqrt{n}$-consistent. Active
variable selection consistency was evaluated as the ratio of correctly
estimated zeros/non-zeros among all variables for $n=5000$. Invariant variable
selection consistency was evaluated by the ratio of variables that did not
change from the initial estimator among the active variables for $n=5000$.

(i)(ii)(iii)

(i)(ii)(iii)(iv)(v)(vi)

(i)(ii)(iii)

(i)(ii)(iii)(iv)(v)(vi)

(i)(ii)(iii)

(i)(ii)(iii)(iv)(v)(vi)

Figure 5: $\log$–$\log$ phase diagrams of convergence rate (top), active
variable selection ratio (middle), and invariant variable selection ratio
(bottom) for the Transfer Lasso (left) and the Adaptive Transfer Lasso
(right). These empirical results confirm the theoretical results of Figure 2
and 3.

Figure 5 illustrates the empirical phase diagrams of $\log$–$\log$ scale for
the Transfer Lasso and the Adaptive Transfer Lasso. As Theorems 3.5–3.11
suggest, the Transfer Lasso achieves both $\sqrt{m}$-consistency and invariant
variable selection consistency in the lower right region (i), but does not
have active variable selection consistency. Other regions also do not satisfy
these properties simultaneously. On the other hand, in the Adaptive Transfer
Lasso, the upper right region (ii) satisfies the properties of
$\sqrt{m}$-consistency and active/invariant variable selection consistency.
The empirical convergence rates and active/invariant variable selection ratios
well reproduce Theorems 4.1, 4.3, and 4.4 in other regions as well. These
empirical results confirm the theoretical results (Theorems 3.5–4.4, Figures
1–3).

###  5.2 Empirical Comparison of Methods

In this subsection, we compare the methods in various experimental settings
based on hyperparameter determination by cross-validation. The experimental
settings include various source/target data sample sizes, number of
dimensions, signal-to-noise ratios, and initial estimators. We mainly
considered two cases: one with a large amount of source data and the other
with the same amount of source data as the target data.

First, we suppose that we have a large amount of source data and its sample
size is $m=10000$. The simulation setting follows the previous subsections. We
used $\sigma=1,3,6,10$; $p=10,20,50,100$; and
$n=10,20,50,100,200,\dots,5000,10000$.

Initial estimators were obtained by the Lasso because the number of dimensions
$p$ can be greater than sample size $n$ in this experiment. We compared other
initial estimators including Ridge, Ridgeless [2, 8], and Lassoless [14, 12]
in Appendix C.2. The search spaces were $\gamma=0.5,1,2$ for the Adaptive
Lasso; $\alpha:=\lambda_{n}/(\lambda_{n}+\eta_{n})=0.75,0.5,0.25$ for the
Transfer Lasso; and $(\gamma_{1},\gamma_{2})=(0.5,0.5),(1,1),(2,2)$, and
$\alpha:=\lambda_{n}/(\lambda_{n}+\eta_{n})=0.75,0.5,0.25$ for the Adaptive
Transfer Lasso. The hyperparameter $\lambda_{n}$ was determined by 10-fold
cross validation with $\lambda_{\min}/\lambda_{\max}=10^{-6}$ where
$\lambda_{\max}$ is automatically determined by Theorem 4 in [16]. If
$|{\tilde{\beta}}_{j}|\leq 10^{-3}$, then we set
$|{\tilde{\beta}}_{j}|=10^{-3}$ to avoid division by zero.

We evaluated the performance by two metrics: $\ell_{2}$ norm for estimation
error and F1 score for variable selection. The F1 score is a harmonic average
of precision and recall, where precision = (the number of correct selected
variables) / (the number of selected variables) and recall = (the number of
correct selected variables) / (the number of true active variables). We used
the F1 score because it allows us to evaluate the performance of variable
selection even when there is an imbalance between the number of active and
inactive variables. We also evaluated other metrics in Appendix C.3. They
included RMSE for prediction evaluation and sensitivity, specificity, positive
predictive value, and the number of active variables for feature selection
evaluation.

x

Figure 6: $\ell_{2}$ estimation errors (top) and variable selection F1-score
(bottom) for a large amount of source data.

The results are shown in Figure 6. In terms of estimation errors, the Transfer
Lasso and the Adaptive Transfer Lasso outperformed the other methods. The
Adaptive Lasso was superior to the Lasso, but it was inferior to the Transfer
Lasso and the Adaptive Transfer Lasso. In terms of variable selection, the
Adaptive Lasso and the Adaptive Transfer Lasso outperformed the others, and
the Adaptive Transfer Lasso was slightly superior to the Adaptive Lasso. These
results imply the superiority of the Adaptive Transfer Lasso with initial
estimators using large amounts of source data. The Adaptive Lasso, however,
does not fully utilize the initial estimators in this setting.

Figure 7: $\ell_{2}$ estimation errors (top) and variable selection F1-scores
(bottom) for a medium amount of source data.

Next, we supposed that we have a medium amount of source data and its sample
size is the same as the target data. We used the same data generation process,
comparison methods, and performance measurements as above.

The results are shown in Figure 7. All methods were comparable in terms of
estimation errors, but in terms of variable selection, the Adaptive Lasso and
the Adaptive Transfer Lasso were superior to those of the others. The Adaptive
Lasso and the Adaptive Transfer Lasso had similar performances for both
estimation error and variable selection. This is consistent with our
theoretical analyses.

##  6 Discussion

We discuss additional comparisons among methods from two perspectives;
regularization contours and prior distributions. We also discuss future work.

###  6.1 Regularization Contours

The regularization contours help to intuitively capture the strength and
pattern of regularization. Figure 8 shows their contours with an initial
estimator with a small value ${\tilde{\beta}}_{1}=0.5$ and a large value
${\tilde{\beta}}_{2}=2$.

The contours of the Adaptive Lasso are pointed at the coordinate axes (where
some elements are zero) and are especially sharp where the initial estimator
is small. The contours of the Transfer Lasso are pointed at the points where
some elements are zero or equal to the initial estimator, but they are not so
sharp. The contours for the Adaptive Transfer Lasso are pointed where some
elements are zero or equal to the initial estimator, and the sharpness varies
depending on the hyperparameters. These observations indicate that the
Adaptive Transfer Lasso flexibly changes the strength of regularization
depending on the initial estimator.

Figure 8: Regularization contours for the Adaptive Lasso (top left), the
Transfer Lasso (top right), and the Adaptive Transfer Lasso (bottom) with
initial estimators ${\tilde{\beta}}=[0.5,2]^{\top}$. Hyperparameters are
$\lambda_{n}=0,0.5,1$ (from left to right); $\eta_{n}=0,0.5,1$ (from top to
bottom); and $\gamma_{1}=\gamma_{2}=1$ for the Adaptive Transfer Lasso.

###  6.2 Prior Distribution

In Bayesian perspectives, the Lasso regularization (2) can be seen as a
negative log-likelihood of Laplace prior,

(42) $\displaystyle\lambda|\beta_{j}|=-\log
P(\beta_{j};\lambda)+const.,\leavevmode\nobreak\
P(z;\lambda):=\frac{\lambda}{2}\exp\left(-\lambda|z|\right).$

A similar view is possible for the Adaptive Lasso, the Transfer Lasso, and the
Adaptive Transfer Lasso. Most generally, the prior distribution of the
Adaptive Transfer Lasso is given by

(43) $\displaystyle\lambda v_{j}|\beta_{j}|+\eta
w_{j}|\beta_{j}-{\tilde{\beta}}_{j}|=-\log
P(\beta_{j};\lambda,\eta,v_{j},w_{j},{\tilde{\beta}}_{j})+const.,$ (44)
$\displaystyle
P(\beta_{j};\lambda,\eta,v_{j},w_{j},{\tilde{\beta}}_{j}):=\frac{1}{Z}\exp\left(-\lambda
v_{j}|\beta_{j}|-\eta w_{j}|\beta_{j}-{\tilde{\beta}}_{j}|\right),$ (45)
$\displaystyle Z:=\frac{2\lambda
v_{j}}{\lambda^{2}v_{j}^{2}-\eta^{2}w_{j}^{2}}\exp\left(-\eta
w_{j}|{\tilde{\beta}}_{j}|\right)-\frac{2\eta
w_{j}}{\lambda^{2}v_{j}^{2}-\eta^{2}w_{j}^{2}}\exp\left(-\lambda
v_{j}|{\tilde{\beta}}_{j}|\right).$

Figure 9: Prior distributions for the Adaptive Lasso (top left), the Transfer
Lasso (top right), and the Adaptive Transfer Lasso (bottom) with various
initial estimators. Hyperparameters are $\gamma_{1}=\gamma_{2}=1$ for the
Adaptive Transfer Lasso.

The prior distributions for the Adaptive Lasso, the Transfer Lasso, and the
Adaptive Transfer Lasso are shown in Figures 9. The prior distributions for
the Adaptive Lasso are all sharp at zero, and the distributions become steeper
as the initial estimator decreases. This means that the Adaptive Lasso
controls the variance of the prior distribution based on how close to zero the
initial estimator is. The prior distribution for the Transfer Lasso is sharp
at two points: zero and the initial estimator. When the initial estimator is
small, it is nearly the same as that for the Lasso, but when the initial
estimator is large, the sharpness changes depending on the magnitudes of
$\lambda$ and $\eta$. The prior distribution for the Adaptive Transfer Lasso
is somewhat different from that of the Transfer Lasso. The prior distribution
tends to peak at zero when the initial estimator is close to zero, whereas the
prior distribution tends to peak at that initial estimator when the initial
estimator is far from zero. This suggests that the Adaptive Transfer Lasso can
make full use of the information from the initial estimator and achieves
accurate active and varying variable selection.

###  6.3 Future Work

In our asymptotic analysis, we considered the case where $p$ is fixed and $n$
is infinitely divergent. The oracle property of Adaptive Lasso [21] can be
extended to the case for $p\gg n$ by high-dimensional asymptotic theory [10],
under different kinds of assumptions. As future research, it would be
interesting to see whether this can be extended to the Transfer Lasso and the
Adaptive Transfer Lasso.

In addition, we assumed that the initial estimator is consistent in our
asymptotic analysis. When the initial estimator is incorrectly specified,
performance deteriorates significantly for the Adaptive Lasso, but not so much
for the Transfer Lasso. It would be interesting to theoretically verify this
property.

##  7 Conclusion

The Adaptive Lasso and the Transfer Lasso are similar but have their
advantages and disadvantages from the viewpoint of an asymptotic perspective.
We proposed the Adaptive Transfer Lasso, which has advantages over the
Adaptive Lasso and the Transfer Lasso. We confirmed it in numerical
simulations.

## Acknowledgment

The research is the collaborative work of Toshiba Corporation and The
Institute of Statistical Mathematics, based on funding from Toshiba
Corporation.

## References

  * [1] Hamsa Bastani.  Predicting with proxies: Transfer learning in high dimension.  Management Science, 67(5):2964–2984, 2021.
  * [2] Mikhail Belkin, Daniel Hsu, and Ji Xu.  Two models of double descent for weak features.  SIAM Journal on Mathematics of Data Science, 2(4):1167–1180, 2020\.
  * [3] Evgenii Chzhen, Mohamed Hebiri, and Joseph Salmon.  On lasso refitting strategies.  Bernoulli, 25(4A):3175–3200, 2019.
  * [4] Jianqing Fan and Runze Li.  Variable selection via nonconcave penalized likelihood and its oracle properties.  Journal of the American statistical Association, 96(456):1348–1360, 2001.
  * [5] Wenjiang Fu and Keith Knight.  Asymptotics for lasso-type estimators.  The Annals of statistics, 28(5):1356–1378, 2000.
  * [6] Charles J Geyer.  On the asymptotics of constrained m-estimation.  The Annals of statistics, pages 1993–2010, 1994.
  * [7] Charles J Geyer.  On the asymptotics of convex stochastic optimization.  Unpublished manuscript, 37, 1996.
  * [8] Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani.  Surprises in high-dimensional ridgeless least squares interpolation.  The Annals of Statistics, 50(2):949–986, 2022.
  * [9] Trevor Hastie, Robert Tibshirani, and Ryan Tibshirani.  Best subset, forward stepwise or lasso? analysis and recommendations based on extensive comparisons.  Statistical Science, 35(4):579–592, 2020.
  * [10] Jian Huang, Shuangge Ma, and Cun-Hui Zhang.  Adaptive lasso for sparse high-dimensional regression models.  Statistica Sinica, pages 1603–1618, 2008.
  * [11] Sai Li, T Tony Cai, Hongzhe Li, et al.  Transfer learning for high-dimensional linear regression: Prediction, estimation and minimax optimality.  Journal of the Royal Statistical Society Series B, 84(1):149–173, 2022.
  * [12] Yue Li and Yuting Wei.  Minimum $\ell_{1}$-norm interpolators: Precise asymptotics and multiple descent.  arXiv preprint arXiv:2110.09502, 2021.
  * [13] Nicolai Meinshausen.  Relaxed lasso.  Computational Statistics & Data Analysis, 52(1):374–393, 2007\.
  * [14] Partha P Mitra.  Understanding overfitting peaks in generalization error: Analytical risk curves for $\ell_{2}$ and $\ell_{1}$ penalized interpolation.  arXiv preprint arXiv:1906.03667, 2019.
  * [15] David Pollard.  Asymptotics for least absolute deviation regression estimators.  Econometric Theory, 7(2):186–199, 1991.
  * [16] Masaaki Takada and Hironori Fujisawa.  Transfer learning via $\ell_{1}$ regularization.  Advances in Neural Information Processing Systems, 33:14266–14277, 2020.
  * [17] Ye Tian and Yang Feng.  Transfer learning under high-dimensional generalized linear models.  Journal of the American Statistical Association, pages 1–14, 2022\.
  * [18] Robert Tibshirani.  Regression shrinkage and selection via the lasso.  Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267–288, 1996.
  * [19] C-H ZHANG.  Nearly unbiased variable selection under minimax concave penalty.  The Annals of Statistics, 38:894–942, 2010.
  * [20] Peng Zhao and Bin Yu.  On model selection consistency of lasso.  The Journal of Machine Learning Research, 7:2541–2563, 2006.
  * [21] Hui Zou.  The adaptive lasso and its oracle properties.  Journal of the American statistical association, 101(476):1418–1429, 2006.

## Appendices

##  Appendix A Additional Asymptotic Properties

In this section, we describe additional theoretical results that are not
described in the main text. In Sections A.1 and A.2, we describe the
asymptotic properties of Adaptive Lasso and Transfer Lasso, respectively, when
the source parameters are deterministic (fixed). In Section A.3, we discuss
additional results for Transfer Lasso when the hyperparameters are at boundary
values.

###  A.1 Adaptive Lasso with deterministic Source Parameter

To avoid zero division, we define Adaptive Lasso with a deterministic source
parameter ${\tilde{\beta}}$ as

(46)
$\displaystyle\hat{\beta}_{n}^{\mathcal{A}}=\operatorname*{argmin}_{\beta:\beta_{j}=0\text{\leavevmode\nobreak\
for\leavevmode\nobreak\
}{\tilde{\beta}}_{j}=0}\left\\{Z_{n}^{\mathcal{A}}(\beta;{\tilde{\beta}},\lambda_{n},\gamma):=\frac{1}{n}\|y-X\beta\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j:{\tilde{\beta}}_{j}\neq
0}w_{j}|\beta_{j}|\right\\},$

and $w_{j}:=1/|{\tilde{\beta}}_{j}|^{\gamma}$ for ${\tilde{\beta}}_{j}\neq 0$.
By the definition (46) and Lemma 2.9, we can easily obtain Lemma A.1 and
Corollary A.2.

######  Lemma A.1 ($\sqrt{n}$-consistency for the Adaptive Lasso with
Deterministic Source Parameter).

Suppose that $\tilde{S}=S$. If $\lambda_{n}/\sqrt{n}\rightarrow\lambda_{0}\geq
0$, then

(47) $\displaystyle\sqrt{n}({\hat{\beta}}_{Sn}^{\mathcal{A}}-\beta^{*}_{S})$
$\displaystyle\overset{d}{\rightarrow}$
$\displaystyle\operatorname*{argmin}_{u}\left\\{u_{S}^{\top}C_{SS}u_{S}-2u_{S}^{\top}W_{S}+\lambda_{0}\sum_{j:{\tilde{\beta}}_{j}\neq
0}w_{j}\left(u_{j}\operatorname{sgn}(\beta^{*}_{j})I(\beta^{*}_{j}\neq
0)+|u_{j}|I(\beta^{*}_{j}=0)\right)\right\\},$

where $W\sim\mathcal{N}(0,\sigma^{2}C)$, and
${\hat{\beta}}_{S^{c}n}^{\mathcal{A}}=0$.

######  Corollary A.2 (Oracle Property for Adaptive Lasso with Deterministic
Source Parameter).

Suppose that $\tilde{S}=S$. If $\lambda_{n}=o(\sqrt{n})$, then the Adaptive
Lasso estimator satisfies the oracle property.

Based on Lemma A.1 and Corollary A.2, the Adaptive Lasso can satisfy the
oracle property even for deterministic source parameters. However, the source
parameter must satisfy exact support recovery, which is very restrictive.

###  A.2 Transfer Lasso with deterministic Source Parameter

Now we consider the asymptotic properties of Transfer Lasso with deterministic
source parameter. Let $T:=\\{j:{\tilde{\beta}}_{j}\neq\beta^{*}_{j}\\}$. We
can identify the range of its estimator (Theorem A.3) for a general condition.
The inequality in the probability is not necessarily tight.

######  Theorem A.3 (Estimation Range for Transfer Lasso).

If $(\lambda_{n}+\eta_{n})/\sqrt{n}\rightarrow\infty$ and
$(\lambda_{n}-\eta_{n})/\sqrt{n}\rightarrow 0$, then the transfer lasso
estimates satisfy

(48)
$\displaystyle\lim_{n\rightarrow\infty}P\left(\min\left\\{0,{\tilde{\beta}}_{j}\right\\}\leq{\hat{\beta}}_{j}\leq\max\left\\{0,{\tilde{\beta}}_{j}\right\\}\right)=1.$

###### Proof.

The proof is given in B.4.1. ∎

To obtain an asymptotic distribution and consistent variable selection, we
need to impose the condition that
$\operatorname{sgn}({\tilde{\beta}}_{j}-\beta^{*}_{j})=\operatorname{sgn}(\beta^{*}_{j})$
for $\forall j\in S\cap T$. This requires “over-estimation” for the initial
estimator. We obtain Theorem A.4 under this condition.

######  Theorem A.4 (Asymptotic distribution and active/varying variable
consistency with deterministic source parameters).

Suppose that
$\operatorname{sgn}({\tilde{\beta}}_{j}-\beta^{*}_{j})=\operatorname{sgn}(\beta^{*}_{j})$
for $\forall j\in S\cap T$. If
$(\lambda_{n}+\eta_{n})/\sqrt{n}\rightarrow\infty$ and
$(\lambda_{n}-\eta_{n})/\sqrt{n}\rightarrow 0$, then the Transfer Lasso
estimator satisfies

(49)
$\displaystyle\sqrt{n}({\hat{\beta}}-\beta^{*})\overset{d}{\rightarrow}\operatorname*{argmin}_{u\in\mathcal{U}}$
$\displaystyle\left\\{-2u^{\top}W+u^{\top}Cu\right\\},$ (53) $\displaystyle
W\sim\mathcal{N}(0,\sigma^{2}C),\leavevmode\nobreak\
\mathcal{U}:=\left\\{u\in\mathbb{R}^{p}\left|\begin{array}[]{l}u_{j}=0\leavevmode\nobreak\
\text{for}\leavevmode\nobreak\ \forall j\in S^{c}\cap T^{c}\\\
u_{j}{\tilde{\beta}}_{j}\geq 0\leavevmode\nobreak\
\text{for}\leavevmode\nobreak\ \forall j\in S^{c}\cap T\\\
u_{j}\beta^{*}_{j}\leq 0\leavevmode\nobreak\ \text{for}\leavevmode\nobreak\
\forall j\in S\cap T^{c}\end{array}\right.\right\\},$

and

(54)
$\displaystyle\begin{cases}\lim_{n\rightarrow\infty}P\left(0<{\hat{\beta}}_{j}<{\tilde{\beta}}_{j}\leavevmode\nobreak\
\text{or}\leavevmode\nobreak\
{\tilde{\beta}}_{j}<{\hat{\beta}}_{j}<0\right)=1\leavevmode\nobreak\
&\text{for}\leavevmode\nobreak\ \forall j\in(S\cap T)\\\
\lim_{n\rightarrow\infty}P\left(0<{\hat{\beta}}_{j}\leq{\tilde{\beta}}_{j}\leavevmode\nobreak\
\text{or}\leavevmode\nobreak\
{\tilde{\beta}}_{j}\leq{\hat{\beta}}_{j}<0\right)=1\leavevmode\nobreak\
&\text{for}\leavevmode\nobreak\ \forall j\in(S\cap T^{c})\\\
\lim_{n\rightarrow\infty}P\left(0\leq{\hat{\beta}}_{j}<{\tilde{\beta}}_{j}\leavevmode\nobreak\
\text{or}\leavevmode\nobreak\ {\tilde{\beta}}_{j}<{\hat{\beta}}_{j}\leq
0\right)=1\leavevmode\nobreak\ &\text{for}\leavevmode\nobreak\ \forall
j\in(S^{c}\cap T)\\\
\lim_{n\rightarrow\infty}P\left({\hat{\beta}}_{j}=0\right)=1\leavevmode\nobreak\
&\text{for}\leavevmode\nobreak\ \forall j\in(S^{c}\cap T^{c}).\end{cases}$

###### Proof.

The proof is given in B.4.2. ∎

Theorem A.4 shows that the asymptotic distribution in (49) and (53) is a
truncated Gaussian-mixture distribution. Each distribution in mixtures must
satisfy 1) Gaussian distribution for $j\in S\cap T$; 2) truncated Gaussian
distribution truncated at zero, or delta distribution at zero for $j\in
S^{c}\cap T$ and $j\in S\cap T^{c}$; and 3) delta distribution at zero for
$j\in S^{c}\cap T^{c}$.

In addition, Theorem A.4 indicates that the true active variables ($j\in S$)
and the true varying variables ($j\in T$) can be recovered asymptotically if
the source parameters of both active and varying variables have the same sign
as the true variables and have larger absolute values than the true variables.

Asymptotic normality (instead of a truncated normal mixture distribution) can
be obtained under a more restrictive condition (Theorem A.5).

######  Theorem A.5 (Oracle property (asymptotic normality and active/varying
variable selection consistency)).

Suppose that $S=T$, and
$\operatorname{sgn}({\tilde{\beta}}_{j}-\beta^{*}_{j})=\operatorname{sgn}(\beta^{*}_{j})$
for $\forall j\in S(=T)$. If
$(\lambda_{n}+\eta_{n})/\sqrt{n}\rightarrow\infty$ and
$(\lambda_{n}-\eta_{n})/\sqrt{n}\rightarrow 0$, then the Transfer Lasso
estimates satisfy the oracle property, that is,

(55)
$\displaystyle\lim_{n\rightarrow\infty}P(\hat{S}_{n}=S)=\lim_{n\rightarrow\infty}P(\hat{T}_{n}=T)=1,$
(56)
$\displaystyle\sqrt{n}({\hat{\beta}}_{S}-\beta^{*}_{S})\overset{d}{\rightarrow}\mathcal{N}(0,\sigma^{2}C_{SS}^{-1}).$

###### Proof.

The proof is given in B.4.3. ∎

Theorem A.5 is the oracle property for the Transfer Lasso. The Adaptive Lasso
requires $\sqrt{n}$-consistency for the initial estimator. In contrast, the
Transfer Lasso requires variable consistency, sign consistency, and “over-
estimation” for the source parameters. This yields $\sqrt{n}$-consistency as
well as variable selection consistency for both active and varying variables.

###  A.3 Transfer Lasso with Initial Estimator in Boundary Region

We have an additional result on the asymptotic property for Transfer Lasso
with an initial estimator when the hyperparameters are at boundary values.

######  Theorem A.6.

Suppose that ${\tilde{\beta}}$ is a $\sqrt{m}$-consistent estimator and define
$z:=\sqrt{m}({\tilde{\beta}}_{m}-\beta^{*})$. Suppose that $n/m\to 0$. If
$\lambda_{n}/\sqrt{n}\to\infty$, $\eta_{n}/\lambda_{n}\to 1$, and
$(\lambda_{n}-\eta_{n})/\sqrt{n}\to\delta_{0}$, then

(57)
$\displaystyle\sqrt{n}\left({\hat{\beta}}_{n}^{\mathcal{T}}-\beta^{*}\right)\overset{d}{\to}\operatorname*{argmin}_{u\in\mathcal{U}}\left\\{u_{S}^{\top}C_{SS}u_{S}-2u_{S}^{\top}W_{S}-\delta_{0}\sum_{j\in
S}|u_{j}|\right\\},$ (59)
$\displaystyle\mathcal{U}:=\left\\{u\in\mathbb{R}^{p}\left|\begin{array}[]{l}\beta^{*}_{j}u_{j}\leq
0\leavevmode\nobreak\ \text{for}\leavevmode\nobreak\ \forall j\in
S\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
u_{j}=0\leavevmode\nobreak\ \text{for}\leavevmode\nobreak\ \forall j\in
S^{c}\end{array}\right.\right\\}.$

In addition, ${\hat{\beta}}_{n}^{\mathcal{T}}$ results in inconsistent active
variable selection.

###### Proof.

The proof is given in B.5.1. ∎

Theorem A.6 implies $\sqrt{n}$-consistency but inconsistent variable selection
for the Transfer Lasso.

##  Appendix B Proofs

###  B.1 Proofs of Lasso and Adaptive Lasso

In these proofs, the superscripts of each method may be omitted when it is
obvious. For example, ${\hat{\beta}}_{n}^{\mathcal{L}}$,
${\hat{\beta}}_{n}^{\mathcal{A}}$, ${\hat{\beta}}_{n}^{\mathcal{T}}$, and
${\hat{\beta}}_{n}^{\\#}$ may be simply written as ${\hat{\beta}}_{n}$.

####  B.1.1 Proof of Lemma 2.2

We have

(60) $\displaystyle Z_{n}^{\mathcal{L}}(\beta;\lambda_{n}):$
$\displaystyle=\frac{1}{n}\|y-X\beta\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}|\beta_{j}|$
(61)
$\displaystyle=\frac{1}{n}\|X\beta^{*}+\varepsilon-X\beta\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}|\beta_{j}|$
(62)
$\displaystyle=(\beta-\beta^{*})^{\top}\left(\frac{1}{n}X^{\top}X\right)(\beta-\beta^{*})-\frac{2}{n}(\beta-\beta^{*})^{\top}X^{\top}\varepsilon+\frac{1}{n}\|\varepsilon\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}|\beta_{j}|.$

Let

(63) $\displaystyle
V(\beta,\lambda_{0}):=(\beta-\beta^{*})^{\top}C(\beta-\beta^{*})+\lambda_{0}\sum_{j}|\beta_{j}|.$

Then, we have

(64) $\displaystyle
Z_{n}^{\mathcal{L}}(\beta;\lambda_{n})-V(\beta;\lambda_{0})-\sigma^{2}$ (65)
$\displaystyle=$
$\displaystyle(\beta-\beta^{*})^{\top}\left(\frac{1}{n}X^{\top}X-C\right)(\beta-\beta^{*})-\frac{2}{n}(\beta-\beta^{*})^{\top}X^{\top}\varepsilon$
(66)
$\displaystyle+\left(\frac{1}{n}\|\varepsilon\|_{2}^{2}-\sigma^{2}\right)+\left(\frac{\lambda_{n}}{n}-\lambda_{0}\right)\sum_{j}|\beta_{j}|$
(67) $\displaystyle\rightarrow$ $\displaystyle 0\leavevmode\nobreak\
(n\rightarrow\infty;\leavevmode\nobreak\ \text{pointwise convergence)}$

Since $Z_{n}^{\mathcal{L}}$ is convex, based on [15], we have

(68) $\displaystyle\sup_{\beta\in
K}|Z_{n}^{\mathcal{L}}(\beta;\lambda_{n})-V(\beta;\lambda_{0})-\sigma^{2}|\overset{p}{\rightarrow}0\leavevmode\nobreak\
\text{for any compact set}\leavevmode\nobreak\ K.$

Let
$\beta_{n}^{(0)}:=\operatorname*{argmin}_{\beta}Z_{n}^{\mathcal{L}}(\beta;0)$.
Because $\beta_{n}^{(0)}=O_{P}(1)$ and
$\|{\hat{\beta}}^{\mathcal{L}}_{n}\|_{1}\leq\|{\hat{\beta}}_{n}^{(0)}\|_{1}$,
we have ${\hat{\beta}}_{n}^{\mathcal{L}}=O_{P}(1)$ and thus

(69)
$\displaystyle{\hat{\beta}}_{n}^{\mathcal{L}}=\operatorname*{argmin}_{\beta}Z_{n}^{\mathcal{L}}(\beta;\lambda_{n})\overset{p}{\rightarrow}\operatorname*{argmin}_{\beta}V(\beta;\lambda_{0}).$

####  B.1.2 Proof of Lemma 2.4

Let $u:=\sqrt{n}(\beta-\beta^{*})$. We have

(70) $\displaystyle Z_{n}^{\mathcal{L}}(\beta;\lambda_{n}):=$
$\displaystyle\frac{1}{n}\|y-X\beta\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}|\beta_{j}|$
(71) $\displaystyle=$
$\displaystyle\frac{1}{n}\left\|\varepsilon-\frac{1}{\sqrt{n}}Xu\right\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}\left|\beta^{*}_{j}+\frac{u_{j}}{\sqrt{n}}\right|,$

and

(72) $\displaystyle
Z_{n}^{\mathcal{L}}(\beta;\lambda_{n})-Z_{n}^{\mathcal{L}}(\beta^{*};\lambda_{n})$
(73) $\displaystyle=$
$\displaystyle\frac{1}{n}\left\|\varepsilon-\frac{1}{\sqrt{n}}Xu\right\|_{2}^{2}-\frac{1}{n}\|\varepsilon\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}\left(\left|\beta^{*}_{j}+\frac{u_{j}}{\sqrt{n}}\right|-|\beta^{*}_{j}|\right)$
(74) $\displaystyle=$
$\displaystyle\frac{1}{n}\underbrace{\left\\{u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-2u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)+\frac{\lambda_{n}}{\sqrt{n}}\sum_{j}\left(\left|u_{j}+\sqrt{n}\beta^{*}_{j}\right|-|\sqrt{n}\beta^{*}_{j}|\right)\right\\}}_{=:V_{n}(u)}.$

Assuming $\lambda_{n}/\sqrt{n}\rightarrow\lambda_{0}$, we obtain

(75) $\displaystyle V_{n}(u)\overset{d}{\rightarrow}$ $\displaystyle
u^{\top}Cu-2u^{\top}W+\lambda_{0}\sum_{j}\left(u_{j}\operatorname{sgn}(\beta^{*}_{j})I(\beta^{*}_{j}\neq
0)+|u_{j}|I(\beta^{*}_{j}=0)\right)=:V(u).$

Since $V_{n}(u)$ is convex and $V(u)$ has a unique minimum, based on [7], we
have

(76)
$\displaystyle\operatorname*{argmin}_{u}V_{n}(u)\overset{d}{\rightarrow}\operatorname*{argmin}_{u}V(u).$

On the other hand, we have

(77) $\displaystyle{\hat{\beta}}_{n}^{\mathcal{L}}=$
$\displaystyle\operatorname*{argmin}_{\beta}Z_{n}^{\mathcal{L}}(\beta;\lambda_{n})=\operatorname*{argmin}_{\beta}(Z_{n}^{\mathcal{L}}(\beta;\lambda_{n})-Z_{n}^{\mathcal{L}}(\beta^{*};\lambda_{n}))$
(78) $\displaystyle=$
$\displaystyle\operatorname*{argmin}_{\beta}V_{n}(u)=\beta^{*}+\frac{1}{\sqrt{n}}\operatorname*{argmin}_{u}V_{n}(u).$

Therefore, we have
$\operatorname*{argmin}_{u}V_{n}(u)=\sqrt{n}({\hat{\beta}}_{n}^{\mathcal{L}}-\beta^{*})$
and

(79)
$\displaystyle\sqrt{n}({\hat{\beta}}_{n}^{\mathcal{L}}-\beta^{*})\overset{d}{\rightarrow}\operatorname*{argmin}_{u}V(u).$

####  B.1.3 Proof of Lemma 2.6

Let $u^{*}:=\operatorname*{argmin}_{u}V(u)$ where $V(u)$ is defined in (74).
Note that $\hat{S}_{n}=S$ implies ${\hat{\beta}}_{j}=0\leavevmode\nobreak\
\forall j\in S^{c}$ and
$\sqrt{n}{\hat{\beta}}_{S^{c}}\overset{d}{\rightarrow}u^{*}_{S^{c}}$. By the
weak convergence result, we have

(80) $\displaystyle P(\hat{S}_{n}=S)\leq
P(\sqrt{n}{\hat{\beta}}_{j}=0\leavevmode\nobreak\ \forall j\in S^{c}),$

and

(81)
$\displaystyle\limsup_{n}P(\hat{S}_{n}=S)\leq\limsup_{n}P(\sqrt{n}{\hat{\beta}}_{j}=0\leavevmode\nobreak\
\forall j\in S^{c})\leq P(u_{j}^{*}=0\leavevmode\nobreak\ \forall j\in
S^{c})=:c.$

If $\lambda_{0}=0$, then we have

(82) $\displaystyle u^{*}$
$\displaystyle=C^{-1}W\sim\mathcal{N}(0,\sigma^{2}C^{-1}),$

resulting in $c=0$. If $\lambda_{0}>0$, then the KKT condition yields

(83)
$\displaystyle\begin{cases}-2W_{S}+2(Cu^{*})_{S}+\lambda_{0}\operatorname{sgn}(\beta^{*}_{S})=0\\\
\left|-2W_{S^{c}}+2(Cu^{*})_{S^{c}}\right|\leq\lambda_{0}.\end{cases}$

When $u^{*}_{S^{c}}=0$, the above conditions indicate

(84)
$\displaystyle\left|-2W_{S^{c}}+C_{S^{c}S}C_{SS}^{-1}(2W_{S}-\lambda_{0}\operatorname{sgn}(\beta^{*}_{S})\right|\leq\lambda_{0},$

thus we have

(85) $\displaystyle c\leq
P\left(\left|-2W_{S^{c}}+C_{S^{c}S}C_{SS}^{-1}(2W_{S}-\lambda_{0}\operatorname{sgn}(\beta^{*}_{S})\right|\leq\lambda_{0}\right)<1.$

####  B.1.4 Proof of Lemma 2.7

Let $u:=(n/\lambda_{n})(\beta-\beta^{*})$. We have

(86) $\displaystyle Z_{n}^{\mathcal{L}}(\beta):=$
$\displaystyle\frac{1}{n}\|y-X\beta\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}|\beta_{j}|$
(87) $\displaystyle=$
$\displaystyle\frac{1}{n}\left\|\varepsilon-\frac{\lambda_{n}}{n}Xu\right\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}\left|\beta^{*}_{j}+\frac{\lambda_{n}}{n}u_{j}\right|,$

and

(88) $\displaystyle Z_{n}^{\mathcal{L}}(\beta)-Z_{n}^{\mathcal{L}}(\beta^{*})$
(89) $\displaystyle=$
$\displaystyle\frac{1}{n}\left\|\varepsilon-\frac{\lambda_{n}}{n}Xu\right\|_{2}^{2}-\frac{1}{n}\left\|\varepsilon\right\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}\left(\left|\beta^{*}_{j}+\frac{\lambda_{n}}{n}u_{j}\right|-\left|\beta^{*}_{j}\right|\right)$
(90) $\displaystyle=$
$\displaystyle\frac{\lambda_{n}^{2}}{n^{2}}\underbrace{\left\\{u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-2\frac{\sqrt{n}}{\lambda_{n}}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)+\sum_{j}\left(\left|u_{j}+\frac{n}{\lambda_{n}}\beta^{*}_{j}\right|-\left|\frac{n}{\lambda_{n}}\beta^{*}_{j}\right|\right)\right\\}}_{=:V_{n}(u)}.$

Assuming $\lambda_{n}/n\rightarrow 0$ and
$\lambda_{n}/\sqrt{n}\rightarrow\infty$, we obtain

(91) $\displaystyle V_{n}(u)\overset{d}{\rightarrow}$ $\displaystyle
u^{\top}Cu+\sum_{j=1}^{p}\left(u_{j}\operatorname{sgn}(\beta^{*}_{j})I(\beta^{*}\neq
0)+|u_{j}|I(\beta^{*}_{j}=0)\right)=:V(u).$

Since $V_{n}(u)$ is convex and $V(u)$ has a unique minimum, based on [7], we
have

(92)
$\displaystyle\operatorname*{argmin}_{u}V_{n}(u)\overset{d}{\rightarrow}\operatorname*{argmin}_{u}V(u).$

On the other hand, we have

(93) $\displaystyle{\hat{\beta}}_{n}^{\mathcal{L}}=$
$\displaystyle\operatorname*{argmin}_{\beta}Z_{n}^{\mathcal{L}}(\beta;\lambda_{n})=\operatorname*{argmin}_{\beta}(Z_{n}^{\mathcal{L}}(\beta;\lambda_{n})-Z_{n}^{\mathcal{L}}(\beta^{*};\lambda_{n}))$
(94) $\displaystyle=$
$\displaystyle\operatorname*{argmin}_{\beta}V_{n}(u)=\beta^{*}+\frac{\lambda_{n}}{n}\operatorname*{argmin}_{u}V_{n}(u).$

Therefore, we have
$\operatorname*{argmin}_{u}V_{n}(u)=(n/\lambda_{n})({\hat{\beta}}_{n}^{\mathcal{L}}-\beta^{*})$
and

(95)
$\displaystyle\frac{n}{\lambda_{n}}({\hat{\beta}}_{n}^{\mathcal{L}}-\beta^{*})\overset{d}{\rightarrow}\operatorname*{argmin}_{u}V(u).$

####  B.1.5 Proof of Lemma 2.9

Asymptotic Normality Part: Let $u:=\sqrt{n}(\beta-\beta^{*})$. We have

(96) $\displaystyle Z_{n}^{\mathcal{A}}(\beta):=$
$\displaystyle\frac{1}{n}\|y-X\beta\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}w_{j}|\beta_{j}|$
(97) $\displaystyle=$
$\displaystyle\frac{1}{n}\left\|\varepsilon-\frac{1}{\sqrt{n}}Xu\right\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}w_{j}\left|\beta^{*}_{j}+\frac{u_{j}}{\sqrt{n}}\right|$

and

(98) $\displaystyle Z_{n}^{\mathcal{A}}(\beta)-Z_{n}^{\mathcal{A}}(\beta^{*})$
(99) $\displaystyle=$
$\displaystyle\frac{1}{n}\left\|\varepsilon-\frac{1}{\sqrt{n}}Xu\right\|_{2}^{2}-\frac{1}{n}\|\varepsilon\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}w_{j}\left(\left|\beta^{*}_{j}+\frac{u_{j}}{\sqrt{n}}\right|-|\beta^{*}_{j}|\right)$
(100) $\displaystyle=$
$\displaystyle\frac{1}{n}\underbrace{\left\\{u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-2u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)+\frac{\lambda_{n}}{\sqrt{n}}\sum_{j}w_{j}\left(\left|u_{j}+\sqrt{n}\beta^{*}_{j}\right|-\left|\sqrt{n}\beta^{*}_{j}\right|\right)\right\\}}_{=:V_{n}(u)}.$

We know that

(101)
$\displaystyle-2u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)\overset{d}{\rightarrow}-2u^{\top}W,\leavevmode\nobreak\
W\sim\mathcal{N}(0,\sigma^{2}C)$ (102) $\displaystyle
u^{\top}\left(\frac{1}{n}X^{\top}X\right)u\rightarrow u^{\top}Cu.$

Now we consider the last term of (100). If $\beta^{*}_{j}\neq 0$, then
$w_{j}\rightarrow 1/|\beta^{*}_{j}|^{\gamma}$ and
$|u_{j}+\sqrt{n}\beta^{*}_{j}|-|\sqrt{n}\beta^{*}_{j}|\rightarrow
u_{j}\operatorname{sgn}(\beta^{*}_{j})$, thus by Slutsky’s theorem,

(103)
$\displaystyle\frac{\lambda_{n}}{\sqrt{n}}w_{j}\left(\left|u_{j}+\sqrt{n}\beta^{*}_{j}\right|-\left|\sqrt{n}\beta^{*}_{j}\right|\right)\overset{p}{\rightarrow}0,$

as $\lambda_{n}/\sqrt{n}\rightarrow 0$. If $\beta^{*}_{j}=0$, then
$\lambda_{n}w_{j}/\sqrt{n}=\lambda_{n}n^{(\gamma-1)/2}|\sqrt{n}{\tilde{\beta}}_{j}|^{-\gamma}$
and $|u_{j}+\sqrt{n}\beta^{*}_{j}|-|\sqrt{n}\beta^{*}_{j}|=|u_{j}|$, thus by
Slutsky’s theorem,

(104)
$\displaystyle\frac{\lambda_{n}}{\sqrt{n}}w_{j}\left(\left|u_{j}+\sqrt{n}\beta^{*}_{j}\right|-\left|\sqrt{n}\beta^{*}_{j}\right|\right)\overset{d}{\rightarrow}\begin{cases}0&\text{if}\leavevmode\nobreak\
u_{j}=0\\\ \infty&\text{otherwise},\end{cases}$

as $\lambda_{n}n^{(\gamma-1)/2}\rightarrow\infty$. Therefore, we have for
every $u$,

(105) $\displaystyle
V_{n}(u)\overset{d}{\rightarrow}V(u):=\begin{cases}-2u_{S}^{\top}W_{S}+u_{S}^{\top}C_{SS}u_{S}&\text{if}\leavevmode\nobreak\
u_{j}=0\leavevmode\nobreak\ \forall j\in S^{c}\\\
\infty&\text{otherwise}.\end{cases}$

Since $V_{n}(u)$ is convex and $V(u)$ has a unique minimum of
$(C_{SS}^{-1}W_{S},0)^{\top}$, based on [6], we have

(106)
$\displaystyle\operatorname*{argmin}_{u}V_{n}(u)\overset{d}{\rightarrow}\operatorname*{argmin}_{u}V(u)=(C_{SS}^{-1}W_{S},0)^{\top}.$

On the other hand,

(107) $\displaystyle{\hat{\beta}}_{n}^{\mathcal{L}}=$
$\displaystyle\operatorname*{argmin}_{\beta}Z_{n}(\beta;\lambda_{n})=\operatorname*{argmin}_{\beta}(Z_{n}(\beta;\lambda_{n})-Z_{n}(\beta^{*};\lambda_{n}))$
(108) $\displaystyle=$
$\displaystyle\operatorname*{argmin}_{\beta}V_{n}(u)=\beta^{*}+\frac{1}{\sqrt{n}}\operatorname*{argmin}_{u}V_{n}(u).$

Therefore, we have
$\operatorname*{argmin}_{u}V_{n}(u)=\sqrt{n}({\hat{\beta}}_{n}-\beta^{*})$ and

(109)
$\displaystyle\sqrt{n}({\hat{\beta}}_{S}-\beta^{*}_{S})\overset{d}{\rightarrow}\mathcal{N}(0,\sigma^{2}C_{SS}^{-1})^{\top},\leavevmode\nobreak\
\sqrt{n}{\hat{\beta}}_{S^{c}}\overset{d}{\rightarrow}0.$

Variable Selection Consistency Part: Asymptotic normality indicates that
${\hat{\beta}}\overset{p}{\rightarrow}\beta^{*}$, thus

(110) $\displaystyle\forall j\in S,\leavevmode\nobreak\
P(j\in\operatorname{supp}({\hat{\beta}}))\rightarrow 1.$

Now, we consider the event $j\in S^{c}$ and
$j\in\operatorname{supp}({\hat{\beta}})$. By the KKT conditions, we have

(111) $\displaystyle
2\mathbf{x}_{j}^{\top}(y-X{\hat{\beta}})+\lambda_{n}w_{j}\operatorname{sgn}({\hat{\beta}}_{j})=0.$

This yields

(112) $\displaystyle
2\left(\frac{1}{n}\mathbf{x}_{j}^{\top}X\right)\sqrt{n}(\beta^{*}-{\hat{\beta}})+2\frac{1}{\sqrt{n}}\mathbf{x}_{j}^{\top}\varepsilon+\lambda_{n}n^{(\gamma-1)/2}|\sqrt{n}{\tilde{\beta}}_{j}|^{-\gamma}\operatorname{sgn}({\hat{\beta}}_{j})=0$

By Slutsky’s theorem, the first and second terms on the left-hand side
converge to some normal distribution, but the the last term on the left-hand
side diverges to infinity. Therefore, we have for $\forall j\in S^{c}$,

(113) $\displaystyle P\left(j\in\operatorname{supp}({\hat{\beta}})\right)\leq
P\left(2\mathbf{x}_{j}^{\top}(y-X{\hat{\beta}})+\lambda_{n}w_{j}\operatorname{sgn}({\hat{\beta}}_{j})=0\right)\rightarrow
0.$

###  B.2 Proofs of Transfer Lasso

####  B.2.1 Proof of Lemma 3.2

The proof is similar to that of Lemma 2.9. If ${\tilde{\beta}}$ is a
$\sqrt{m}$-consistent initial estimator and
$\lambda_{n}\sqrt{m^{\gamma}/n}\rightarrow\infty$, (104) reduces to

(114)
$\displaystyle\lambda_{n}w_{j}\left(\left|\beta^{*}_{j}+\frac{u_{j}}{\sqrt{n}}\right|-|\beta^{*}_{j}|\right)=\lambda_{n}\sqrt{\frac{m^{\gamma}}{n}}\left|\sqrt{m}{\tilde{\beta}}_{j}\right|^{-\gamma}|u_{j}|\overset{d}{\rightarrow}\begin{cases}0&\text{if}\leavevmode\nobreak\
u_{j}=0\\\ \infty&\text{otherwise}\end{cases}$

and (112) reduces to

(115) $\displaystyle
2\left(\frac{1}{n}\mathbf{x}_{j}^{\top}X\right)\sqrt{n}(\beta^{*}-{\hat{\beta}})+2\frac{1}{\sqrt{n}}\mathbf{x}_{j}^{\top}\varepsilon+\lambda_{n}\sqrt{\frac{m^{\gamma}}{n}}|\sqrt{m}{\tilde{\beta}}_{j}|^{-\gamma}\operatorname{sgn}({\hat{\beta}}_{j})=0.$

These modifications does not affect the remaining proofs.

####  B.2.2 Proof of Theorem 3.5 and Theorem A.6

Let $u:=l(\beta-\beta^{*})$ where $l=l(n,m,\lambda_{n})$ is a certain function
as defined later. Let $z:=\sqrt{m}({\tilde{\beta}}-\beta^{*})$. Since
${\tilde{\beta}}$ is a $\sqrt{m}$-consistent estimator, $z$ follows some
distribution. Suppose that $n/m\to r_{0}\geq 0$. The objective function for
Transfer Lasso is

(116) $\displaystyle Z_{n}^{\mathcal{T}}(\beta):=$
$\displaystyle\frac{1}{n}\|y-X\beta\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}|\beta_{j}|+\frac{\eta_{n}}{n}\sum_{j}|\beta_{j}-{\tilde{\beta}}_{j}|$
(117) $\displaystyle=$
$\displaystyle\frac{1}{n}\left\|\varepsilon-\frac{1}{l}Xu\right\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}\left|\beta^{*}_{j}+\frac{u_{j}}{l}\right|+\frac{\eta_{n}}{n}\sum_{j}\left|\frac{u_{j}}{l}-\frac{z_{j}}{\sqrt{m}}\right|$
(118) $\displaystyle=$
$\displaystyle\frac{1}{l^{2}}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2}{\sqrt{n}l}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)+\frac{1}{n}\|\varepsilon\|_{2}^{2}$
(119) $\displaystyle+$
$\displaystyle\frac{\lambda_{n}}{nl}\sum_{j}\left|u_{j}+l\beta^{*}_{j}\right|+\frac{\eta_{n}}{nl}\sum_{j}\left|u_{j}-\frac{l}{\sqrt{m}}z_{j}\right|,$

and we have

(120) $\displaystyle
Z_{n}^{\mathcal{T}}(\beta)-Z_{n}^{\mathcal{T}}(\beta^{*})=$
$\displaystyle\frac{1}{l^{2}}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2}{\sqrt{n}l}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
$\displaystyle+$
$\displaystyle\frac{\lambda_{n}}{nl}\sum_{j}\left(\left|u_{j}+l\beta^{*}_{j}\right|-\left|l\beta^{*}_{j}\right|\right)+\frac{\eta_{n}}{nl}\sum_{j}\left(\left|u_{j}-\frac{l}{\sqrt{m}}z_{j}\right|-\left|\frac{l}{\sqrt{m}}z_{j}\right|\right).$

We divide the case into three cases: $l=\sqrt{m}$ (Case I), $\sqrt{n}$ (Case
II), and $n/\lambda_{n}$ (Case III).

Case I. Let $l=\sqrt{m}$. Then, (120) reduces to

(121) $\displaystyle
Z_{n}^{\mathcal{T}}(\beta)-Z_{n}^{\mathcal{T}}(\beta^{*})=$
$\displaystyle\frac{1}{m}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2}{\sqrt{nm}}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
(122) $\displaystyle+$
$\displaystyle\frac{\lambda_{n}}{n\sqrt{m}}\sum_{j}\left(\left|u_{j}+\sqrt{m}\beta^{*}_{j}\right|-\left|\sqrt{m}\beta^{*}_{j}\right|\right)+\frac{\eta_{n}}{n\sqrt{m}}\sum_{j}\left(\left|u_{j}-z_{j}\right|-\left|z_{j}\right|\right).$

Let
$V_{n}(u):=(n\sqrt{m}/\eta_{n})(Z_{n}^{\mathcal{T}}(\beta)-Z_{n}^{\mathcal{T}}(\beta^{*}))+\sum_{j}|z_{j}|$,
then we have

(123) $\displaystyle V_{n}(u)$
$\displaystyle=\frac{n}{\sqrt{m}\eta_{n}}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2\sqrt{n}}{\eta_{n}}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
(124)
$\displaystyle+\frac{\lambda_{n}}{\eta_{n}}\sum_{j}\left(\left|u_{j}+\sqrt{m}\beta^{*}_{j}\right|-\left|\sqrt{m}\beta^{*}_{j}\right|\right)+\sum_{j}\left|u_{j}-z_{j}\right|.$

On the other hand, we have

(125) $\displaystyle{\hat{\beta}}_{n}^{\mathcal{T}}=$
$\displaystyle\operatorname*{argmin}_{\beta}Z_{n}^{\mathcal{T}}(\beta)=\operatorname*{argmin}_{\beta}\left(Z_{n}^{\mathcal{T}}(\beta)-Z_{n}^{\mathcal{T}}(\beta^{*})\right)=\beta^{*}+\frac{1}{\sqrt{m}}\operatorname*{argmin}_{u}V_{n}(u),$

and hence

(126)
$\displaystyle\sqrt{m}({\hat{\beta}}_{n}-\beta^{*})=\operatorname*{argmin}_{u}V_{n}(u).$

Consider the case (i) where $\eta_{n}/\sqrt{n}\to\infty$,
$\lambda_{n}/\eta_{n}\to\rho_{0}$, and $0\leq\rho_{0}<1$. Let
$V(u):=\lim_{n\to\infty}V_{n}(u)$, then we have

(127) $\displaystyle
V(u)=\sum_{j}\left(\left(\rho_{0}u_{j}\operatorname{sgn}(\beta^{*}_{j})+\left|u_{j}-z_{j}\right|\right)I(\beta^{*}_{j}\neq
0)+\left(\rho_{0}|u_{j}|+|u_{j}-z_{j}|\right)I(\beta^{*}_{j}=0)\right).$

Because $V_{n}(u)$ is convex and $V(u)$ has a unique minimum, we obtain

(128)
$\displaystyle\sqrt{m}({\hat{\beta}}_{n}-\beta^{*})\to\operatorname*{argmin}_{u}V(u)=z.$

Case II. Let $l=\sqrt{n}$. Then, (120) reduces to

(129) $\displaystyle
Z_{n}^{\mathcal{T}}(\beta)-Z_{n}^{\mathcal{T}}(\beta^{*})$ (130)
$\displaystyle=$
$\displaystyle\frac{1}{n}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2}{n}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
(131) $\displaystyle+$
$\displaystyle\frac{\lambda_{n}}{n\sqrt{n}}\sum_{j}\left(\left|u_{j}+\sqrt{n}\beta^{*}_{j}\right|-\left|\sqrt{n}\beta^{*}_{j}\right|\right)+\frac{\eta_{n}}{n\sqrt{n}}\sum_{j}\left(\left|u_{j}-\sqrt{\frac{n}{m}}z_{j}\right|-\left|\sqrt{\frac{n}{m}}z_{j}\right|\right).$

Let $V_{n}(u):=n(Z_{n}^{\mathcal{T}}(\beta)-Z_{n}^{\mathcal{T}}(\beta^{*}))$,
then we have

(132) $\displaystyle V_{n}(u)$
$\displaystyle=u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-2u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
(133)
$\displaystyle+\frac{\lambda_{n}}{\sqrt{n}}\sum_{j}\left(\left|u_{j}+\sqrt{n}\beta^{*}_{j}\right|-\left|\sqrt{n}\beta^{*}_{j}\right|\right)+\frac{\eta_{n}}{\sqrt{n}}\sum_{j}\left(\left|u_{j}-\sqrt{\frac{n}{m}}z_{j}\right|-\left|\sqrt{\frac{n}{m}}z_{j}\right|\right).$

Consider the case (ii) where $\lambda_{n}/\sqrt{n}\to\lambda_{0}\geq 0$ and
$\eta_{n}/\sqrt{n}\to\eta_{0}\geq 0$. Let $V(u):=\lim_{n\to\infty}V_{n}(u)$,
then we have

(134) $\displaystyle V(u)=$ $\displaystyle u^{\top}Cu-2u^{\top}W$ (135)
$\displaystyle+$
$\displaystyle\lambda_{0}\sum_{j}\left(u_{j}\operatorname{sgn}(\beta^{*}_{j})I(\beta^{*}_{j}\neq
0)+\left|u_{j}\right|I(\beta^{*}_{j}=0)\right)+\eta_{0}\sum_{j}\left(\left|u_{j}-\sqrt{r_{0}}z_{j}\right|-\left|\sqrt{r_{0}}z_{j}\right|\right).$

On the other hand,we have

(136) $\displaystyle{\hat{\beta}}_{n}^{\mathcal{T}}=$
$\displaystyle\operatorname*{argmin}_{\beta}Z_{n}^{\mathcal{T}}(\beta)=\operatorname*{argmin}_{\beta}\left(Z_{n}^{\mathcal{T}}(\beta)-Z_{n}^{\mathcal{T}}(\beta^{*})\right)=\beta^{*}+\frac{1}{\sqrt{n}}\operatorname*{argmin}_{u}V_{n}(u),$

and hence

(137)
$\displaystyle\sqrt{n}({\hat{\beta}}_{n}-\beta^{*})=\operatorname*{argmin}_{u}V_{n}(u).$

Because $V_{n}(u)$ is convex and $V(u)$ has a unique minimum, we obtain

(138)
$\displaystyle\sqrt{n}\left({\hat{\beta}}_{n}-\beta^{*}\right)\to\operatorname*{argmin}_{u}\Biggl{\\{}u^{\top}Cu-2u^{\top}W$
(139)
$\displaystyle+\lambda_{0}\sum_{j}\left(u_{j}\operatorname{sgn}(\beta^{*}_{j})I(\beta^{*}_{j}\neq
0)+\left|u_{j}\right|I(\beta^{*}_{j}=0)\right)+\eta_{0}\sum_{j}\left(\left|u_{j}-\sqrt{r_{0}}z_{j}\right|-\left|\sqrt{r_{0}}z_{j}\right|\right)\Biggr{\\}}.$

Case III. Let $l=n/\lambda_{n}(\to\infty)$. Then, (120) reduces to

(140) $\displaystyle
Z_{n}^{\mathcal{T}}(\beta)-Z_{n}^{\mathcal{T}}(\beta^{*})$ (141)
$\displaystyle=$
$\displaystyle\frac{\lambda_{n}^{2}}{n^{2}}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{\lambda_{n}}{n\sqrt{n}}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
(142) $\displaystyle+$
$\displaystyle\frac{\lambda_{n}^{2}}{n^{2}}\sum_{j}\left(\left|u_{j}+\frac{n}{\lambda_{n}}\beta^{*}_{j}\right|-\left|\frac{n}{\lambda_{n}}\beta^{*}_{j}\right|\right)+\frac{\lambda_{n}\eta_{n}}{n^{2}}\sum_{j}\left(\left|u_{j}-\sqrt{\frac{n}{m}}\frac{\sqrt{n}}{\lambda_{n}}z_{j}\right|-\left|\sqrt{\frac{n}{m}}\frac{\sqrt{n}}{\lambda_{n}}z_{j}\right|\right)$

Let
$V_{n}(u):=(n^{2}/\lambda_{n}^{2})(Z_{n}^{\mathcal{T}}(\beta)-Z_{n}^{\mathcal{T}}(\beta^{*}))$,
then we have

(143) $\displaystyle V_{n}(u)$
$\displaystyle=u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2\sqrt{n}}{\lambda_{n}}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
(144)
$\displaystyle+\sum_{j}\left(\left|u_{j}+\frac{n}{\lambda_{n}}\beta^{*}_{j}\right|-\left|\frac{n}{\lambda_{n}}\beta^{*}_{j}\right|\right)+\frac{\eta_{n}}{\lambda_{n}}\sum_{j}\left(\left|u_{j}-\sqrt{\frac{n}{m}}\frac{\sqrt{n}}{\lambda_{n}}z_{j}\right|-\left|\sqrt{\frac{n}{m}}\frac{\sqrt{n}}{\lambda_{n}}z_{j}\right|\right).$

Consider the case (iii) where $\lambda_{n}/\sqrt{n}\to\infty$,
$\lambda_{n}/n\to 0$, and $\eta_{n}/\lambda_{n}\to\rho_{0}^{\prime}\geq 0$.
Let $V(u):=\lim_{n\to\infty}V_{n}(u)$, then we have

(145) $\displaystyle
V(u)=u^{\top}Cu+\sum_{j}\left(\left(u_{j}\operatorname{sgn}(\beta^{*}_{j})+\rho_{0}^{\prime}\left|u_{j}\right|\right)I(\beta^{*}_{j}\neq
0)+(1+\rho_{0}^{\prime})\left|u_{j}\right|I(\beta^{*}_{j}=0)\right).$

On the other hand,we have

(146) $\displaystyle{\hat{\beta}}_{n}^{\mathcal{T}}=$
$\displaystyle\operatorname*{argmin}_{\beta}Z_{n}^{\mathcal{T}}(\beta)=\operatorname*{argmin}_{\beta}\left(Z_{n}^{\mathcal{T}}(\beta)-Z_{n}^{\mathcal{T}}(\beta^{*})\right)=\beta^{*}+\frac{\lambda_{n}}{n}\operatorname*{argmin}_{u}V_{n}(u),$

and hence

(147)
$\displaystyle\frac{n}{\lambda_{n}}({\hat{\beta}}_{n}-\beta^{*})=\operatorname*{argmin}_{u}V_{n}(u).$

Because $V_{n}(u)$ is convex and $V(u)$ has a unique minimum, we obtain

(148) $\displaystyle\frac{n}{\lambda_{n}}({\hat{\beta}}_{n}-\beta^{*})$
$\displaystyle\to$
$\displaystyle\operatorname*{argmin}_{u}\left\\{u^{\top}Cu+\sum_{j}\left(\left(u_{j}\operatorname{sgn}(\beta^{*}_{j})+\rho_{0}^{\prime}\left|u_{j}\right|\right)I(\beta^{*}_{j}\neq
0)+(1+\rho_{0}^{\prime})\left|u_{j}\right|I(\beta^{*}_{j}=0)\right)\right\\}.$

In addition, if $\rho_{0}^{\prime}\geq 1$, then the right-hand side of (148)
reduces to $0$.

####  B.2.3 Proof of Theorem 3.9

By Theorem 3.5 and Corollary 3.6, the Transfer Lasso estimator satisfies
${\hat{\beta}}_{n}^{\mathcal{T}}\overset{p}{\to}\beta^{*}$, thus

(150) $\displaystyle\forall j\in S,\leavevmode\nobreak\
\limsup_{n\to\infty}P(j\in\operatorname{supp}({\hat{\beta}}_{n}^{\mathcal{T}}))=1.$

Let $u^{*}:=\operatorname*{argmin}_{u}V(u)$ where $V(u)$ is the asymptotic
objective function (127) in the case (i) and (134) in the case (ii). By the
weak convergence result, we have

(151) $\displaystyle\limsup_{n\to\infty}P(\hat{S}_{n}^{\mathcal{T}}=S)=$
$\displaystyle\limsup_{n\to\infty}P({\hat{\beta}}_{j}^{\mathcal{T}}\neq
0\leavevmode\nobreak\ \forall j\in S\leavevmode\nobreak\
\text{and}\leavevmode\nobreak\
{\hat{\beta}}_{j}^{\mathcal{T}}=0\leavevmode\nobreak\ \forall j\in S^{c})$
(152) $\displaystyle\leq$
$\displaystyle\limsup_{n\to\infty}P(l{\hat{\beta}}_{j}^{\mathcal{T}}=0\leavevmode\nobreak\
\forall j\in S^{c})$ (153) $\displaystyle\leq$ $\displaystyle
P(u_{j}^{*}=0\leavevmode\nobreak\ \forall j\in S^{c}),$

where $l$ is $\sqrt{m}$ in the case (i) and $\sqrt{n}$ in the case (ii). We
evaluate the probability of $u_{S^{c}}^{*}=0$ in each case.

Consider the case (i) where $\eta_{n}/\sqrt{n}\to\infty$ and
$\lambda_{n}/\eta_{n}\to\rho_{0}$ with $0\leq\rho_{0}<1$. The asymptotic
distribution of the Transfer Lasso is equal to that of the initial estimator
$z$. Because we suppose that $z$ is inconsistent in terms of variable
selection, we obtain

(154) $\displaystyle P(u_{j}^{*}=0\leavevmode\nobreak\ \forall j\in
S^{c})=P(z_{j}=0\leavevmode\nobreak\ \forall j\in S^{c})\leq c<1,$

hence ${\hat{\beta}}_{n}^{\mathcal{T}}$ is inconsistent in terms of variable
selection.

Consider the case (ii) where $\lambda_{n}/\sqrt{n}\to\lambda_{0}\geq 0$ and
$\eta_{n}/\sqrt{n}\to\eta_{0}\geq 0$. Suppose that $u_{S^{c}}^{*}=0$. Let
$S_{1}:=\\{j:j\in S\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
u_{j}^{*}\neq\sqrt{r_{0}}z_{j}\\}$, $S_{2}:=\\{j:j\in S\leavevmode\nobreak\
\text{and}\leavevmode\nobreak\ u_{j}^{*}=\sqrt{r_{0}}z_{j}\\}$,
$S_{1}^{c}:=\\{j:j\in S^{c}\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
u_{j}^{*}\neq\sqrt{r_{0}}z_{j}\\}$, and $S_{2}^{c}:=\\{j:j\in
S^{c}\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
u_{j}^{*}=\sqrt{r_{0}}z_{j}\\}$. By the KKT conditions of
$\operatorname*{argmin}_{u}V(u)$, we have

(155)
$\displaystyle\begin{cases}2C_{S_{1}S_{1}}u_{S_{1}}^{*}+2\sqrt{r_{0}}C_{S_{1}S_{2}}z_{S_{2}}-2W_{S_{1}}+\lambda_{0}\operatorname{sgn}(\beta^{*}_{S_{1}})+\eta_{0}\operatorname{sgn}(u_{S_{1}}^{*}-\sqrt{r_{0}}z_{S_{1}})=0\\\
\left|2C_{S_{2}S_{1}}u_{S_{1}}^{*}+2\sqrt{r_{0}}C_{S_{2}S_{2}}z_{S_{2}}-2W_{S_{2}}+\lambda_{0}\operatorname{sgn}(\beta^{*}_{S_{2}})\right|\leq\eta_{0}\\\
\left|2C_{S^{c}S_{1}}u_{S_{1}}^{*}+2\sqrt{r_{0}}C_{S_{1}^{c}S_{2}}z_{S_{2}}-2W_{S_{1}^{c}}+\eta_{0}\operatorname{sgn}(-\sqrt{r_{0}}z_{S_{1}^{c}})\right|\leq\lambda_{0}\\\
\left|2C_{S^{c}S_{1}}u_{S_{1}}^{*}+2\sqrt{r_{0}}C_{S_{2}^{c}S_{2}}z_{S_{2}}-2W_{S_{2}^{c}}\right|\leq\lambda_{0}+\eta_{0}.\end{cases}$

If $S_{1}\neq\emptyset$, the first equation yields

(156) $\displaystyle
u_{S_{1}}^{*}=C_{S_{1}S_{1}}^{-1}\left(W_{S_{1}}-\sqrt{r_{0}}C_{S_{1}S_{2}}z_{S_{2}}-\frac{\lambda_{0}}{2}\operatorname{sgn}(\beta^{*}_{S_{1}})-\frac{\eta_{0}}{2}\operatorname{sgn}(u_{S_{1}}^{*}-\sqrt{r_{0}}z_{S_{1}})\right).$

Because $W$ follows a Gaussian and $z$ follows some distribution, the
probability holding the second, third, and fourth inequalities in (155) is
less than $1$. This indicates inconsistent variable selection. If
$S_{1}=\emptyset$, by the KKT conditions, we have

(157)
$\displaystyle\begin{cases}\left|2\sqrt{r_{0}}C_{S_{2}S_{2}}z_{S_{2}}-2W_{S_{2}}+\lambda_{0}\operatorname{sgn}(\beta^{*}_{S_{2}})\right|\leq\eta_{0}\\\
\left|2\sqrt{r_{0}}C_{S_{1}^{c}S_{2}}z_{S_{2}}-2W_{S_{1}^{c}}+\eta_{0}\operatorname{sgn}(-\sqrt{r_{0}}z_{S_{1}^{c}})\right|\leq\lambda_{0}\\\
\left|2\sqrt{r_{0}}C_{S_{2}^{c}S_{2}}z_{S_{2}}-2W_{S_{2}^{c}}\right|\leq\lambda_{0}+\eta_{0}.\end{cases}$

The probability holding these inequalities is less than $1$. This indicates
inconsistent variable selection.

####  B.2.4 Proof of Theorem 3.10

Consider the case (i) in Theorem 3.5. Consider the event where
${\hat{\beta}}_{j}\neq{\tilde{\beta}}_{j}$ for some $j\in S$. By the KKT
conditions,

(158)
$\displaystyle\begin{dcases}2\left(\frac{1}{n}\mathbf{x}_{j}^{\top}X\right)\sqrt{\frac{n}{m}}\sqrt{m}({\hat{\beta}}-\beta^{*})-\frac{2}{\sqrt{n}}\mathbf{x}_{j}^{\top}\varepsilon+\frac{\lambda_{n}}{\sqrt{n}}\operatorname{sgn}({\hat{\beta}}_{j})+\frac{\eta_{n}}{\sqrt{n}}\operatorname{sgn}({\hat{\beta}}_{j}-{\tilde{\beta}}_{j})=0,\\\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \text{for}\leavevmode\nobreak\ {\hat{\beta}}_{j}\neq
0,\\\
\left|2\left(\frac{1}{n}\mathbf{x}_{j}^{\top}X\right)\sqrt{\frac{n}{m}}\sqrt{m}({\hat{\beta}}-\beta^{*})-\frac{2}{\sqrt{n}}\mathbf{x}_{j}^{\top}\varepsilon+\frac{\eta_{n}}{\sqrt{n}}\operatorname{sgn}({\hat{\beta}}_{j}-{\tilde{\beta}}_{j})\right|\leq\frac{\lambda_{n}}{\sqrt{n}},\leavevmode\nobreak\
\text{for}\leavevmode\nobreak\ {\hat{\beta}}_{j}=0.\\\ \end{dcases}$

The term including $\eta_{n}/\sqrt{n}$ in (158) in both ${\hat{\beta}}_{j}\neq
0$ and ${\hat{\beta}}_{j}=0$ cases diverge to infinity faster compared to the
rest terms. Therefore, we have

(159) $\displaystyle\forall j\in S,\leavevmode\nobreak\
\lim_{n\to\infty}P({\hat{\beta}}_{j}\neq{\tilde{\beta}}_{j})=0.$

This concludes

(160)
$\displaystyle\lim_{n\to\infty}P({\hat{\beta}}_{S}={\tilde{\beta}}_{S})=1.$

####  B.2.5 Proof of Theorem 3.11

By Theorem 3.5 and Corollary 3.6, the Transfer Lasso estimator satisfies
${\hat{\beta}}_{n}^{\mathcal{T}}\overset{p}{\to}\beta^{*}$, thus

(161) $\displaystyle\forall j\in S,\leavevmode\nobreak\
P(j\in\operatorname{supp}({\hat{\beta}}_{n}^{\mathcal{T}}))\to 1.$

Consider the case (ii) where $\lambda_{n}/\sqrt{n}\to\lambda_{0}\geq 0$ and
$\eta_{n}/\sqrt{n}\to\eta_{0}\geq 0$. Let
$u^{*}:=\operatorname*{argmin}_{u}V(u)$ where $V(u)$ is the asymptotic
objective function (134) for the case (ii). By the weak convergence result, we
have

(162)
$\displaystyle\limsup_{n\to\infty}P({\hat{\beta}}_{S}^{\mathcal{T}}={\tilde{\beta}}_{S})\leq
P\left(u_{j}^{*}=\sqrt{r_{0}}z_{j}\leavevmode\nobreak\ \forall j\in S\right).$

Suppose that $u_{S}^{*}=\sqrt{r_{0}}z_{S}$. Let $S_{1}:=\\{j:j\in
S\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\ u_{j}^{*}\neq 0\\}$,
$S_{2}:=\\{j:j\in S\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
u_{j}^{*}=0\\}$, $S_{1}^{c}:=\\{j:j\in S^{c}\leavevmode\nobreak\
\text{and}\leavevmode\nobreak\ u_{j}^{*}\neq 0\leavevmode\nobreak\
\text{and}\leavevmode\nobreak\ u_{j}^{*}\neq\sqrt{r_{0}}z_{j}\\}$,
$S_{2}^{c}:=\\{j:j\in S^{c}\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
u_{j}^{*}\neq 0\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
u_{j}^{*}=\sqrt{r_{0}}z_{j}\\}$, $S_{3}^{c}:=\\{j:j\in
S^{c}\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
u_{j}^{*}=0\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
u_{j}^{*}\neq\sqrt{r_{0}}z_{j}\\}$, and $S_{4}^{c}:=\\{j:j\in
S^{c}\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
u_{j}^{*}=0\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
u_{j}^{*}=\sqrt{r_{0}}z_{j}\\}$. By the KKT conditions of
$\operatorname*{argmin}_{u}V(u)$, we have

(163)
$\displaystyle\begin{cases}\left|2\sqrt{r_{0}}C_{SS}z_{S}+2C_{SS^{c}}u_{S^{c}}^{*}-2W_{S}+\lambda_{0}\operatorname{sgn}(\beta^{*}_{S})\right|\leq\eta_{0}\\\
2\sqrt{r_{0}}C_{S_{1}^{c}S}z_{S}+2C_{S_{1}^{c}S^{c}}u_{S^{c}}^{*}-2W_{S_{1}^{c}}+\lambda_{0}\operatorname{sgn}(u_{S_{1}^{c}})+\eta_{0}\operatorname{sgn}(u_{S_{1}^{c}}^{*}-\sqrt{r_{0}}z_{S_{1}^{c}})=0\\\
\left|2\sqrt{r_{0}}C_{S_{2}^{c}S}z_{S}+2C_{S_{2}^{c}S^{c}}u_{S^{c}}^{*}-2W_{S_{2}^{c}}+\lambda_{0}\operatorname{sgn}(u_{S_{2}^{c}})\right|\leq\eta_{0}\\\
\left|2\sqrt{r_{0}}C_{S_{3}^{c}S}z_{S}+2C_{S_{3}^{c}S^{c}}u_{S^{c}}^{*}-2W_{S_{3}^{c}}+\eta_{0}\operatorname{sgn}(u_{S_{3}^{c}}^{*}-\sqrt{r_{0}}z_{S_{3}^{c}})\right|\leq\lambda_{0}\\\
\left|2\sqrt{r_{0}}C_{S_{4}^{c}S}z_{S}+2C_{S_{4}^{c}S^{c}}u_{S^{c}}-2W_{S_{4}^{c}}\right|\leq\lambda_{0}+\eta_{0}.\end{cases}$

Note that $u_{S_{2}^{c}}=\sqrt{r_{0}}z_{S_{2}^{c}}$, $u_{S_{2}^{c}}=0$, and
$u_{S_{4}^{c}}=\sqrt{r_{0}}z_{S_{4}^{c}}=0$. If $S_{1}\neq\emptyset$, the
second equation yields

(164) $\displaystyle u_{S_{1}^{c}}^{*}=C_{S_{1}S_{1}}^{-1}\biggl{(}$
$\displaystyle
W_{S_{1}^{c}}-\sqrt{r_{0}}C_{S_{1}^{c}S}z_{S}-\sqrt{r_{0}}C_{S_{1}^{c}S_{2}^{c}}z_{S_{2}^{c}}$
(165)
$\displaystyle-\frac{\lambda_{0}}{2}\operatorname{sgn}(u_{S_{1}^{c}}^{*})-\frac{\eta_{0}}{2}\operatorname{sgn}(u_{S_{1}^{c}}^{*}-\sqrt{r_{0}}z_{S_{1}^{c}})\biggr{)}.$

Hence, we have

(166)
$\displaystyle\begin{cases}\left|2\sqrt{r_{0}}C_{SS}z_{S}+2C_{SS_{1}^{c}}u_{S_{1}^{c}}^{*}+2\sqrt{r_{0}}C_{SS_{2}^{c}}z_{S_{2}^{c}}-2W_{S}+\lambda_{0}\operatorname{sgn}(\beta^{*}_{S})\right|\leq\eta_{0}\\\
\left|2\sqrt{r_{0}}C_{S_{2}^{c}S}z_{S}+2C_{S_{2}^{c}S_{1}^{c}}u_{S_{1}^{c}}^{*}+2\sqrt{r_{0}}C_{S_{2}^{c}S_{2}^{c}}z_{S_{2}^{c}}-2W_{S_{2}^{c}}+\lambda_{0}\operatorname{sgn}(u_{S_{2}^{c}})\right|\leq\eta_{0}\\\
\left|2\sqrt{r_{0}}C_{S_{3}^{c}S}z_{S}+2C_{S_{3}^{c}S_{1}^{c}}u_{S_{1}^{c}}^{*}+2\sqrt{r_{0}}C_{S_{3}^{c}S_{2}^{c}}z_{S_{2}^{c}}-2W_{S_{3}^{c}}+\eta_{0}\operatorname{sgn}(u_{S_{3}^{c}}-\sqrt{r_{0}}z_{S_{3}^{c}})\right|\leq\lambda_{0}\\\
\left|2\sqrt{r_{0}}C_{S_{4}^{c}S}z_{S}+2C_{S_{4}^{c}S_{1}^{c}}u_{S_{1}^{c}}^{*}+2\sqrt{r_{0}}C_{S_{4}^{c}S_{2}^{c}}z_{S_{2}^{c}}-2W_{S_{4}^{c}}\right|\leq\lambda_{0}+\eta_{0}.\end{cases}$

Because $W$ follows a Gaussian distribution and $u_{S_{1}^{c}}^{*}$ and $z$
follow some distribution, the probability holding these inequalities is less
than $1$. This indicates inconsistent invariant variable selection. If
$S_{1}=\emptyset$, by the KKT conditions, we have

(167)
$\displaystyle\begin{cases}\left|2\sqrt{r_{0}}C_{SS}z_{S}+2\sqrt{r_{0}}C_{SS_{2}^{c}}z_{S_{2}^{c}}-2W_{S}+\lambda_{0}\operatorname{sgn}(\beta^{*}_{S})\right|\leq\eta_{0}\\\
\left|2\sqrt{r_{0}}C_{S_{2}^{c}S}z_{S}+2\sqrt{r_{0}}C_{S_{2}^{c}S_{2}^{c}}z_{S_{2}^{c}}-2W_{S_{2}^{c}}+\lambda_{0}\operatorname{sgn}(u_{S_{2}^{c}})\right|\leq\eta_{0}\\\
\left|2\sqrt{r_{0}}C_{S_{3}^{c}S}z_{S}+2\sqrt{r_{0}}C_{S_{3}^{c}S_{2}^{c}}z_{S_{2}^{c}}-2W_{S_{3}^{c}}+\eta_{0}\operatorname{sgn}(u_{S_{3}^{c}}-\sqrt{r_{0}}z_{S_{3}^{c}})\right|\leq\lambda_{0}\\\
\left|2\sqrt{r_{0}}C_{S_{4}^{c}S}z_{S}+2\sqrt{r_{0}}C_{S_{4}^{c}S_{2}^{c}}z_{S_{2}^{c}}-2W_{S_{4}^{c}}\right|\leq\lambda_{0}+\eta_{0}.\end{cases}$

Because $W$ follows a Gaussian distribution and $z$ follows some distribution,
the probability holding these inequalities is less than $1$. This indicates
inconsistent variable selection.

###  B.3 Proofs of Adaptive Transfer Lasso

####  B.3.1 Proof of Theorem 4.1

Let $u:=l(\beta-\beta^{*})$ where $l=l(n,m,\lambda_{n})$ is a certain function
as defined later. Let $z:=\sqrt{m}({\tilde{\beta}}-\beta^{*})$. Since
${\tilde{\beta}}$ is a $\sqrt{m}$-consistent estimator, $z$ follows some
distribution. We suppose that $n/m\to r_{0}\geq 0$. The objective function for
the Adaptive Transfer Lasso is

(168) $\displaystyle Z_{n}^{\\#}(\beta):=$
$\displaystyle\frac{1}{n}\|y-X\beta\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}\frac{|\beta_{j}|}{|{\tilde{\beta}}_{j}|^{\gamma_{1}}}+\frac{\eta_{n}}{n}\sum_{j}|{\tilde{\beta}}_{j}|^{\gamma_{2}}|\beta_{j}-{\tilde{\beta}}_{j}|$
(169) $\displaystyle=$
$\displaystyle\frac{1}{n}\left\|\varepsilon-\frac{1}{l}Xu\right\|_{2}^{2}+\frac{\lambda_{n}}{n}\sum_{j}\frac{\left|\beta^{*}_{j}+\frac{u_{j}}{l}\right|}{\left|\frac{z_{j}}{\sqrt{m}}+\beta^{*}_{j}\right|^{\gamma_{1}}}+\frac{\eta_{n}}{n}\sum_{j}\left|\frac{z_{j}}{\sqrt{m}}+\beta^{*}_{j}\right|^{\gamma_{2}}\left|\frac{u_{j}}{l}-\frac{z_{j}}{\sqrt{m}}\right|$
(170) $\displaystyle=$
$\displaystyle\frac{1}{l^{2}}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2}{\sqrt{n}l}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)+\frac{1}{n}\|\varepsilon\|_{2}^{2}$
(171) $\displaystyle+$
$\displaystyle\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{nl}\sum_{j}\frac{1}{\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{1}}}\left|u_{j}+l\beta^{*}_{j}\right|$
(172) $\displaystyle+$
$\displaystyle\frac{\eta_{n}}{n\sqrt{m^{\gamma_{2}}}l}\sum_{j}\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{2}}\left|u_{j}-\frac{l}{\sqrt{m}}z_{j}\right|,$

and we have

(173) $\displaystyle Z_{n}^{\\#}(\beta)-Z_{n}^{\\#}(\beta^{*})=$
$\displaystyle\frac{1}{l^{2}}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2}{\sqrt{n}l}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
$\displaystyle+$
$\displaystyle\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{nl}\sum_{j}\frac{1}{\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{1}}}\left(\left|u_{j}+l\beta^{*}_{j}\right|-\left|l\beta^{*}_{j}\right|\right)$
$\displaystyle+$
$\displaystyle\frac{\eta_{n}}{n\sqrt{m^{\gamma_{2}}}l}\sum_{j}\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{2}}\left(\left|u_{j}-\frac{l}{\sqrt{m}}z_{j}\right|-\left|\frac{l}{\sqrt{m}}z_{j}\right|\right).$

We divide the case into three cases: $l=\sqrt{m}$ (Case I), $l=\sqrt{n}$ (Case
II), and $l=n/\lambda_{n}$ (Case III).

Case I. Let $l=\sqrt{m}$. Then, (173) reduces to

(174) $\displaystyle Z_{n}^{\\#}(\beta)-Z_{n}^{\\#}(\beta^{*})$ (175)
$\displaystyle=$
$\displaystyle\frac{1}{m}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2}{\sqrt{nm}}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
$\displaystyle+$
$\displaystyle\frac{\sqrt{m^{\gamma_{1}-1}}\lambda_{n}}{n}\sum_{j}\frac{1}{\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{1}}}\left(\left|u_{j}+\sqrt{m}\beta^{*}_{j}\right|-\left|\sqrt{m}\beta^{*}_{j}\right|\right)$
$\displaystyle+$
$\displaystyle\frac{\eta_{n}}{n\sqrt{m^{\gamma_{2}+1}}}\sum_{j}\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{2}}\left(\left|u_{j}-z_{j}\right|-\left|z_{j}\right|\right)$
(176) $\displaystyle=$
$\displaystyle\frac{1}{m}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2}{\sqrt{nm}}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
$\displaystyle+$
$\displaystyle\frac{\lambda_{n}}{n\sqrt{m}}\sum_{j}\left(\frac{\left|u_{j}+\sqrt{m}\beta^{*}_{j}\right|-\left|\sqrt{m}\beta^{*}_{j}\right|}{\left|\beta^{*}_{j}+\frac{z_{j}}{\sqrt{m}}\right|^{\gamma_{1}}}I(\beta^{*}_{j}\neq
0)+\frac{\sqrt{m^{\gamma_{1}}}|u_{j}|}{\left|z_{j}\right|^{\gamma_{1}}}I(\beta^{*}_{j}=0)\right)$
$\displaystyle+$
$\displaystyle\frac{\eta_{n}}{n\sqrt{m}}\sum_{j}\left(\left|\beta^{*}_{j}+\frac{z_{j}}{\sqrt{m}}\right|^{\gamma_{2}}I(\beta^{*}_{j}\neq
0)+\frac{\left|z_{j}\right|^{\gamma_{2}}}{\sqrt{m^{\gamma_{2}}}}I(\beta^{*}_{j}=0)\right)\left(\left|u_{j}-z_{j}\right|-\left|z_{j}\right|\right).$

Consider the case (i) where

(177)
$\displaystyle\frac{\eta_{n}}{n\sqrt{m^{\gamma_{2}+1}}}\gg\frac{1}{\sqrt{nm}}\leavevmode\nobreak\
\text{and}\leavevmode\nobreak\
\frac{\eta_{n}}{n\sqrt{m^{\gamma_{2}+1}}}\gg\frac{\sqrt{m^{\gamma_{1}-1}}\lambda_{n}}{n},$

that is,

(178)
$\displaystyle\frac{\eta_{n}}{\sqrt{nm^{\gamma_{2}}}}\to\infty\leavevmode\nobreak\
\text{and}\leavevmode\nobreak\
\frac{\sqrt{m^{\gamma_{1}+\gamma_{2}}}\lambda_{n}}{\eta_{n}}\to 0.$

Let
$V_{n}(u):=(n\sqrt{m^{\gamma_{2}+1}}/\eta_{n})(Z_{n}^{\\#}(\beta)-Z_{n}^{\\#}(\beta^{*}))+(\text{term
irrelevant to}\leavevmode\nobreak\ \beta)$, then we have

(179) $\displaystyle V_{n}(u)=$
$\displaystyle\frac{\sqrt{nm^{\gamma_{2}}}}{\eta_{n}}\sqrt{\frac{n}{m}}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2\sqrt{nm^{\gamma_{2}}}}{\eta_{n}}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
$\displaystyle+$
$\displaystyle\frac{\sqrt{m^{\gamma_{1}+\gamma_{2}}}\lambda_{n}}{\eta_{n}}\sum_{j}\left(\frac{\left|u_{j}+\sqrt{m}\beta^{*}_{j}\right|-\left|\sqrt{m}\beta^{*}_{j}\right|}{\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{1}}}I(\beta^{*}_{j}\neq
0)+\frac{|u_{j}|}{\left|z_{j}\right|^{\gamma_{1}}}I(\beta^{*}_{j}=0)\right)$
$\displaystyle+$
$\displaystyle\sum_{j}\left(\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{2}}I(\beta^{*}_{j}\neq
0)+\left|z_{j}\right|^{\gamma_{2}}I(\beta^{*}_{j}=0)\right)\left|u_{j}-z_{j}\right|.$

Let $V(u):=\lim_{n\to\infty}V_{n}(u)$, then we have

(180) $\displaystyle V(u)=$
$\displaystyle\begin{dcases}\sum_{j}\left|z_{j}\right|^{\gamma_{2}}\left|u_{j}-z_{j}\right|I(\beta^{*}_{j}=0)&\text{for}\leavevmode\nobreak\
u_{S}=z_{S},\\\ \infty&\text{otherwise}.\end{dcases}$

Therefore, we obtain

(181)
$\displaystyle\sqrt{m}({\hat{\beta}}_{n}^{\\#}-\beta^{*})=\operatorname*{argmin}_{u}V_{n}(u)\overset{d}{\to}\operatorname*{argmin}_{u}V(u)=z.$

Consider the case (ii) where

(182)
$\displaystyle\frac{\sqrt{m^{\gamma_{1}-1}}\lambda_{n}}{n}\gg\frac{1}{\sqrt{nm}},\frac{\sqrt{m^{\gamma_{1}-1}}\lambda_{n}}{n}\gg\frac{\eta_{n}}{n\sqrt{m^{\gamma_{2}+1}}},$
(183)
$\displaystyle\frac{\eta_{n}}{n\sqrt{m}}\gg\frac{1}{\sqrt{nm}},\frac{\eta_{n}}{n\sqrt{m}}\gg\frac{\lambda_{n}}{n\sqrt{m}},$

that is,

(184)
$\displaystyle\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{\sqrt{n}}\to\infty,\frac{\eta_{n}}{\sqrt{n}}\to\infty,\frac{\eta_{n}}{\lambda_{n}}\to\infty,\frac{\eta_{n}}{\sqrt{m^{\gamma_{1}+\gamma_{2}}}\lambda_{n}}\to
0.$

We divide into three cases:

(185) $\displaystyle\text{(ii-a)}\leavevmode\nobreak\
\frac{\eta_{n}}{n\sqrt{m}}\gg\frac{\sqrt{m^{\gamma_{1}-1}}\lambda_{n}}{n},\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \text{(ii-b)}\leavevmode\nobreak\
\frac{\eta_{n}}{n\sqrt{m}}\ll\frac{\sqrt{m^{\gamma_{1}-1}}\lambda_{n}}{n},$
(186) $\displaystyle\text{(ii-c)}\leavevmode\nobreak\
\frac{\eta_{n}}{n\sqrt{m}}\asymp\frac{\sqrt{m^{\gamma_{1}-1}}\lambda_{n}}{n},$

that is,

(187) $\displaystyle\text{(ii-a)}\leavevmode\nobreak\
\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{\eta_{n}}\to 0,\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \text{(ii-b)}\leavevmode\nobreak\
\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{\eta_{n}}\to\infty,\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \text{(ii-c)}\leavevmode\nobreak\
\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{\eta_{n}}\to\rho_{0}>0.$

In the case (ii-a), let
$V_{n}(u):=(n/\sqrt{m^{\gamma_{1}-1}}\lambda_{n})(Z_{n}^{\\#}(\beta)-Z_{n}^{\\#}(\beta^{*}))+(\text{term
irrelevant to}\leavevmode\nobreak\ \beta)$, then we have

(188) $\displaystyle V_{n}(u)=$
$\displaystyle\frac{\sqrt{n}}{\sqrt{m^{\gamma_{1}}}\lambda_{n}}\sqrt{\frac{n}{m}}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2\sqrt{n}}{\sqrt{m^{\gamma_{1}}}\lambda_{n}}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
$\displaystyle+$
$\displaystyle\sum_{j}\left(\frac{\left|u_{j}+\sqrt{m}\beta^{*}_{j}\right|-\left|\sqrt{m}\beta^{*}_{j}\right|}{\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{1}}}I(\beta^{*}_{j}\neq
0)+\frac{|u_{j}|}{\left|z_{j}\right|^{\gamma_{1}}}I(\beta^{*}_{j}=0)\right)$
$\displaystyle+$
$\displaystyle\frac{\eta_{n}}{\sqrt{m^{\gamma_{1}}}\lambda_{n}}\sum_{j}\left(\left|\beta^{*}_{j}+\frac{z_{j}}{\sqrt{m}}\right|^{\gamma_{2}}I(\beta^{*}_{j}\neq
0)+\frac{\left|z_{j}\right|^{\gamma_{2}}}{\sqrt{m^{\gamma_{2}}}}I(\beta^{*}_{j}=0)\right)\left|u_{j}-z_{j}\right|.$

Let $V(u):=\lim_{n\to\infty}V_{n}(u)$, then we have

(189) $\displaystyle V(u)=$
$\displaystyle\begin{dcases}\sum_{j}\frac{|u_{j}|}{|z_{j}|^{\gamma_{1}}}I(\beta^{*}_{j}=0)&\text{for}\leavevmode\nobreak\
u_{S}=z_{S},\\\ \infty&\text{otherwise}.\end{dcases}$

Therefore, we obtain

(190)
$\displaystyle\sqrt{m}({\hat{\beta}}_{n}^{\\#}-\beta^{*})=\operatorname*{argmin}_{u}V_{n}(u)\overset{d}{\to}\operatorname*{argmin}_{u}V(u)=\begin{dcases}0&\text{for}\leavevmode\nobreak\
j\in S^{c},\\\ z_{j}&\text{for}\leavevmode\nobreak\ j\in S.\end{dcases}$

In the case (ii-b), let
$V_{n}(u):=(n\sqrt{m}/\eta_{n})(Z_{n}^{\\#}(\beta)-Z_{n}^{\\#}(\beta^{*}))+(\text{term
irrelevant to}\leavevmode\nobreak\ \beta)$, then we have

(191) $\displaystyle V_{n}(u)=$
$\displaystyle\frac{\sqrt{n}}{\eta_{n}}\sqrt{\frac{n}{m}}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2\sqrt{n}}{\eta_{n}}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
$\displaystyle+$
$\displaystyle\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{\eta_{n}}\sum_{j}\left(\frac{\left|u_{j}+\sqrt{m}\beta^{*}_{j}\right|-\left|\sqrt{m}\beta^{*}_{j}\right|}{\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{1}}}I(\beta^{*}_{j}\neq
0)+\frac{|u_{j}|}{\left|z_{j}\right|^{\gamma_{1}}}I(\beta^{*}_{j}=0)\right)$
$\displaystyle+$
$\displaystyle\sum_{j}\left(\left|\beta^{*}_{j}+\frac{z_{j}}{\sqrt{m}}\right|^{\gamma_{2}}I(\beta^{*}_{j}\neq
0)+\frac{\left|z_{j}\right|^{\gamma_{2}}}{\sqrt{m^{\gamma_{2}}}}I(\beta^{*}_{j}=0)\right)\left|u_{j}-z_{j}\right|.$

Let $V(u):=\lim_{n\to\infty}V_{n}(u)$, then we have

(192) $\displaystyle V(u)=$
$\displaystyle\begin{dcases}\sum_{j}|\beta^{*}_{j}|^{\gamma_{2}}\left|u_{j}-z_{j}\right|I(\beta^{*}_{j}\neq
0)&\text{for}\leavevmode\nobreak\ u_{S^{c}}=0,\\\
\infty&\text{otherwise}.\end{dcases}$

Therefore, we obtain

(193)
$\displaystyle\sqrt{m}({\hat{\beta}}_{n}^{\\#}-\beta^{*})=\operatorname*{argmin}_{u}V_{n}(u)\overset{d}{\to}\operatorname*{argmin}_{u}V(u)=\begin{dcases}0&\text{for}\leavevmode\nobreak\
j\in S^{c},\\\ z_{j}&\text{for}\leavevmode\nobreak\ j\in S.\end{dcases}$

In the case (ii-c), let
$V_{n}(u):=(n\sqrt{m}/\eta_{n})(Z_{n}^{\\#}(\beta)-Z_{n}^{\\#}(\beta^{*}))+(\text{term
irrelevant to}\leavevmode\nobreak\ \beta)$, then we have

(194) $\displaystyle V_{n}(u)=$
$\displaystyle\frac{\sqrt{n}}{\eta_{n}}\sqrt{\frac{n}{m}}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2\sqrt{n}}{\eta_{n}}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
$\displaystyle+$
$\displaystyle\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{\eta_{n}}\sum_{j}\left(\frac{\left|u_{j}+\sqrt{m}\beta^{*}_{j}\right|-\left|\sqrt{m}\beta^{*}_{j}\right|}{\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{1}}}I(\beta^{*}_{j}\neq
0)+\frac{|u_{j}|}{\left|z_{j}\right|^{\gamma_{1}}}I(\beta^{*}_{j}=0)\right)$
$\displaystyle+$
$\displaystyle\sum_{j}\left(\left|\beta^{*}_{j}+\frac{z_{j}}{\sqrt{m}}\right|^{\gamma_{2}}I(\beta^{*}_{j}\neq
0)+\frac{\left|z_{j}\right|^{\gamma_{2}}}{\sqrt{m^{\gamma_{2}}}}I(\beta^{*}_{j}=0)\right)\left|u_{j}-z_{j}\right|.$

Let $V(u):=\lim_{n\to\infty}V_{n}(u)$, then we have

(195) $\displaystyle
V(u)=\sum_{j}\left(|\beta^{*}_{j}|^{\gamma_{2}}\left|u_{j}-z_{j}\right|I(\beta^{*}_{j}\neq
0)+\frac{\rho_{0}|u_{j}|}{|z_{j}|^{\gamma_{1}}}I(\beta^{*}_{j}=0)\right).$

Therefore, we obtain

(196)
$\displaystyle\sqrt{m}({\hat{\beta}}_{n}^{\\#}-\beta^{*})=\operatorname*{argmin}_{u}V_{n}(u)\overset{d}{\to}\operatorname*{argmin}_{u}V(u)=\begin{dcases}0&\text{for}\leavevmode\nobreak\
j\in S^{c},\\\ z_{j}&\text{for}\leavevmode\nobreak\ j\in S.\end{dcases}$

Case II. Let $l=\sqrt{n}$. Then, (173) reduces to

(197) $\displaystyle Z_{n}^{\\#}(\beta)-Z_{n}^{\\#}(\beta^{*})$ (198)
$\displaystyle=$
$\displaystyle\frac{1}{n}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2}{n}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
$\displaystyle+$
$\displaystyle\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{n\sqrt{n}}\sum_{j}\frac{1}{\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{1}}}\left(\left|u_{j}+\sqrt{n}\beta^{*}_{j}\right|-\left|\sqrt{n}\beta^{*}_{j}\right|\right)$
$\displaystyle+$
$\displaystyle\frac{\eta_{n}}{n\sqrt{nm^{\gamma_{2}}}}\sum_{j}\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{2}}\left(\left|u_{j}-\sqrt{\frac{n}{m}}z_{j}\right|-\left|\sqrt{\frac{n}{m}}z_{j}\right|\right)$
(199) $\displaystyle=$
$\displaystyle\frac{1}{n}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2}{n}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
$\displaystyle+$
$\displaystyle\frac{\lambda_{n}}{n\sqrt{n}}\sum_{j}\left(\frac{\left|u_{j}+\sqrt{n}\beta^{*}_{j}\right|-\left|\sqrt{n}\beta^{*}_{j}\right|}{\left|\beta^{*}_{j}+\frac{z_{j}}{\sqrt{m}}\right|^{\gamma_{1}}}I(\beta^{*}_{j}\neq
0)+\frac{\sqrt{m^{\gamma_{1}}}|u_{j}|}{\left|z_{j}\right|^{\gamma_{1}}}I(\beta^{*}_{j}=0)\right)$
$\displaystyle+$
$\displaystyle\frac{\eta_{n}}{n\sqrt{n}}\sum_{j}\left(\left|\beta^{*}_{j}+\frac{z_{j}}{\sqrt{m}}\right|^{\gamma_{2}}I(\beta^{*}_{j}\neq
0)+\frac{\left|z_{j}\right|^{\gamma_{2}}}{\sqrt{m^{\gamma_{2}}}}I(\beta^{*}_{j}=0)\right)\left(\left|u_{j}-\sqrt{\frac{n}{m}}z_{j}\right|-\left|\sqrt{\frac{n}{m}}z_{j}\right|\right).$

Consider the case (iii) where

(200)
$\displaystyle\frac{1}{n}\gtrsim\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{n\sqrt{n}}\leavevmode\nobreak\
\text{and}\leavevmode\nobreak\ \frac{1}{n}\gtrsim\frac{\eta_{n}}{n\sqrt{n}},$

that is,

(201)
$\displaystyle\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{\sqrt{n}}\to\lambda_{1}\geq
0\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
\frac{\eta_{n}}{\sqrt{n}}\to\eta_{0}\geq 0.$

Let $V_{n}(u):=n(Z_{n}^{\\#}(\beta)-Z_{n}^{\\#}(\beta^{*}))$, then we have

(202) $\displaystyle V_{n}(u)=$ $\displaystyle
u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-2u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
$\displaystyle+$
$\displaystyle\frac{\lambda_{n}}{\sqrt{n}}\sum_{j}\left(\frac{\left|u_{j}+\sqrt{n}\beta^{*}_{j}\right|-\left|\sqrt{n}\beta^{*}_{j}\right|}{\left|\beta^{*}_{j}+\frac{z_{j}}{\sqrt{m}}\right|^{\gamma_{1}}}I(\beta^{*}_{j}\neq
0)+\frac{\sqrt{m^{\gamma_{1}}}|u_{j}|}{\left|z_{j}\right|^{\gamma_{1}}}I(\beta^{*}_{j}=0)\right)$
$\displaystyle+$
$\displaystyle\frac{\eta_{n}}{\sqrt{n}}\sum_{j}\left(\left|\beta^{*}_{j}+\frac{z_{j}}{\sqrt{m}}\right|^{\gamma_{2}}I(\beta^{*}_{j}\neq
0)+\frac{\left|z_{j}\right|^{\gamma_{2}}}{\sqrt{m^{\gamma_{2}}}}I(\beta^{*}_{j}=0)\right)\left(\left|u_{j}-\sqrt{\frac{n}{m}}z_{j}\right|-\left|\sqrt{\frac{n}{m}}z_{j}\right|\right).$

Let $V(u):=\lim_{n\to\infty}V_{n}(u)$, then we have

(203) $\displaystyle V(u)=$ $\displaystyle u^{\top}Cu-2u^{\top}W$ (204)
$\displaystyle+$
$\displaystyle\sum_{j}\left(\frac{\lambda_{1}|u_{j}|}{|z_{j}|^{\gamma_{1}}}I(\beta^{*}_{j}=0)+\eta_{0}\left|\beta^{*}_{j}\right|^{\gamma_{2}}\left(\left|u_{j}-\sqrt{r_{0}}z_{j}\right|-\left|\sqrt{r_{0}}z_{j}\right|\right)I(\beta^{*}_{j}\neq
0)\right).$

Therefore, we obtain

(205)
$\displaystyle\sqrt{n}({\hat{\beta}}_{n}^{\\#}-\beta^{*})=\operatorname*{argmin}_{u}V_{n}(u)\overset{d}{\to}\operatorname*{argmin}_{u}V(u)$
(206) $\displaystyle=$
$\displaystyle\operatorname*{argmin}_{u}\left\\{u^{\top}Cu-2u^{\top}W+\sum_{j}\left(\frac{\lambda_{1}|u_{j}|}{|z_{j}|^{\gamma_{1}}}I(\beta^{*}_{j}=0)+\eta_{0}\left|\beta^{*}_{j}\right|^{\gamma_{2}}\left|u_{j}-\sqrt{r_{0}}z_{j}\right|I(\beta^{*}_{j}\neq
0)\right)\right\\}.$

Consider the case (iv) where

(207)
$\displaystyle\frac{1}{n}\gtrsim\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{n\sqrt{n}}\leavevmode\nobreak\
\text{and}\leavevmode\nobreak\
\frac{\eta_{n}}{n\sqrt{n}}\gg\frac{1}{n}\gtrsim\frac{\eta_{n}}{n\sqrt{nm^{\gamma_{2}}}},$

that is,

(208)
$\displaystyle\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{\sqrt{n}}\to\lambda_{1}\geq
0,\leavevmode\nobreak\ \frac{\eta_{n}}{\sqrt{n}}\to\infty,\leavevmode\nobreak\
\text{and}\leavevmode\nobreak\
\frac{\eta_{n}}{\sqrt{nm^{\gamma_{2}}}}\to\eta_{1}\geq 0.$

Let $V_{n}(u):=n(Z_{n}^{\\#}(\beta)-Z_{n}^{\\#}(\beta^{*}))+(\text{term
irrelevant to}\leavevmode\nobreak\ \beta)$ and
$V(u):=\lim_{n\to\infty}V_{n}(u)$. Then, we have (202) and

(209) $\displaystyle
V(u)=\begin{dcases}u^{\top}Cu-2u^{\top}W+\sum_{j}\left(\frac{\lambda_{1}}{|z_{j}|^{\gamma_{1}}}\left|u_{j}\right|+\eta_{1}|z_{j}|^{\gamma_{2}}|u_{j}-\sqrt{r_{0}}z_{j}|\right)I(\beta^{*}_{j}=0)&\text{for}\leavevmode\nobreak\
u_{S}=r_{0}z_{S},\\\ \infty&\text{otherwise}.\end{dcases}$

Therefore, we obtain

(210)
$\displaystyle\sqrt{n}({\hat{\beta}}_{n}^{\\#}-\beta^{*})=\operatorname*{argmin}_{u}V_{n}(u)$
(211) $\displaystyle\overset{d}{\to}$
$\displaystyle\operatorname*{argmin}_{u}V(u)=\operatorname*{argmin}_{u\in\mathcal{U}}\left\\{u^{\top}Cu-2u^{\top}W+\sum_{j}\left(\frac{\lambda_{1}}{|z_{j}|^{\gamma_{1}}}\left|u_{j}\right|+\eta_{1}|z_{j}|^{\gamma_{2}}|u_{j}-\sqrt{r_{0}}z_{j}|\right)\right\\},$
(212) $\displaystyle\mathcal{U}:=\left\\{u\leavevmode\nobreak\
\middle|\leavevmode\nobreak\ u_{S}=r_{0}z_{S}\right\\}.$

Consider the case (v) where

(213)
$\displaystyle\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{n\sqrt{n}}\gg\frac{1}{n}\gtrsim\frac{\lambda_{n}}{n\sqrt{n}}\leavevmode\nobreak\
\text{and}\leavevmode\nobreak\ \frac{1}{n}\gtrsim\frac{\eta_{n}}{n\sqrt{n}},$

that is,

(214)
$\displaystyle\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}}{\sqrt{n}}\to\infty,\frac{\lambda_{n}}{\sqrt{n}}\to\lambda_{0}\geq
0,\leavevmode\nobreak\ \text{and}\leavevmode\nobreak\
\frac{\eta_{n}}{\sqrt{n}}\to\eta_{0}\geq 0.$

Let $V_{n}(u):=n(Z_{n}^{\\#}(\beta)-Z_{n}^{\\#}(\beta^{*}))$ and
$V(u):=\lim_{n\to\infty}V_{n}(u)$. Then, we have (202) and

(215) $\displaystyle
V(u)=\begin{dcases}u^{\top}Cu-2u^{\top}W+\sum_{j}\left(\lambda_{0}u_{j}\operatorname{sgn}(\beta^{*}_{j})/|\beta^{*}_{j}|^{\gamma_{1}}+\eta_{0}\left|\beta^{*}_{j}\right|^{\gamma_{2}}\left|u_{j}-\sqrt{r_{0}}z_{j}\right|\right)&\text{for}\leavevmode\nobreak\
u_{S^{c}}=0,\\\ \infty&\text{otherwise}.\end{dcases}$

Therefore, we obtain

(216)
$\displaystyle\sqrt{n}({\hat{\beta}}_{n}^{\\#}-\beta^{*})=\operatorname*{argmin}_{u}V_{n}(u)$
(217) $\displaystyle\overset{d}{\to}$
$\displaystyle\operatorname*{argmin}_{u}V(u)=\operatorname*{argmin}_{u\in\mathcal{U}}\left\\{u^{\top}Cu-2u^{\top}W+\sum_{j}\left(\frac{\lambda_{0}u_{j}\operatorname{sgn}(\beta^{*}_{j})}{|\beta^{*}_{j}|^{\gamma_{1}}}+\eta_{0}\left|\beta^{*}_{j}\right|^{\gamma_{2}}\left|u_{j}-\sqrt{r_{0}}z_{j}\right|\right)\right\\},$
(218) $\displaystyle\mathcal{U}:=\left\\{u\leavevmode\nobreak\
\middle|\leavevmode\nobreak\ u_{S^{c}}=0\right\\}.$

Case III. Let $l=n/\lambda_{n}$. Suppose that $n/\lambda_{n}\to\infty$ Then,
(173) reduces to

(219) $\displaystyle Z_{n}^{\\#}(\beta)-Z_{n}^{\\#}(\beta^{*})$ (220)
$\displaystyle=$
$\displaystyle\frac{\lambda_{n}^{2}}{n^{2}}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2\lambda_{n}}{n\sqrt{n}}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
$\displaystyle+$
$\displaystyle\frac{\sqrt{m^{\gamma_{1}}}\lambda_{n}^{2}}{n^{2}}\sum_{j}\frac{1}{\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{1}}}\left(\left|u_{j}+\frac{n}{\lambda_{n}}\beta^{*}_{j}\right|-\left|\frac{n}{\lambda_{n}}\beta^{*}_{j}\right|\right)$
$\displaystyle+$
$\displaystyle\frac{\lambda_{n}\eta_{n}}{n^{2}\sqrt{m^{\gamma_{2}}}}\sum_{j}\left|z_{j}+\sqrt{m}\beta^{*}_{j}\right|^{\gamma_{2}}\left(\left|u_{j}-\frac{n}{\sqrt{m}\lambda_{n}}z_{j}\right|-\left|\frac{n}{\sqrt{m}\lambda_{n}}z_{j}\right|\right)$
(221) $\displaystyle=$
$\displaystyle\frac{\lambda_{n}^{2}}{n^{2}}u^{\top}\left(\frac{1}{n}X^{\top}X\right)u-\frac{2\lambda_{n}}{n\sqrt{n}}u^{\top}\left(\frac{1}{\sqrt{n}}X^{\top}\varepsilon\right)$
$\displaystyle+$
$\displaystyle\frac{\lambda_{n}^{2}}{n^{2}}\sum_{j}\left(\frac{\left|u_{j}+\frac{n}{\lambda_{n}}\beta^{*}_{j}\right|-\left|\frac{n}{\lambda_{n}}\beta^{*}_{j}\right|}{\left|\beta^{*}_{j}+\frac{z_{j}}{\sqrt{m}}\right|^{\gamma_{1}}}I(\beta^{*}_{j}\neq
0)+\frac{\sqrt{m^{\gamma_{1}}}|u_{j}|}{\left|z_{j}\right|^{\gamma_{1}}}I(\beta^{*}_{j}=0)\right)$
$\displaystyle+$
$\displaystyle\frac{\lambda_{n}\eta_{n}}{n^{2}}\sum_{j}\left(\left|\beta^{*}_{j}+\frac{z_{j}}{\sqrt{m}}\right|^{\gamma_{2}}I(\beta^{*}_{j}\neq
0)+\frac{\left|z_{j}\right|^{\gamma_{2}}}{\sqrt{m^{\gamma_{2}}}}I(\beta^{*}_{j}=0)\right)\left(\left|u_{j}-\frac{n}{\sqrt{m}\lambda_{n}}z_{j}\right|-\left|\frac{n}{\sqrt{m}\lambda_{n}}z_{j}\right|\right).$

Consider the case (vi) where

(222)