We introduce a tensor-based model of shared representation  for meta-learning from a diverse set of tasks.  Prior works on  learning linear representations for meta-learning assume that there is a common shared representation across different tasks, and do not consider the additional task-specific observable side information. In this work, we model the meta-parameter through an order-$3$ tensor, which can adapt to the observed task features of the task. We propose two methods to estimate the underlying tensor. The first method solves a tensor regression problem and works under natural assumptions on the data generating process. The second method uses the method of moments under additional distributional assumptions and has an improved sample complexity in terms of the number of tasks.

We also focus on the meta-test phase, and consider estimating task-specific parameters on a new task. Substituting the estimated tensor from the first step allows us estimating the task-specific parameters with very few samples of the new task, thereby showing the benefits of learning tensor representations for meta-learning. Finally, through simulation and several real-world datasets, we evaluate our methods and show that it improves over previous linear models of shared representations for meta-learning.

§ INTRODUCTION

One of the major challenges in modern machine learning is  training a model with limited amounts of data. This is particularly important in settings where data is scarce and new data is costly to acquire. In recent years, several techniques like data augmentation, transfer learning have been proposed to address problems with limited data. The focus of this paper is meta-learning, which has turned out to be an important framework to address such problems. The main idea behind meta-learning is to design learning algorithms that can leverage prior learning experience to adapt to a new problem quickly, and learn a useful algorithm with few samples. Such approaches have been quite successful in diverse applications like natural language processing <cit.>, robotics <cit.>, and healthcare <cit.>.

Meta-learning algorithms are often given a family of related tasks and attempt to use few samples on a new related task by utilizing the overlap between the new test task and already seen training tasks. In that sense, a meta-learning algorithm is learning to learn on new tasks, and performance improves with experience and number of tasks <cit.>. Despite immense success, we are yet to fully understand the theoretical foundations of meta-learning algorithms. The most promising theoretical direction stems from representation learning. The main idea is that the tasks share a common shared representation and a task-specific representation <cit.>: if the shared representation is learned from training tasks, then the task-specific representation for the new task can be learned with few samples.

Current  models of shared representations for meta-learning do not take into account two observations – (1) the training tasks are often heterogeneous, and the shared representation cannot be captured by a single parameter, (2) tasks often come with additional task-specific observable side information, and they should be part of any representation-based model of meta-learning. The first situation often arises in robotics, and various reinforcement learning environments <cit.>, while the latter is prevalent in recommender system <cit.>, where items (tasks) that users rate often come with observable features.

We aim to understand meta-learning of features for settings where task-specific observable features affect the outcome. In particular, we are interested in the following questions. (1) What is the appropriate generalization of meta-learning of linear representation with task-specific observable features? (2) Moreover, given samples from $T$ tasks, how can we efficiently learn such a representation and does it improve sample efficiency on a new task?

§.§ Contributions

Tensor Based Model. We propose a tensor based model of representations for meta-learning representations for a diverse set of tasks. In particular, we model the meta-parameter through a tensor of order-$3$, which can be thought of as a multi-linear function mapping a tuple of (input feature, observed task feature, unobserved task feature) to a real-valued output. As our model considers task-specific observed features, the meta-parameter can adapt to particular task and generalizes the matrix-based linear representations proposed by <cit.>.

Estimation. We first determine the identifiable component of the shared representation based model, and estimate the first two factors of the underlying tensor in the meta-training phase. We propose two methods – (1) tensor regression based method works with natural assumptions on the data generating process, and (2) method of moments based estimation works under additional distributional assumptions, but has improved sample complexity in terms of the number of tasks.

Meta-Test Phase. After estimating the shared parameters, we focus on the meta-test phase, where a new task is given. We show that substituting the estimated factors from the first step provably improves error in estimating the task-specific parameters on a new task. In particular, the excess test error on the new task is bounded by $O\left(\frac{r^2}{N_2}\right)$ where $r$ is the rank of the underlying tensor and $N_2$ is the number of samples from the new task. As tensor rank $r$ can be quite small compared to the dimensions, this highlights the benefits of learning task-adaptive representations in meta-learning. Finally, through a simulated dataset and several real-world datasets, we evaluate our methods and show that it improves over previous models of learning shared representations for meta-learning.

§.§ Related Work

<cit.> was the first  to prove generalization bound for multitask learning problem. However, they considered a model of multitask learning where tasks with shared representation are sampled from a generative model. <cit.> developed general uniform-convergence based framework to analyze multitask representation learning. However, they assume oracle access to a global empirical risk minimizer. On the other hand, we provide specific algorithms and also consider task-specific side information.

The work closest to ours is <cit.>, who proposed a linear model for learning representation in meta-learning. Our model can be thought of as a general model of theirs as we do not assume a fixed low-dimensional representation across tasks, and can adapt to observable side-information of the tasks. We also note that <cit.> generalized the linear model <cit.> to consider transfer learning with general class of functions, however, they assume oracle access to a global empirical risk minimizer, and the common representation (a shared function) does not adapt to observable features of the tasks. Finally, <cit.> also considered the problem of learning shared representations and obtained similar results. Compared to <cit.>, they consider general non-linear representations, but the representation again does not depend on the observable features of the task.

Our work is also related to the conditional meta-learning framework introduced by <cit.>. Conditional meta-learning aims to learn a conditioning function that maps task-specific side information to a meta-parameter suitable for the task. <cit.> studies a biased regularization formulation where the goal is to find task-specific parameter close to a bias vector, possibly dependent on side-information. On the other hand, <cit.> takes a structured prediction framework, and only proves generalization bounds.
Although our framework falls within the conditional meta-learning framework, we want to understand the benefits of representation learning on a new task.

In this work, we aim to understand meta-learning through a representation learning viewpoint. However, in recent years, several works have attempted to improve our understanding of meta-learning from other viewpoints. These include optimization <cit.>, train-validation split <cit.>, and convexity <cit.>. Additionally, there are several recent works on understanding gradient based meta-learning <cit.>, but their setting is very different from ours.

Finally, we use tensors to model the meta-parameter in the presence of task-specific side information. Our estimation method uses tensor regression <cit.> and tensor decomposition <cit.>. For tensor regression, we build upon the algorithm proposed by <cit.>, and for tensor decomposition we use a robust version introduced by <cit.>.

§ PRELIMINARIES

We will consider standard two-stage model of meta-learning, consisting of a meta-training phase and a meta-test phase. In the meta-training stage, we see $N$ samples from $T$ training tasks and learn a meta parameter. In the meta-test stage, we see $N_2$ samples from a fixed target task (say task $0$) and learn a target-specific parameter conditioned on the meta-parameter and  features of the new task $0$. We first define our response model which specifies the particular model of shared linear representation.

Response model.
There are $T$ training tasks and each task is associated with a pair of observed and unobserved task feature vector.
Task $t$ is characterized by $(Y_t, Z_t)$ where $Y_t \in \R^{d_2}$ is the observed task feature vector and $Z_t \in \R^{d_3}$ is the unobserved task feature vector for the $t$-th task.

A sample from task $t$ is specified by a tuple $(X,Y_t,Z_t)$, where $X$ is some user feature vector. Given such a data tuple $(X,Y_t,Z_t)$ the response is given as

\begin{align}
R & = A(X,Y_t,Z_t) + \veps
= \sum_{i,j,k} A_{i,j,k} X_i Y_{tj} Z_{tk} + \veps \label{eq:response-model}
\end{align}

Here the noise variable $\veps \sim \Normal(0,\sigma^2)$, and $A \in \R^{d_1 \times d_2 \times d_3}$ is the system tensor which we treat as a multi-linear real-valued function on $\R^{d_1} \times \R^{d_2} \times \R^{d_3}$. Note that, our model generalizes the linear model proposed by <cit.>, and the meta-parameter (tensor $A$) is not a fixed parameter, and adapts to the observed feature / side information for task $t$. We will assume that the tensor $A$ has CP-rank $r$ i.e. there exist matrices $A^1 \in \R^{d_1 \times r}$, $A^2 \in \R^{d_2 \times r}$, and $A^3 \in \R^{d_3 \times r}$ such that

\[
A_{i,j,k} = \sum_{s=1}^r A^1_{si} A^2_{sj} A^3_{sk}
\]

Following <cit.>, we will write $A = \llbracket \Identity_r; A^1, A^2, A^3\rrbracket$ to denote the rank-$r$ decomposition of the tensor $A$. Notice that here we assume that all the singular values of the tensor $A$ is one. Without making strong assumptions on the unobserved task features $Z_t$, general singular values cannot be identified.[We provide a counter-example in the appendix.]

Training data.
Let $P$ be a distribution over the feature vectors $X_i$'s which we will often refer to as user feature vectors. Let $Q$ be a joint distribution over observed and unobserved task feature vectors.
Let $\{ X_i : i \in [N] \}$ and $\{ (Y_t,Z_t) : t \in [T] \}$ be independent random variables, where $X_1,\dotsc,X_N \simiid P$ and $(Y_1,Z_1),\dotsc,(Y_T,Z_T) \simiid Q$.
Conditional on these (random) feature vectors, let $R_{1}, \dotsc, R_{N}$ be independent realizations of $R$ from the response model in <Ref>, where

\begin{equation}\label{eq:training-samples}
R_{i} = A(X_i, Y_{t(i)}, Z_{t(i)}) + \veps_i.
\end{equation}

Here $t: [N] \rightarrow [T]$ is a mapping that specifies, for each training instance $i$, corresponding task $t(i)$. Therefore, the training data is given as $\{X_i, Y_{t(i)},  R_i\}_{i \in [N]}$.

Meta-Test data. At test time we are given a fixed task (say $0$) with observed feature $Y_0$ and unobserved feature $Z_0$. We are given $N_2$ instances from this new task,  $\{X_i, Y_0, R_i\}_{i\in[N_2]}$ where $X_1,\ldots,X_{N_{2}} \simiid P$. Our goal is to design a predictor $f \colon \R^{d_1} \times \R^{d_2} \to \R$ that maps an input feature and an observed task feature to a predicted response. We will evaluate our predictor by its mean squared error on the new task.

\begin{align*}
\MSE(f)
& = \E\sbr{ (f(X,Y_0) - A(X,Y_0,Z_0) -\veps)^2 } = \sigma^2 + \E\sbr{ (f(X,Y_0) - A(X,Y_0,Z_0))^2 } .
\end{align*}

In order to design the predictor on the new task, we need estimates of tensor $A$, and unobserved task feature on the new task $Z_0$.

For a matrix $B \in \R^{d_1\times d_2}$ we will write $\norm{B}_\op$ to denote its operator norm, which is defined as $\norm{B}_\op = \max_{x \in \R^{d_2}} \frac{\norm{Bx}_2}{\norm{x}_2}$. For matrix $B$, we will write $\norm{B}_F = \sqrt{\sum_{i,j} B_{ij}^2}$ to denote its Frobenius norm.
For a tensor $A \in \R^{d_1 \times d_2 \times d_3}$ we write its spectral norm $\norm{A}_\op = \max_{\norm{x}_2 = \norm{y}_2 = \norm{z}_2 = 1} \abs{A(x,y,z)}$. Like a matrix, we will also write $\norm{A}_F = \sqrt{\sum_{i,j,k} A_{i,j,k}^2}$ to denote the Frobenius norm of the tensor $A$. We sometimes use the tensor by slices, for the $3$-order tensor $A$, we denote its horizontal slices as $A_{j::}$, for $j\in [d_{1}]$.

We will use two types of special matrix products in our paper. Given matrices $A \in \R^{d_1 \times d_2}$ and $B \in \R^{d_3 \times d_4}$, the Kronecker product $A \otimes B \in \R^{d_1 d_3 \times d_2 d_4}$ is

A \otimes B = [a_1 \otimes b_1\  a_1 \otimes b_2 \ldots a_{d_2} \otimes b_{d_4-1}\  a_{d_2} \otimes b_{d_4}]

For matrices $A \in \R^{d_1 \times d_2}$ and $B \in \R^{d_3 \times d_2}$, their Khatri-Rao product $A \odot B \in \R^{d_1 d_3 \times d_2}$ is

A \odot B = [a_1 \otimes b_1 \ a_2 \otimes b_2 \ \ldots a_{d_2} \otimes b_{d_2}]

In addition, we denote standard basis vector as $\mathbf{e}_{i}$ whose coordinates are all zero, except $i$-th equals $1$.

§ ESTIMATION

We estimate the parameters of our model in two steps – (1)  estimate the shared tensor using the meta-training data, and (2)  estimate the parameters of the test task using the meta-test data and the estimate of the shared tensor. However, it turns out that even when there is a single task in the meta-training phase, the third factor $A^3$ cannot be identified for general tensor with orthogonal factors. In the appendix, we construct an example which shows that two different tensors with identical $A^1, A^2$ but different $A^3$ leads to the same observed outcomes. Therefore, we estimate the first two factors of the tensor $A$ in the meta-training phase. In the meta-test phase, we substitute estimates of $A^1$ and $A^2$ and recover the parameters for the new task. We provide two ways to estimate the factors. The first method uses tensor regression; the second method uses the method-of-moments.

§.§ Tensor-Regression Based Estimation

In order to see why tensor regression can help us recover the shared tensor, we first show an alternate way to write the response $R_i$, as defined in <Ref>.

\begin{equation}\label{eq:unobserved-task-feature}
\calZ = \begin{bmatrix} Z_1 & \cdots & Z_T \end{bmatrix}^\T \in \R^{T\times d_3}
\end{equation}

to be the matrix corresponding to the unobserved features of the $T$ training tasks. Then $A \times_3 \calZ = A(I_{d_1}, I_{d_2}, \calZ) \in \R^{d_1 \times d_2 \times T}$ is the tensor corresponding to  unobserved parameters, defined as

\begin{equation*}
(A\times_3 \calZ)_{i,j,t} = \sum_{k=1}^d A_{i,j,k} \calZ_{t,k}.
\end{equation*}

Additionally, we define a covariate tensor $\calX_i \in \R^{d_1 \times d_2 \times T}$ corresponding to the observed features as:

\begin{equation}\label{eq:covariate-tensor}
\begin{split}
\calX_i(\cdot,\cdot,t) =
\begin{cases}
X_i Y_{t(i)}^\T & \text{if $t = t(i)$} \\
0_{d_1 \times d_2} & \text{o.w.}
\end{cases}
\end{split}
\end{equation}

Then, according to <Ref>, we have the following linear regression model for the $i$-th response.

\begin{equation}\label{eq::linear-regression-model-AxZ}
R_i =  \ip{ \calX_i, A\times_3 \calZ } + \veps_i.
\end{equation}

Therefore, we can use tensor regression to get an estimate of $A \times_3 \calZ$.  Since the three factors of $A$ are $A^1, A^2$, and $A^3$, it can be easily seen that the CP-decomposition of $A\times_3 \calZ$ is $\llbracket \Identity_r; A^1, A^2, \calZ A^3\rrbracket = \llbracket G^{-1}; A^1, A^2, \calZ A^3 G\rrbracket$. Here $G$ is a diagonal matrix with $i$-th entry $1/\norm{\calZ A^3_i}_2$ and normalizes the columns of $\calZ A^3$. Because of this particular form of the tensor $A \times_3 \calZ$, we can run a tensor decomposition of the estimate of $A\times_3 \calZ$ to recover $A^1, A^2$, and $\calZ A^3 G$. However, there is a catch as we have an estimate of $A \times_3 \calZ$, instead of the exact tensor. So we need the tensor decomposition method to be robust to the estimation error.

Algorithm <ref> describes the full algorithm for recovering $A$ from the training samples.

$(X_i,Y_{t(i)}, R_i)$ for $i=1,\ldots,N$

* Solve the following tensor regression problem:

\begin{align}\label{eq:tensor-regression-1}
&\BB = \argmin_{B \in \R^{d_1 \times d_2 \times T} } \left\{ \frac{1}{N} \sum_{i=1}^{N} \left(R_i - \left \langle \calX_i, B \right \rangle \right)^2 + \lambda \norm{B}_{S} \right \}
\end{align}

* Run a robust tensor decomposition of $\BB$ of CP-rank $r$:

\llbracket\WW; \BB^1, \BB^2, \BB^3\rrbracket \leftarrow \textrm{Robust-Tensor-Decomposition}(\BB, r)

Return $\widehat{A \times_3 \calZ} = \llbracket \Identity_r; \BB^1, \BB^2, \BB^3 \WW\rrbracket$.

Tensor-Regression Based Estimation

Tensor Regression Details and Guarantees.
Throughout this section, we will make the following assumptions about the data generating distribution.

*  $X_1,\ldots,X_N \simiid \Normal(0,\Sigma)$.

*  $Y_1,\ldots,Y_T \simiid \Normal(0, \Sigma_y)$.

*  For each $i$, $t(i) \sim \textrm{Unif}\set{1,\ldots,T}$.

<Ref> is the tensor regression step to obtain an estimate of $B = A \times_3 \calZ$. We use a regularized
least squared regression, introduced by <cit.>. Here $\norm[0]{B}_{S}$ is the overlapped Schatten-1 norm of the tensor $B$, which is defined as the average of mode-wise nuclear norms i.e. $\norm[0]{B}_S = 1/3 \sum_{k=1}^3 \norm[0]{B_{(k)}}_{\star}$. Since matrix nuclear norm is a convex function, the tensor regression problem stated in <Ref> is also a convex problem, and can be solved efficiently.

For a tensor of dimension $d_1 \times d_2 \times T$, we introduce the following notation, which will appear frequently in our bounds.

\begin{equation}\label{eq:D1-defn}
D_1 = \sqrt{d_1} + \sqrt{d_2} + \sqrt{T} + \sqrt{d_1 T} + \sqrt{d_2 T} + \sqrt{d_1 d_2}
\end{equation}

The next theorem states the guarantees of the tensor regression step.

Suppose Assumptions <ref>-<ref> hold, and $N \ge O\left(\frac{\lambda_{\max}(\Sigma)}{\lambda_{\min}(\Sigma_y)}rD_1^2\right)$. Then, with probability at least $1 - e^{-\Omega(D_1^2)}$, we have

\norm[1]{\BB- B}_F \le O\left(\frac{\sigma T  D_1 \sqrt{r}}{\lambda_{\min}(\Sigma_y) \lambda_{\min}(\Sigma) \sqrt{N}} \right).

The full proof is provided in the supplementary material. Here we provide an overview of the main steps of the proof. Our analysis builds upon the work by <cit.>, who analyzed the performance of tensor regression with overlapped Schatten-1 norm. The main ingredient of the proof is to show that under certain assumptions restricted strong convexity (RSC) holds. This property was introduced by <cit.> in the context of several matrix estimation problems, and ensures that the loss function has sufficient curvature to ensure consistent recovery of the unknown parameter. <cit.> proves that when the covariate tensors $\calX_i$ are normally distributed, RSC holds with a fixed constant. For our setting, the covariate tensors are defined in <Ref> and are not necessarily distributed from a multivariate Gaussian distribution. However, we can generalize the original proof of <cit.> to show that under Assumptions <ref> ($X_i$-s are normally distributed) and <ref> (tasks are sampled uniformly at random), RSC still holds for our setting, but with constant $O(1/T)$. Then we show that the parameter $\lambda$ can be chosen to be a suitably large constant to get the error bounds of <Ref>.

Tensor Decomposition Details and Guarantees:
Having recovered the tensor $A \times_{3} \calZ$, we now aim to recover the factors $A^1,A^2$, and $\calZ A^3$. Since we do not have the exact tensor $A \times_{3} \calZ$, but rather an estimate of the tensor, we apply robust tensor decomposition method (step 2 of Algorithm <ref>) to recover the factors of  $A \times_3 \calZ$.

For robust tensor decomposition method, we will apply the algorithm of <cit.>. It is in general impossible to recover the factors of a noisy tensor without making any assumptions. So we will make the following assumptions about the underlying tensor $A = [\Identity_r; A^1, A^2, A^3]$. We will write $d = \max\{d_1, d_2, T\}$.

* The columns of the factors of $A$ are orthogonal i.e., $\langle A^1_i, A^1_j \rangle = \langle A^2_i, A^2_j \rangle = \langle A^3_i, A^3_j \rangle = 0$ for all $i \neq j$.

* The components have bounded $2\rightarrow p$ for some $p$ i.e.
$\exists p < 3$,

$$ \max\set{\norm[1]{A^{1\T}}_{2 \rightarrow p}, \norm[1]{}1]{A^{2\T}}_{2 \rightarrow p}, \norm[1]{A^{3\T}}_{2 \rightarrow p}} \le 1 + o(1) . $$

* Rank is bounded i.e. $r = o(d)$.

Additionally, recall the definition of $\calZ$, the matrix of unobserved features.

\begin{equation*}%\label{eq::unobserved-task-feature}
\calZ = \begin{bmatrix} Z_1 & \cdots & Z_T \end{bmatrix}^\T \in \R^{T\times d_3}
\end{equation*}

* $\frac{1}{d_3^{0.5+\gamma}}\Identity_{d_3} \preccurlyeq \calZ^\T \calZ \preccurlyeq \frac{1}{\sqrt{d_3}}\Identity_{d_3}$ for some $\gamma > 0$.

* $\kappa(\calZ^\T \calZ) = \frac{\lambda_{\max}(\calZ^\T \calZ)}{\lambda_{\min}(\calZ^\T \calZ)} \le 1 + O(\sqrt{r/d})$.

Although assumptions (Z1) and (Z2) might seem strong requirements on the matrix of unobserved features, they are usually satisfied when the unobserved task feature matrix is drawn from gaussian distribution. For example, if $Z_t \simiid \Normal(0, \nu \Identity_{d_3})$ then the assumptions hold for small enough $\nu$.

Suppose tensor $A$ satisfies the assumptions (B1)-(B3),  the matrix of unobserved features $\calZ$ satisfies assumptions (Z1)-(Z2), and $N \ge \tilde{O}\left(\frac{\sigma^2 T^2  D_1^2 r }{\lambda_{\min}^2(\Sigma_y) \lambda^2_{\min}(\Sigma) } \right)$. Then the tensor $\hat{A} = [\Identity_r; \widehat{A^1},\widehat{A^2}, \widehat{\calZ A^3}]$ output by Algorithm <ref> satisfies

\begin{align*}
&\max\left\{ \norm[1]{\widehat{A^1} - A^1}_F, \norm[1]{\widehat{A^2} - A^2}_F\right\} \le \tilde{O}\left( \frac{\sigma T D_1 r}{\rho \sqrt{N}}\right), \ \ \norm[1]{\widehat{\calZ A^3} - \calZ A^3}_F \le  \tilde{O}\left( \frac{\sigma T D_1 r^{1.5}}{\rho \sqrt{ N}}\right)%\\ &\textrm{and} \
%\norm{\widehat{W} - W}_2 \le \tilde{O}\left(\frac{\sigma T D_1 r}{\rho \sqrt{N}}\right)
\end{align*}

where $\rho = \sqrt{\lambda_{\min}(\calZ^\T \calZ) } \lambda_{\min}(\Sigma_y) \lambda_{\min}(\Sigma)$.

The proof shows that when the assumptions (B1)-(B3) and (Z1)-(Z3) are satisfied, we can apply robust tensor decomposition method to the tensor $A \times_3 \calZ$. Note that the bound for the third factor $\calZ A^3$ is worse by a factor of $\sqrt{r}$. This is because we recover an estimate of $\calZ A^3G$ for a diagonal matrix $G$ from tensor decomposition and then post-multiply this estimate by another diagonal matrix to obtain an estimate of $\calZ A^3$.

§.§.§ Meta-Test

During the meta-test phase, we are given a new task (i.e. task $0$ with observed feature $Y_0 \in\R^{d_2}$, and hidden feature $Z_0 \in \R^{d_3}$), and our goal is to learn the unobserved parameter of this task with as few samples as possible. As is standard in the meta-learning literature, we get a new training sample from the new task, and our goal is to perform well on the test sample drawn from the new task. There are $N_2$ training samples from the new task, where the features $X_1,\ldots,X_{N_2}$ are drawn iid from a distribution $P$. We will assume each feature $X_i$ is mean-zero, has covariance matrix $\Sigma$ ($\E[X_i X_i^\T] = \Sigma$), and $\Sigma$-subgaussian i.e. $\E[\exp(v^\T X_i)] \le \exp\left(1/2\norm[0]{\Sigma^{1/2} v}^2_2\right)$. The observed responses on these $N_2$ points are given as

R_i = A(X_i,Y_0,Z_0) + \veps_i

where $\veps_i \sim_{\textrm{iid}} \Normal(0,1)$. Define $\calX_0$ to be the matrix corresponding to the features on the new task i.e.

\begin{equation*}
\calX_0 = \begin{bmatrix} X_1 & \cdots & X_{N_2} \end{bmatrix}^T \in \R^{N_2\times d_1}
\end{equation*}

We aim to estimate $A^{3\T} Z_0$ by substituting the estimates of $A^1$ and $A^2$.
Notice that the response $R_i$ can also be expressed as

R_i = (Y_0^\T A^2 \odot X_i^\T A^1) A^{3\T} Z_0 + \veps_i.

Therefore, we can solve the following least square regression problem.

\begin{equation}\label{eq:meta-test-1}
\widehat{A^{3\T} Z_0} = \argmin_{\alpha_0 \in \R^r} \norm{R - (Y_0^\T \hat{A^2} \odot \calX_0 \hat{A^1}) \alpha_0}_2^2.
\end{equation}

If we write $\VV = (Y_0^\T \hat{A^2} \odot \calX_0 \hat{A^1})$, then the solution of Problem  <ref>

is given as
$\widehat{A^{3\T} Z_0} = ( \VV^\T  \VV )^{-1} \VV^\T  R$. Now our prediction on a new test instance $X_0$ from the new task is given as $(Y_0^\T \hat{A^2} \odot X_0^\T \hat{A^1}) \widehat{A^{3\T} Z_0}$. With slight abuse of notation we will write this prediction as $\hat{A}(X_0, Y_0, \widehat{Z}_0)$. The next theorem bounds the mean squared error in the meta-test phase.

Suppose  $\max\{\norm[0]{\hat{A^1} - A^1}_F, \norm[0]{\hat{A^2} - A^2}_F \} \le \delta$. Additionally, $N_2 \ge \tilde{O}(r)$ and $\abs[0]{Y_0^\T \hat{A^2_i}} \ge \eta \norm[0]{Y_0}_2$ for all $i \in [r]$. Then for dimension-independent constants $B_1$, and $B_2$ we have

\begin{align*}&\E_{X_0}\left[ \left(R_0 - \hat{A}(X_0, Y_0, \widehat{Z}_0) \right)^2\right] = O\left(\sigma^2 + \frac{B_1}{\eta^2} r^2 \delta^2 + \frac{B_2}{\eta^2} \frac{r^2}{N_2} \right)
\end{align*}

with high probability.

The proof of the theorem shows that the mean squared error can be bounded as $O\del[0]{ r \norm[0]{\widehat{A^{3\T} {Z_0}} - A^{3\T} Z_0}_2^2 } + O\del[0]{ \delta^2 \norm[0]{\widehat{A^{3\T} Z_0}}_2^2 }$. Then we write down the first term as a sum of bias and variance term and establish respective bounds of $O(r^2/N_2)$ and $O(r^2 \delta^2)$. Finally, we show that the $L_2$-norm of $\widehat{A^{3\T} Z_0}$ cannot be too large and is bounded by $O(r)$. Substituting these three bounds on the upper bound on the mean squared error gives us the desired result. Note that the theorem requires a lower bound on the inner product between the new task feature $Y_0$ and the columns of $\hat{A^2}$. This can be avoided with a slightly worse dependence on $r$. First, we can eliminate all columns $i$ such that $\abs[0]{Y_0^\T \hat{A^2_i}} \ge \eta \norm[0]{Y_0}_2$. If there are $r'$ such columns, we work with a tensor of rank $r-r'$ in the meta-test phase. The reduction in rank increases mean squared error by at most $O(r^2 \eta^2)$. Now if we choose $\eta = \sigma / r$ we get a bound of $O(B_1 r^4 \delta^2 + B_2 r^4/N_2)$ on the excess error.

This theorem implies that for a new task, the number of samples needed is $N_2 = O(r^2/\epsilon)$ if we want to achieve a test error of $\epsilon$ on the new task. If we were to run a least squares regression on the new task from scratch, the required number of samples would have been $O((d_1 + d_2)/\epsilon)$. As the CP-rank of the tensor $A$ can be smaller (often a constant) than the dimension of the unobserved features, transfer of the knowledge of the tensor $A$ provides a significant reduction in the number of samples on the new task.

§ METHOD-OF-MOMENTS BASED ESTIMATION

In this section, we provide a new algorithm that estimates the underlying tensor $A$ and also has optimal dependence on the number of tasks ($T$) under some additional distributional assumptions. In particular, we will assume $X_i \simiid \Normal(0, \Identity_{d_1})$, $Y_t \simiid \Normal(0, \Identity_{d_2})$, and $Z_t \simiid \Normal(0, \Identity_{d_3})$. Our algorithm is based on repeated applications a method-of-moments based estimator proposed in <cit.>, and we  briefly summarize that estimator.

Suppose the $i$-th response is given as $R_i = X_i^\T B \alpha_{t(i)} + \veps_i$ and each $X_i \simiid \Normal(0, I_{d_1})$, and $B \in \R^{d_1 \times r}$ has orthonormal columns.

Then it is possible to recover $B$ from the top $r$ singular values of the statistic $\frac{1}{N}\sum_{i=1}^N R_i^2 X_i X_i^\T $.

Recovering $A^1$. For our setting, the $i$-th response is given as $R_i = A(X_i, Y_{t(i)}, Z_{t(i)}) + \veps_i$. If we want to recover the first factor $A^1$ then we can rewrite the $i$-th response as

\begin{align*}
R_i = X_i^\T A_{(1)} (Z_{t(i)} \otimes Y_{t(i)}) + \veps_i = X_i^\T A^{1} \underbrace{(A^{3} \odot A^{2})^\T (Z_{t(i)} \otimes Y_{t(i)})}_{:=\alpha_{t(i)}} + \veps_i.
\end{align*}

Since each $X_i$ is drawn from a standard normal distribution, we can recover $A^1$ from the top-$r$ singular values of the statistic $\frac{1}{N}\sum_{i=1}^N R_i^2 X_i X_i^\T$.

Recovering $A^2$. We can recover $A^2$ through a similar method. We can rewrite the $i$-th response as

\begin{align*}
R_i = Y_{t(i)}^\T A_{(2)} (Z_{t(i)} \otimes X_i) + \veps_i = Y_{t(i)}^\T A^{2} \underbrace{W (A^{3} \odot A^{1})^\T (Z_{t(i)} \otimes X_i)}_{:= \alpha_{t(i)}} + \veps_i
\end{align*}

Since each $ Y_{t}$ is drawn from a standard normal distribution, we can recover $A^2$ from the top-$r$ singular values of the statistic $\frac{1}{N}\sum_{i=1}^N R_i^2 Y_{t(i)} Y_{t(i)}^\T$.

$(X_i,Y_{t(i)}, R_i)$ for $i=1,\ldots,N$.

* $UDU^\T \leftarrow \textrm{top}-r \textrm{ SVD of } \frac{1}{N}\sum_{i=1}^N R_i^2 X_i X_i^\T$. Set $\hat{A}^1 = U$.

* $UDU^\T \leftarrow \textrm{top}-r \textrm{ SVD of } \frac{1}{N}\sum_{i=1}^N R_i^2 Y_{t(i)} Y_{t(i)}^\T$. Set $\hat{A}^2 = U$.

Return $\hat{A^1}$ and $\hat{A^2}$.

Method-of-Moments Based Estimation

Suppose $X_i \simiid \Normal(0, \Identity_{d_1})$, $Y_t \simiid \Normal(0, I_{d_2})$, and $Z_t \simiid \Normal(0, I_{d_3})$. Then the factors $\hat{A^1}$ and $\hat{A^2}$ returned by Algorithm <ref> satisfies  the following guarantees

\begin{align*}
\sin \theta (\hat{A^1}, A^1) \le O\left( \sqrt{\frac{d_1 r}{TN}} \right), \ \textrm{and}\ \sin \theta (\hat{A^2}, A^2) \le O\left(\sqrt{\frac{d_2 r}{N}} \right)%, \ \textrm{and}\ \sin \theta (\hat{A^3}, A^3) \le O\left(r^{1.5}\sqrt{\frac{d_3}{N}} \right)
\end{align*}

with probability at least $1 - T \exp(-\Omega(\min\{d_1,d_2\}))$.

Once Algorithm <ref> estimates $\hat{A^1}$ and $\hat{A^2}$, we  again estimate $A^{3\T} Z_0$ in the meta-test phase. We can show a meta-test theorem similar to <Ref>, and the details are provided in the appendix.

§ EXPERIMENTS

We first evaluate our tensor-based representation learning through a simulation setup. For this experiment, we generated data from a low-rank tensor of order-$3$. We chose a tensor of dimension $100 \times 50 \times 50$ and of CP-rank $10$. We generated a training dataset of $N=1000$ points and estimated the factors $A^1$ and $A^2$ using both the tensor regression (Algorithm <ref>) and the method of moments (Algorithm <ref>). For the meta-test phase, we selected a new test task with observed feature $Y_0$ of dimension $50$ and unobserved feature $Z_0$ of dimension $50$. As described in <Ref>, we estimate $\widehat{A^{3\T} Z_0}$ by substituting the estimated factors from the meta-training step.

We plot the meta-test error for various values of $N_2$, the number of samples available from the new task. As we increase $N_2$, test error for predicting outcome on a new test instance $X_0$ decreases significantly, as shown in <Ref>. We compare our method with the matrix-based representations for meta learning developed by <cit.>. They assume that the response from a task $t$ with unobserved feature $Z_t \in \R^r$ and $i$-th feature $X_i$ is given as

\begin{equation*}
R_i = X_i B Z_t + \veps_i
\end{equation*}

where matrix $B \in \R^{d\times r}$. Recall that, for our setting, each training instance is given as $(X_i, Y_{t(i)}, R_i)$. Since <cit.> assume that there is no available side-information for the tasks, the most natural comparison would be to ignore the observable task features $Y_t$ and consider each input as $(X_i, R_i)$. So we consider two natural dimensions of the matrix $B$. First, we estimate a matrix of dimension $d_1 d_2 \times d_3$ where $X_i \otimes Y_{t(i)}$ is the $i$-th feature. Second, we estimate a matrix of dimension $(d_1 + d_2) \times d_3$ where $[X_i Y_{t(i)}]$ is the $i$-th input feature.
We compare these two different types of matrix based methods with both tensor regression and method-of-moments based method. As <Ref> shows both tensor methods perform equally well, but they are significantly better than the matrix methods.

Synthetic Dataset

Schools Dataset

Lenk Dataset

Test-error vs the number of samples from a new task ($N_2$)

We now consider two real-world datasets. Both the datasets were used in the context of conditional meta-learning to show the benefits of task-specific side-information <cit.>.

Schools Dataset <cit.>.  This dataset consists of examination records from $T = 139$ schools (task). The number of samples per task ($n_t$) varied from $24$ to $251$. Each instance represents an individual student, and is represented by a feature of dimension $d_1 = 26$. The outcomes are their exam scores. As task specific feature of task $t$ we use $Y_t = \frac{1}{n_t} \sum_{i=1}^{n_t} \phi(x_i)$ where $\phi(x_i)$ is a vector of dimension $d_2=50$ constructed from a random Fourier feature map. This is built as follows. First sample $v$ from $\textrm{Unif}[0,2\pi]^{d_2}$. Then a matrix $U \in \R^{d_2 \times d_1}$ is sampled from $\Normal(0,\sigma^2 \Identity)$. Finally, we set

\phi(x_i) = \sqrt{\frac{2}{d_2}} \cos\left( Ux_i + v\right) \in \R^{d_2}.

Lenk Dataset <cit.>. This is a computer survey data where  $T=180$ people (tasks) rated the likelihood of purchasing one of $20$ different personal computers. So there are $20$ different samples from each task. The input has dimension $d_1 = 13$ and represents different computers’ characteristics, while the output is an integer rating from $0$ to $10$. As task specific feature of a task $t$ we use $Y_t = \frac{1}{20} \sum_{i=1}^{20} \phi(z_i)$ where $\phi(z_i) = \textrm{vec}(x_i(R_i,1)^\T)$; the sum is over all $z_i$-s belonging to the task $t$.

To construct the meta-training set, we sampled $50$ tasks uniformly at random and then sampled $n_t$ ($n_t = 20$ for Schools and $n_t =10$ for Lenk) responses from each task. Since we do not know the value of $r$, we also constructed a meta-evaluation set by selecting another set of $n_t$ samples from the selected tasks. The meta-evaluation set was used to select the best value of $r$ during meta-training phase. The meta-test set was constructed by selecting a fixed task and then gradually increasing the number of samples from that task. Figures <ref> and <ref> respectively compare our method with two different types of matrix based representation learning for different values of $N_2$. We found that the tensor regression method performs better than the method-of-moments based estimator and only results for Algorithm <ref> are shown. Our method performs significantly better than the matrix based methods for Lenk. Although our method performs slightly worse on Schools, the test error increases by at most $5\%$. Overall, the performance on the synthetic dataset and two real-world datasets demonstrate the benefits of using tensor based representations for meta-learning.

§ CONCLUSION AND OPEN QUESTIONS

In this work, we develop a tensor-based model of shared representation for learning from a diverse set of tasks. The main difference with previous models on shared representations for meta-learning is that our model incorporates the observable side information of the tasks. We designed two methods to estimate the underlying tensor and compared them in terms of recovery guarantees, required assumptions on the tensor, and mean squared error on a new task.

There are many interesting directions for future work. An interesting direction is to generalize our model and consider non-linear models of shared representations that incorporates the observable side-information of the tasks. Finally, we just leveraged the framework of order-$3$ tensor in this work, and it would be interesting to see if we can leverage higher order tensors for learning shared representations for meta-learning.

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

titleTensor decompositions for learning latent variable models
journaltitleJournal of Machine Learning Research

titleGuaranteed Non-Orthogonal Tensor Decomposition via
Alternating Rank-$1 $ Updates
journaltitlearXiv preprint arXiv:1402.5180

titleConvex multi-task feature learning
journaltitleMachine learning

titleHow Important is the Train-Validation Split in
journaltitlearXiv preprint arXiv:2010.05843

booktitleInternational Conference on Machine Learning
titleProvable guarantees for gradient-based meta-learning

titleA model of inductive bias learning
journaltitleJournal of artificial intelligence research

titleMeta-learning with negative learning rates
journaltitlearXiv preprint arXiv:2102.00940

booktitleInternational Conference on Machine Learning
titleLearning-to-learn stochastic gradient descent with biased

titleThe Advantage of Conditional Meta-Learning for Biased
Regularization and Fine Tuning
journaltitleAdvances in Neural Information Processing Systems

titleFew-shot learning via learning the representation, provably
journaltitlearXiv preprint arXiv:2002.09434

booktitleInternational Conference on Machine Learning
titleModel-agnostic meta-learning for fast adaptation of deep

booktitleInternational Conference on Machine Learning
titleOnline meta-learning

titleModeling and Optimization Trade-off in Meta-learning
journaltitleAdvances in Neural Information Processing Systems

booktitleNeural Information Processing Systems
titleAdaptive Gradient-Based Meta-Learning Methods

titleTensor decompositions and applications
journaltitleSIAM review

titleOn Sketching the q to p Norms
journaltitleApproximation, Randomization, and Combinatorial
Optimization. Algorithms and Techniques

Springer Science & Business Media

titleProbability in Banach Spaces: isoperimetry and processes

titleHierarchical Bayes conjoint analysis: Recovery of partworth
heterogeneity from reduced experimental designs
journaltitleMarketing Science

booktitleProceedings of the 57th Annual Meeting of the Association
for Computational Linguistics
titleMulti-Task Deep Neural Networks for Natural Language

titleThe benefit of multitask representation learning
journaltitleJournal of Machine Learning Research

JMLR. org

titleNew perspectives on k-support and cluster norms
journaltitleThe Journal of Machine Learning Research

booktitleConference on Robot Learning
titleDeep dynamics models for learning dexterous manipulation

titleEstimation of (near) low-rank matrices with noise and
high-dimensional scaling
journaltitleThe Annals of Statistics

booktitleConference on Learning Theory
titleExcess risk bounds for multitask learning with trace norm

booktitleInternational Conference on Machine Learning
titleA sample complexity separation between non-convex and convex

booktitleLearning to learn
titleLearning to learn: Introduction and overview

booktitleAdvances in neural information processing systems
titleStatistical performance of convex tensor decomposition

titleProvable meta-learning of linear representations
journaltitleInternational Conference on Machine Learning

titleOn the Theory of Transfer Learning: The Importance of Task
journaltitleAdvances in Neural Information Processing Systems

Now Publishers

titleAn Introduction to Matrix Concentration Inequalities
journaltitleFoundations and Trends in Machine Learning

booktitleProceedings of the 31st International Conference on
Neural Information Processing Systems
titleA meta-learning perspective on cold-start recommendations for

Cambridge university press

titleHigh-dimensional probability: An introduction with
applications in data science

booktitleAdvances in Neural Information Processing Systems
titleMultimodal Model-Agnostic Meta-Learning via Task-Aware

Cambridge University Press

titleHigh-dimensional statistics: A non-asymptotic viewpoint

titleStructured Prediction for Conditional Meta-Learning
journaltitleAdvances in Neural Information Processing Systems

booktitleProceedings of the 25th ACM SIGKDD International
Conference on Knowledge Discovery & Data Mining
titleMetapred: Meta-learning for clinical risk prediction with
limited patient electronic health records

Taylor & Francis

titleTensor regression with applications in neuroimaging data
journaltitleJournal of the American Statistical Association

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

titleTensor decompositions for learning latent variable models
journaltitleJournal of Machine Learning Research

titleGuaranteed Non-Orthogonal Tensor Decomposition via
Alternating Rank-$1 $ Updates
journaltitlearXiv preprint arXiv:1402.5180

titleConvex multi-task feature learning
journaltitleMachine learning

titleHow Important is the Train-Validation Split in
journaltitlearXiv preprint arXiv:2010.05843

booktitleInternational Conference on Machine Learning
titleProvable guarantees for gradient-based meta-learning

titleA model of inductive bias learning
journaltitleJournal of artificial intelligence research

titleMeta-learning with negative learning rates
journaltitlearXiv preprint arXiv:2102.00940

booktitleInternational Conference on Machine Learning
titleLearning-to-learn stochastic gradient descent with biased

titleThe Advantage of Conditional Meta-Learning for Biased
Regularization and Fine Tuning
journaltitleAdvances in Neural Information Processing Systems

titleFew-shot learning via learning the representation, provably
journaltitlearXiv preprint arXiv:2002.09434

booktitleInternational Conference on Machine Learning
titleModel-agnostic meta-learning for fast adaptation of deep

booktitleInternational Conference on Machine Learning
titleOnline meta-learning

titleModeling and Optimization Trade-off in Meta-learning
journaltitleAdvances in Neural Information Processing Systems

booktitleNeural Information Processing Systems
titleAdaptive Gradient-Based Meta-Learning Methods

titleTensor decompositions and applications
journaltitleSIAM review

titleOn Sketching the q to p Norms
journaltitleApproximation, Randomization, and Combinatorial
Optimization. Algorithms and Techniques

Springer Science & Business Media

titleProbability in Banach Spaces: isoperimetry and processes

titleHierarchical Bayes conjoint analysis: Recovery of partworth
heterogeneity from reduced experimental designs
journaltitleMarketing Science

booktitleProceedings of the 57th Annual Meeting of the Association
for Computational Linguistics
titleMulti-Task Deep Neural Networks for Natural Language

titleThe benefit of multitask representation learning
journaltitleJournal of Machine Learning Research

JMLR. org

titleNew perspectives on k-support and cluster norms
journaltitleThe Journal of Machine Learning Research

booktitleConference on Robot Learning
titleDeep dynamics models for learning dexterous manipulation

titleEstimation of (near) low-rank matrices with noise and
high-dimensional scaling
journaltitleThe Annals of Statistics

booktitleConference on Learning Theory
titleExcess risk bounds for multitask learning with trace norm

booktitleInternational Conference on Machine Learning
titleA sample complexity separation between non-convex and convex

booktitleLearning to learn
titleLearning to learn: Introduction and overview

booktitleAdvances in neural information processing systems
titleStatistical performance of convex tensor decomposition

titleProvable meta-learning of linear representations
journaltitleInternational Conference on Machine Learning

titleOn the Theory of Transfer Learning: The Importance of Task
journaltitleAdvances in Neural Information Processing Systems

Now Publishers

titleAn Introduction to Matrix Concentration Inequalities
journaltitleFoundations and Trends in Machine Learning

booktitleProceedings of the 31st International Conference on
Neural Information Processing Systems
titleA meta-learning perspective on cold-start recommendations for

Cambridge university press

titleHigh-dimensional probability: An introduction with
applications in data science

booktitleAdvances in Neural Information Processing Systems
titleMultimodal Model-Agnostic Meta-Learning via Task-Aware

Cambridge University Press

titleHigh-dimensional statistics: A non-asymptotic viewpoint

titleStructured Prediction for Conditional Meta-Learning
journaltitleAdvances in Neural Information Processing Systems

booktitleProceedings of the 25th ACM SIGKDD International
Conference on Knowledge Discovery & Data Mining
titleMetapred: Meta-learning for clinical risk prediction with
limited patient electronic health records

Taylor & Francis

titleTensor regression with applications in neuroimaging data
journaltitleJournal of the American Statistical Association


[a]Konstantin Stankevich

# Astrophysical neutrino oscillations accounting for neutrino charge radii

Konstantin Kouzakov  Fedor Lazarev  Vadim Shakhov  Alexander Studenikin

###### Abstract

We derive for the first time an effective neutrino evolution Hamiltonian
accounting for neutrino interactions with external magnetic field due to
neutrino charge radii and anapole moment. The results are interesting for
possible applications in astrophysics.

It is well known that neutrino electromagnetic interactions [1] are important
for neutrino evolution and oscillations in different astrophysical
environments (see, for example, [2, 3, 4]). We consider for the first time
neutrino flavour, spin and spin-flavour oscillations engendered by neutrino
interactions with an external magnetic field due to neutrino charge radii and
anapole moment. Note, that in this case only the toroidal and poloidal
magnetic fields matters. We perform similar calculations that were performed
for derivations of the neutrino effective evolution Hamiltonians in the
presence of magnetic fields and moving matter [5, 6].

The neutrino electromagnetic interactions is described by the effective
interaction Hamiltonian[1]

$H_{int}(x)=j_{\mu}^{(\nu)}(x)A^{\mu}(x)=\sum_{k,j}^{2}\bar{\nu}_{k}(x)\Lambda^{\mu}_{kj}\nu_{j}(x)A_{\mu}(x),$
(1)

where $\Lambda^{\mu}_{kj}$ is the neutrino electromagnetic vertex function.
Here below, we are interested only in the charge radii
$\dfrac{\braket{r^{2}}}{6}$ and anapole form factors $f^{A}$ in
$\Lambda^{\mu}_{kj}$, thus we use

$\Lambda^{\mu}_{fi}(q)=(q^{2}\gamma_{\mu}-q^{\mu}\gamma^{\nu}q^{\nu})\left[\dfrac{\braket{r^{2}}_{fi}}{6}+f^{A}_{fi}\gamma_{5}\right].$
(2)

Within calculations similar to those of [5, 6], we get the following effective
interaction Hamiltonian for the considered case:

$H_{\alpha\alpha^{\prime}}^{ss^{\prime}}=u_{s\alpha}^{\dagger}\left\\{\left[\text{curl}\boldsymbol{B}\right]_{||}\left(\dfrac{\braket{r^{2}}_{\alpha\alpha^{\prime}}}{6}+f^{A}_{\alpha\alpha^{\prime}}\sigma_{3}\right)+\left[\text{curl}\boldsymbol{B}\right]_{\perp}\left(\gamma_{\alpha\alpha^{\prime}}^{-1}f^{A}_{\alpha\alpha^{\prime}}\sigma_{1}-i\tilde{\gamma}_{\alpha\alpha^{\prime}}^{-1}\dfrac{\braket{r^{2}}_{\alpha\alpha^{\prime}}}{6}\sigma_{2}\right)\right\\}u_{s^{\prime}\alpha^{\prime}},$
(3)

where $\left[\text{curl}\boldsymbol{B}\right]_{||}$ is the component of the
curl of the magnetic field parallel to the neutrino propagation and
$\left[\text{curl}\boldsymbol{B}\right]_{\perp}$ is the perpendicular
component, $u_{s\alpha}$ is the neutrino spinor. The gamma factors are given
by
$\gamma_{\alpha}^{-1}=\frac{m_{\alpha}}{E_{\alpha}},\gamma_{\alpha\beta}^{-1}=\frac{1}{2}\left(\gamma_{\alpha}^{-1}+\gamma_{\beta}^{-1}\right),\tilde{\gamma}_{\alpha\beta}^{-1}=\frac{1}{2}\left(\gamma_{\alpha}^{-1}-\gamma_{\beta}^{-1}\right).$

One can see that $\left[\text{curl}\boldsymbol{B}\right]_{||}$ is responsible
for the flavour oscillations and
$\left[\text{curl}\boldsymbol{B}\right]_{\perp}$ is responsible for the spin
and spin-flavour oscillations. Note, the spin and spin-flavour oscillations
are suppressed by gamma factors. In the flavour basis
$\nu_{f}=(\nu^{L}_{e},\nu^{L}_{x},\nu^{R}_{e},\nu^{R}_{x})$ the evolution
Hamiltonian can be decomposed into two parts

$H^{f}=[\text{curl}\boldsymbol{B}]_{||}H^{f}_{1}+[\text{curl}\boldsymbol{B}]_{\perp}H^{f}_{2},$
(4)

where

$H^{f}_{1}=\begin{pmatrix}\frac{\langle
r^{2}\rangle_{ee}}{6}+f^{A}_{ee}&\frac{\langle
r^{2}\rangle_{ex}}{6}+f^{A}_{ex}&0&0\\\ \frac{\langle
r^{2}\rangle_{ex}}{6}+f^{A}_{ex}&\frac{\langle
r^{2}\rangle_{xx}}{6}+f^{A}_{xx}&0&0\\\ 0&0&\frac{\langle
r^{2}\rangle_{ee}}{6}-f^{A}_{ee}&\frac{\langle
r^{2}\rangle_{ex}}{6}-f^{A}_{ex}\\\ 0&0&\frac{\langle
r^{2}\rangle_{ex}}{6}-f^{A}_{ex}&\frac{\langle
r^{2}\rangle_{xx}}{6}-f^{A}_{xx}\end{pmatrix},$ (5)

$H^{f}_{2}=\begin{pmatrix}0&0&\left(\frac{\braket{r^{2}}}{\gamma}\right)_{ee}+\left(\frac{f^{A}}{\gamma}\right)_{ee}&\left(\frac{\braket{r^{2}}}{\gamma}\right)_{ex}+\left(\frac{f^{A}}{\gamma}\right)_{ex}\\\
0&0&\left(\frac{\braket{r^{2}}}{\gamma}\right)_{e\mu}+\left(\frac{f^{A}}{\gamma}\right)_{e\mu}&\left(\frac{\braket{r^{2}}}{\gamma}\right)_{xx}+\left(\frac{f^{A}}{\gamma}\right)_{xx}\\\
\left(\frac{f^{A}}{\gamma}\right)_{ee}-\left(\frac{\braket{r^{2}}}{\gamma}\right)_{ee}&\left(\frac{f^{A}}{\gamma}\right)_{ex}-\left(\frac{\braket{r^{2}}}{\gamma}\right)_{ex}&0&0\\\
\left(\frac{f^{A}}{\gamma}\right)_{ex}-\left(\frac{\braket{r^{2}}}{\gamma}\right)_{ex}&\left(\frac{f^{A}}{\gamma}\right)_{xx}-\left(\frac{\braket{r^{2}}}{\gamma}\right)_{xx}&0&0\end{pmatrix}.$
(6)

The form factors in the flavour basis are defined as

$\displaystyle\langle r^{2}\rangle_{ee}$ $\displaystyle=\langle
r^{2}\rangle_{11}\cos^{2}\theta+\langle
r^{2}\rangle_{22}\sin^{2}\theta+\langle r^{2}\rangle_{12}\sin 2\theta,\ \ \ \
\ f^{A}_{ee}=f^{A}_{11}\cos^{2}\theta+f^{A}_{22}\sin^{2}\theta+f^{A}_{12}\sin
2\theta,$ (7) $\displaystyle\langle r^{2}\rangle_{xx}$ $\displaystyle=\langle
r^{2}\rangle_{11}\sin^{2}\theta+\langle
r^{2}\rangle_{22}\cos^{2}\theta-\langle r^{2}\rangle_{12}\sin 2\theta,\ \ \ \
\ f^{A}_{xx}=f^{A}_{11}\sin^{2}\theta+f^{A}_{22}\cos^{2}\theta-f^{A}_{12}\sin
2\theta,$ $\displaystyle\langle r^{2}\rangle_{ex}$ $\displaystyle=\langle
r^{2}\rangle_{12}\cos 2\theta+\dfrac{1}{2}\left(\langle
r^{2}\rangle_{22}-\langle r^{2}\rangle_{11}\right)\sin 2\theta,\ \ \ \ \ \ \
f^{A}_{ex}=f^{A}_{12}\cos
2\theta+\dfrac{1}{2}\left(f^{A}_{22}-f^{A}_{11}\right)\sin 2\theta,$

and

$\displaystyle\left(\dfrac{f^{A}}{\gamma}\right)_{ee}$
$\displaystyle=\dfrac{f^{A}_{11}}{\gamma_{11}}\cos^{2}\theta+\dfrac{f^{A}_{22}}{\gamma_{22}}\sin^{2}\theta+\dfrac{f^{A}_{12}}{\gamma_{12}}\sin
2\theta,\ \ \ \ \ \ \
\left(\frac{\braket{r^{2}}}{\gamma}\right)_{ee}=\tilde{\gamma}_{12}^{-1}\frac{\braket{r^{2}}_{12}}{6}\sin
2\theta,$ (8) $\displaystyle\left(\dfrac{f^{A}}{\gamma}\right)_{xx}$
$\displaystyle=\dfrac{f^{A}_{11}}{\gamma_{11}}\sin^{2}\theta+\dfrac{f^{A}_{22}}{\gamma_{22}}\cos^{2}\theta-\dfrac{f^{A}_{12}}{\gamma_{12}}\sin
2\theta,\ \ \ \ \ \ \
\left(\frac{\braket{r^{2}}}{\gamma}\right)_{xx}=-\tilde{\gamma}_{12}^{-1}\frac{\braket{r^{2}}_{12}}{6}\sin
2\theta,$ $\displaystyle\left(\dfrac{f^{A}}{\gamma}\right)_{ex}$
$\displaystyle=\dfrac{f^{A}_{12}}{\gamma_{12}}\cos
2\theta+\dfrac{1}{2}\left(\dfrac{f^{A}_{22}}{\gamma_{22}}-\dfrac{f^{A}_{11}}{\gamma_{11}}\right)\sin
2\theta,\ \ \ \ \ \ \ \
\left(\frac{\braket{r^{2}}}{\gamma}\right)_{ex}=\tilde{\gamma}_{12}^{-1}\frac{\braket{r^{2}}_{12}}{6}\cos
2\theta.$

The obtained evolution Hamiltonian can be used for the analysis of the
flavour, spin and spin-flavour oscillations and corresponding resonances due
to the neutrino electromagnetic interactions with the external magnetic field
engendered by neutrino charge radii and anapole moment. This work was
supported by the Russian Foundation for Basic Research under Grant No.
20-52-53022-GFEN-a. The work of KS was also supported by the Russian
Foundation for Basic Research under Grant No. 20-32-90107.

## References

  * [1] C. Giunti and A. Studenikin, Rev. Mod. Phys. 87 (2015), 531.
  * [2] A. de Gouvea and S. Shalgar, JCAP 10 (2012), 027.
  * [3] A. de Gouvea and S. Shalgar, JCAP 04 (2013), 018.
  * [4] S. Abbar, Phys. Rev. D 101 (2020) no.10, 103032.
  * [5] R. Fabbricatore, A. Grigoriev and A. Studenikin, J. Phys. Conf. Ser. 718 (2016) no.6, 062058
  * [6] P. Pustoshny and A. Studenikin, Phys. Rev. D 98 (2018) no.11, 113009.