Thus, using Hoeffding's inequality (see Fact <ref>) we have for some absolute constant $c>0$, with probability at least $1-\delta$:

\[
|\Phit(\X; \thetabt^\pare{1})| \leq TR (S + P)\sqrt{2\log(1/\delta)}.
\]

§.§.§ Proof of P3: NTK separability

We prove property P3 in two steps each stated in a separate lemma below

\begin{align}\label{eq:W not normalized}
\Wstar&=\mub_+\mub_+^{\top} + \mub_{-}\mub_{-}^{\top} + \sum_{\ell\in[M]}\nub_\ell(\mub_++\mub_{-})^{\top}, \\
\Ub_\star &= \ones_T\ub_\star^{\top}.
\end{align}

Given the initialization $\thetab^\pare{1}$ in (<ref>) and $\thetab_\star := (\overline{\Ub}_\star, \, \sign{\alpha}\overline{\W}_\star)$ we have for any $(\X,y)$ sampled from <ref> and satisfying Assumption <ref>:

\[
%\E_{\alpha \sim \Unif{\pm1}}y\inpb{\nabla_{\W}\Phi(\X; \thetab^\pare{1})}{\frac{\alpha\Wstar}{\norm{\alpha\Wstar}_F}} \geq \gamma,
\E_{\thetab^\pare{1}}\,y\inp{\nabla_{\thetab}\Phi(\X; \thetab^\pare{1})}{\thetab_\star} \geq \gamma_\star,
%\E_{\alpha \sim \Unif{\pm1}}y\inp{\nabla_{\W}\Phi(\X; \thetab^\pare{1})}{\alpha\Wstar} \geq \gamma,
\]

where the expectation is taken over $\alpha \sim \Unif{\pm1}$ and

\begin{align*}
\gamma_\star &:= \frac{T(1-\zeta)\zeta}{4\sqrt{2(M+1)}}\bigg(\zeta S^4-7\zbar S^2-12\zbar^2 -16\frac{\zbar^3}{S^2}\bigg) -PT^{5/2}(S+Z)^3 +  \frac{S\,\sqrt{T}}{\sqrt{2}}\, \Big(\zeta- 2(1-\zeta) \frac{\zmu}{S^2}\Big), \\
& \text{where} \, \, \zbar = \zmu \maxi \znu, \, \, \text{and} \, \,  \overline{\Ub}_\star, \overline{\W}_\star \, \,  \text{denote normalized} \, \, \, \Ub_\star, \W_\star, \, \text{respectively}.
%\gamma &= \frac{T(1-\zeta)}{3}\bigg[\zeta S^6 -\zeta (5\zmu+(1+\zeta)\znu)S^4-2((4-\zeta^2)\zmu + 2\zeta(1-\zeta)\znu)\zmu S^2\\&\qquad-4(1-\zeta)(2-\zeta)(\zmu+\znu)\zmu^2\bigg]-\frac{\sqrt{2(M+1)}}{3}S^2PT^{5/2}(S+(1-\zeta)Z)(\zeta S^2 + (1-\zeta)(S+Z)^2).
%\gamma &= \frac{1}{3}\Bigg\{
%\frac{\zeta T}{2}  \bigg[ \zeta^2 \, S^6 - 2 \, \zeta \, (1-\zeta) \, \zmu \, S^4 - (1-\zeta) \, \zmu \, (2 \, (1-\zeta) \, \zmu^2 + 2 \, \zeta \, \zmu + \zeta \, \znu) \, S^2 - 4 \, (1-\zeta)^2 \, \zmu^2 \, (\zmu + \znu) \bigg] \nn \\ & + \frac{\zeta T}{2} \bigg[ -\zeta^3 \, S^6 - (\zeta^2 \, (1-\zeta) + 2 \, \zeta \, (1-\zeta) \, \zmu) \, S^4 - (1-\zeta) \, (2\zmu + \zeta \znu) \, (\zeta + 2 \,(1-\zeta) \, \zmu) \, S^2 \nn \\ & - 2 \, (1-\zeta)^2 \, (\zeta + 2(1-\zeta) \, \zmu) \, (\zmu + \znu) \, \zmu \bigg]  - PT\sqrt{\zeta S^2 + (1-\zeta)(S+Z)^2}\bigg[ \zeta \sqrt{\zeta S^8 + (1-\zeta)S^4\zmu^2} \nn \\
%&+ (1-\zeta) \sqrt{\zeta (S^4+S^2(\zmu+\znu))^2 + (1-\zeta)(2\zmu^2+2\zmu\znu+2S^2\zmu)^2}\bigg] -PT(\zeta S + (1-\zeta)(S+Z))\nn \\
%&\bigg[\zeta \Big(\zeta S^4 + (1-\zeta)S^2\zmu\Big) + (1-\zeta)\Big(\zeta (S^4+S^2(\zmu+\znu))
%   + (1-\zeta)(2\zmu^2+2\zmu\znu+2S^2\zmu)\Big)\bigg]\Bigg\}.
\end{align*}

\begin{align*}
\inp{\nabla_{\thetab}\Phi(\X; \thetab^\pare{1})}{\thetab_\star} = \sign{\alpha}\inp{\nabla_{\W}\Phi(\X; \thetab^\pare{1})}{\overline{\W}_\star} + \inp{\nabla_{\Ub}\Phi(\X; \thetab^\pare{1})}{\overline{\Ub}_\star} \, .
\end{align*}

Using $\thetab^\pare{1}=\concat(\Ub^\pare{1},0)$ and $\overline{\Ub}_\star = \frac{\Ub_\star}{\norm{\Ub_\star}_F} = \frac{1}{S\sqrt{2T}}\ones_T \ub_\star^{\top}$, we have:

\begin{align}\label{eq: margin-U}
y\,\inp{\nabla_{\Ub}\Phi(\X; \thetab^\pare{1})} {\overline{\Ub}_\star} &= y\,\inp{\sft{\X \W^{\pare{1}} \X^\top}\X}{\overline{\Ub}_\star} = \frac{y}{TS\sqrt{2T}}\inp{\ones_T\ones_T^{\top}\X}{\ones_T \ub_\star^{\top}} \nn \\
&=\frac{y}{S\sqrt{2T}}\ub_\star^{\top}\left(\sum_{t}\x_t\right) \geq \frac{S\,\sqrt{T}}{\sqrt{2}}\, \Big(\zeta- 2(1-\zeta) \frac{\zmu}{S^2}\Big) \, .
\end{align}

The gradient with respect to $\W$ evaluated at $\thetab^\pare{1}=\concat(\Ub^\pare{1},0)$ is

\begin{align}
\nabla_{\W}\Phi(\X; \thetab^\pare{1}) = \frac{\alpha\zeta}{2}\sum_{t \in [T]}\x_t\ub_\star^\top\X^{\top}\Rb_{t}\X + \alpha \sum_{t \in [T]}\x_t\pb^\top\X^{\top}\Rb_{t}\X\,,\nn
\end{align}

\[\Rb_t =  \Rb_0:= \frac{1}{T} \cdot \Iden_T - \frac{1}{T^2} \cdot \ones_T \ones_{T}^{\top}, \,\, \forall t \in [T].
\]

\begin{align} \label{eq: grad w ntk}
&y\inp{\nabla_{\W} \Phi\left(\X;\thetab^\pare{1} \right)}{ \sign{\alpha}\overline{\W}_\star} \nn \\
&\qquad \qquad = \frac{\abs{\alpha}}{\norm{\Wstar}_F}\bigg(\frac{\zeta}{2} \underbrace{y\sum_{t \in [T]}\ub_\star^{\top}\X^{\top}\Rb_0 \rb_t(\X;\Wstar)}_{\term{I}} +  \underbrace{y\inpb{\sum_{t \in [T]}\x_t\pb^\top\X^{\top}\Rb_0\X}{\Wstar}}_{\term{II}}\bigg),
%&= y\alpha\tr \left({\W^\star}^\top\nabla_{\W}\Phi(\X; \thetab^\pare{1}) \right) \\
%= y\alpha^2\sum_{t \in [T]}\tr \left(\x_t\ub_\star^{\top}\X^{\top}\Rb^{(t)}\X\W^{*\top}\right) \notag\\ &= y\alpha^2 \sum_{t \in [T]}\tr \left(\ub^{*\top}\X^{\top}\left(\frac{1}{T} \cdot \Iden - \frac{1}{T^2} \cdot \ones\ones^{\top} \right)\X\W^{*\top}\x_t\right) \notag \\
%&= \frac{\alpha^2\zeta}{2}\cdot \underbrace{y\sum_{t \in [T]}\ub_\star^{\top}\X^{\top}\Rb_0 \rb_t(\X;\W^*)}_{\term{I}} \nn + \alpha^2 \cdot\underbrace{y\pb^{\top}\X^{\top}\Rb_0 \sum_{t\in[T]}\rb_t(\X;\W^*)}_{\term{II}}\,,
\end{align}

where we set for convenience:

\[
\rb_t(\X;\Wstar) = \X{\Wstar}^\top\x_t\in\R^T\,.
\]

To compute $\rb_t(\X;\Wstar)$, we consider two cases where row corresponds to a signal relevant of noisy token. We denote the $t'\in[T]$ entry of $\rb_t$ as $[\rb_t]_{t'}\in\R.$

Case 1. Relevance scores of signal tokens:
Consider signal token $t\in\Rc$ so that $\x_t=\mub_y$. Using orthogonality in Assumption <ref> we can compute for all $t'\in[T]$

\begin{align} \label{eq:relevance W signal exact}   t \in \Rc: [\rb_t]_{t'} =
\begin{cases}
S^4 & \text{,} \, t' \in \Rc\,, \\
S^2(\mub_y^{\top}\z_{t'}) & \text{,} \, t' \in \Rc^c\,. \\
\end{cases}
\end{align}

Therefore, again using Assumption <ref>,

\begin{align} \label{eq:relevance W signal}
t \in \Rc: [\rb_t]_{t'}
\begin{cases}
=S^4 & \text{,} \, t' \in \Rc\,, \\
\leq S^2\zmu & \text{,} \, t' \in \Rc^c\,. \\
\end{cases}
\end{align}

Case 2. Relevance scores of noisy tokens: Similar to the calculations above, using Assumption <ref> for parameters $\W^\star$ as in (<ref>) it holds for noisy tokens $t\in\Rcc$ that

\begin{align}\label{eq:relevance W noise exact}
t \in \Rc^c: [\rb_t]_{t'} =
\begin{cases}
S^4 + S^2(\mub_y^{\top}\z_{t}) + S^2(\sum_\ell \nub_\ell^{\top}\z_t) & \text{,} \, t' \in \Rc \\
(\mub_{+}^{\top}\z_{t})(\mub_{+}^{\top}\z_{t'}) +(\mub_{-}^{\top}\z_{t})(\mub_{-}^{\top}\z_{t'}) & \text{,} \, t' \in \Rc^c \\ + \sum_\ell (\nub_\ell^{\top}\z_t)(\mub_{+}^{\top}\z_{t'})
+ \sum_\ell (\nub_\ell^{\top}\z_t)(\mub_{-}^{\top}\z_{t'}) + S^2(\mub_{+}+\mub_{-})^{\top}\z_{t'}\\
\end{cases}
\end{align}

Now, we can start to bound each of the two terms in (<ref>) separately below.

Bounding $\term{I}$:
Recall the data matrix complies with Assumption <ref>, hence.

\begin{align}
[\X \ub_\star]_{t'} =
\begin{cases}
y S^2 & \text{,} \, t' \in \Rc \\
\z_{t'}^{\top} (\mubp - \mubn) & \text{,} \, t' \in \Rc^c \\
\end{cases} \, .
\end{align}

Using this, we can compute:

\begin{align*}
\frac{y}{T} \sum_{t \in [T]}\ub_\star^{\top}\X^{\top}\rb_t(\X;\W^*) &= \frac{y}{T}\Biggl\{\sum_{t \in \Rc}\ub_\star^{\top}\X^{\top} \rb_t(\X;\Wstar) + \sum_{t \in \Rc^c}\ub_\star^{\top}\X^{\top}\rb_t(\X;\Wstar) \Biggr\} \\
&= \frac{y}{T}\Biggl\{\sum_{t \in \Rc} \left(\sum_{t' \in \Rc}[\X\ub_\star]_{t'} \, \rb_t(\X;\Wstar)_{t'} + \sum_{t' \in \Rc^c}[\X\ub_\star]_{t'} \, \rb_t(\X;\Wstar)_{t'}\right) \\
&\quad \;\; + \sum_{t \in \Rc^c}\left(\sum_{t' \in \Rc}[\X\ub_\star]_{t'} \, \rb_t(\X;\Wstar)_{t'} + \sum_{t' \in \Rc^c}[\X\ub_\star]_{t'} \, \rb_t(\X;\Wstar)_{t'}\right)\Biggr\} \\
&= \frac{y}{T}\Biggl\{\zeta T \left(\zeta T \, yS^2 \, S^4 + \sum_{t' \in \Rc^c}\z_{t'}^{\top}(\mubp - \mubn) \, S^2(\mub_y^{\top}\z_{t'})\right) \\
&\quad \;\;+ \sum_{t \in \Rc^c}\bigg[\zeta TyS^2 \cdot \left(S^4 + S^2(\mub_y^{\top}\z_{t}) + S^2(\sum_{j_t} \nub_{j_t}^{\top}\z_t)\right) \\
&\quad \;\;+ \sum_{t' \in \Rc^c} \z_{t'}^{\top}(\mubp - \mubn) \cdot \bigg( (\mubp^{\top}\z_{t}) (\mubp^{\top}\z_{t'}) +(\mubn^{\top}\z_{t})(\mubn^{\top}\z_{t'}) \\
&\quad \;\;+ \; \, \sum_{j_t}  (\nub_{j_t}^{\top}\z_t)(\mubp^{\top}\z_{t'}) + \sum_{j_t} (\nub_{j_t}^{\top}\z_t)(\mubn^{\top}\z_{t'}) + S^2(\mubp+\mubn)^{\top}\z_{t'}\bigg)\bigg]\Biggr\}.
\end{align*}

Further using the noise bounds in Assumption <ref>, and $\rb_t(\X; \Wstar)$ from (<ref>), (<ref>) we have:

\begin{align}\nn
&\frac{y}{T} \sum_{t \in [T]} \ub_\star^{\top} \X^{\top} \rb_t(\X;\Wstar) \\ \nn
&\geq \zeta T\, \bigg[ \zeta S^6 - 2 (1-\zeta) \zmu^2 S^2 \bigg] + (1-\zeta)T \, \bigg[\zeta S^6-\zeta (\zmu + \znu) S^4 - 2 \zmu (1-\zeta) (2\zmu^2 + 2\zmu \znu + 2 \zmu S^2) \bigg] \,
\\
\label{eq:gamma-term11}
&\geq {T}  \bigg[ \zeta \, S^6 - \, \zeta \, (1-\zeta) \, (\zmu+\znu) \, S^4 - 2(1-\zeta) \,   (\zeta + 2(1-\zeta)) \, \zmu^2\, S^2 - 4 \, (1-\zeta)^2 \, \zmu^2 \, (\zmu + \znu) \bigg] \, .
\end{align}

For the second part of $\term{I}$:

\begin{align*}
- &\frac{y}{T^2} \sum_{t \in [T]} \ub_\star^{\top} \X^{\top} \ones_T \ones_T^{\top} \rb_t(\X;\Wstar) = - \frac{y}{T^2} \sum_{t \in [T]} \ones_T^\top \X \ub_\star \, \ones_T^\top \rb_t(\X;\Wstar) \\ & \quad= - \frac{1}{T^2} \left\{ \sum_{t' \in \Rc} y^2 S^2 + \sum_{t' \in \Rc^c} y \z_{t'}^{\top} (\mubp - \mubn) \right\} \cdot \Bigg\{ \sum_{t \in \Rc} \left[ \sum_{t' \in \Rc} S^4 + \sum_{t' \in \Rc^c} S^2 \z_{t'}^\top \mub_y \right] \\ &\qquad \;\;\; + \sum_{t \in \Rc^c} \bigg[ \sum_{t' \in \Rc} \left(S^4 + S^2 \z_{t}^\top \mub_y + S^2 \sum_{j_t} \nub_{j_t}^\top \z_t \right) + \sum_{t' \in \Rc^c} \bigg( (\z_{t}^{\top}\mubp)(\z_{t'}^{\top}\mubp) + (\z_{t}^{\top}\mubn)\mubn) \\ &\qquad \;\;\;  + (\sum_{j_t}\nub_{j_t}^{\top}\z_t) (\z_{t'}^{\top} \mubp) + (\sum_{j_t} \nub_{j_t}^{\top}\z_t) (\z_{t'}^{\top} \mubn) + S^2 \z_{t'}^\top (\mubp+\mubn) \bigg) \bigg] \Bigg\} \, .
\end{align*}

Using Assumption <ref> to simplify the second term:

\begin{align*}
&- \frac{y}{T^2} \sum_{t \in [T]} \ub_\star^{\top} \X^{\top} \ones_T \ones_T^{\top} \rb_t(\X;\Wstar) \\ &\qquad \geq T \bigg[ -\zeta^3 S^6 - \zeta^2 (1-\zeta) \zmu S^4 - \zeta^2 (1-\zeta) S^6 - \zeta^2 (1-\zeta) (\zmu + \znu) S^4 - 2 \zeta (1-\zeta)^2 \zmu S^4 \\ & \qquad \qquad \, \, - 2 \zeta (1-\zeta)^2 \zmu (\zmu + \znu)S^2 - 2 \zeta^2 (1-\zeta) \zmu S^4 - 2 \zeta (1-\zeta)^2 \zmu^2 S^2 - 2 \zeta (1-\zeta)^2 \zmu S^4\\ & \qquad \qquad \, \,   - 2 \zeta (1-\zeta)^2 \zmu (\zmu + \znu) S^2 - 4 (1-\zeta)^3 \zmu^2 S^2 - 4 (1-\zeta)^3 \zmu^2 (\zmu + \znu)  \bigg] \, .
\end{align*}

\begin{align}\label{eq:gamma-term12}
-\frac{y}{T^2} \sum_{t \in [T]} \ub_\star^{\top} \X^{\top} \ones_T \ones_T^{\top} \rb_t(\X;\Wstar) &\geq T\bigg[-\zeta^2S^6 -\zeta(1-\zeta)(4\zmu+\zeta\znu)S^4 - 2(1-\zeta)^2((2+\zeta)\zmu\nn \\ &\qquad+2\zeta\znu)\zmu S^2 -4(1-\zeta)^3\zmu^2(\zmu+\znu)\bigg] \, .
\end{align}

Bounding $\term{II}$:

\begin{align}\label{eq:gamma-p}
y\inp{\sum_{t \in [T]}\x_t\pb^\top\X^{\top}\Rb_0\X}{\Wstar} &\geq -\sum_{t \in [T]}\norm{\x_t}\norm{\pb}\norm{\X^{\top}\Rb_0\X}_F\norm{\Wstar}_F \nn \\
&\geq-P(\zeta TS+(1-\zeta)T(S+Z))\norm{\Rb_0}_F\norm{\X}_F^2S^2\sqrt{2(M+1)}\nn \\
&\geq-\sqrt{2(M+1)}S^2PT^{5/2}(\zeta S+(1-\zeta)(S+Z))(\zeta S^2 + (1-\zeta)(S+Z)^2) \, .
\end{align}

Combining (<ref>), (<ref>), (<ref>) in (<ref>) we get:

\begin{align} \label{eq:gamma-attn}
&y\inp{\nabla_{\W}\Phi(\X; \thetab^\pare{1})}{\Wstar} \nn \\
&\geq \alpha\Bigg\{
\frac{T\zeta}{2} \bigg[ \zeta \, S^6 - \, \zeta \, (1-\zeta) \, (\zmu+\znu) \, S^4 - 2(1-\zeta) \,   (\zeta + 2(1-\zeta)) \, \zmu^2\, S^2 - 4 \, (1-\zeta)^2 \, \zmu^2 \, (\zmu + \znu) \bigg] \nn \\ &\qquad - \frac{T\zeta}{2}\bigg[\zeta^2S^6 +\zeta(1-\zeta)(4\zmu+\zeta\znu)S^4 + 2(1-\zeta)^2((2+\zeta)\zmu+2\zeta\znu)\zmu S^2 +4(1-\zeta)^3\zmu^2(\zmu+\znu)\bigg] \nn \\ &\qquad-\sqrt{2(M+1)}S^2PT^{5/2}(\zeta S+(1-\zeta)(S+Z))(\zeta S^2 + (1-\zeta)(S+Z)^2) \Bigg\} \nn \\
&= \alpha \Bigg\{\frac{T(1-\zeta)\zeta}{2}\bigg[\zeta S^6 -\zeta (5\zmu+(1+\zeta)\znu)S^4-2((4-\zeta^2)\zmu + 2\zeta(1-\zeta)\znu)\zmu S^2 \nn \\&\qquad-4(1-\zeta)(2-\zeta)(\zmu+\znu)\zmu^2\bigg]-\sqrt{2(M+1)}S^2PT^{5/2}(S+(1-\zeta)Z)(\zeta S^2 + (1-\zeta)(S+Z)^2)\Bigg\} \nn \\
&\geq \alpha \Bigg\{\frac{T(1-\zeta)\zeta}{2}\bigg[\zeta S^6-(5\zmu+2\znu)S^4-4(2\zmu+\znu)\zmu S^2-8(\zmu+\znu)\zmu^2\bigg] \nn \\ &\qquad -\sqrt{2(M+1)}S^2PT^{5/2}(S+Z)(S^2+(S+Z)^2)\Bigg\} \nn \\
&\geq \alpha \bigg\{\frac{T(1-\zeta)\zeta}{2}\bigg(\zeta S^6-7\zbar S^4-12\zbar^2 S^2-16\zbar^3\bigg) -2\sqrt{2(M+1)}PT^{5/2}S^2(S+Z)^3\bigg\},
\end{align}

where $\zbar = \zmu \, \maxi \, \znu$. Using this, we get

\begin{align}
\mu:&= \E_{\alpha \sim \Unif{\pm1}}y\inpb{\nabla_{\W}\Phi(\X; \thetab^\pare{1})}{\frac{\sign{\alpha}\Wstar}{\norm{\Wstar}_F}} \nn \\
&\geq \E_{\alpha \sim \Unif{\pm1}}\frac{\abs{\alpha}}{\norm{\Wstar}} \bigg\{\frac{T(1-\zeta)\zeta}{2}\bigg(\zeta S^6-7\zbar S^4-12\zbar^2 S^2-16\zbar^3\bigg) -2\sqrt{2(M+1)}PT^{5/2}S^2(S+Z)^3\bigg\} \nn \\
&\geq \frac{T(1-\zeta)\zeta}{4\sqrt{2(M+1)}}\bigg(\zeta S^4-7\zbar S^2-12\zbar^2 -16\frac{\zbar^3}{S^2}\bigg) -PT^{5/2}(S+Z)^3  \, . \label{eq: margin-W}
\end{align}

Combining (<ref>) and (<ref>) concludes the proof.

Assume initialization $\thetabt^\pare{1}=\concat\big(\thetab_1^\pare{1},\ldots,\thetab_1^\pare{H}\big)$ as in (<ref>) and $\alpha_h\sim\Unif{\pm1}$ for all $h\in[H]$. Recall $\gammastar$ and $\thetab_\star(\cdot)$ from Lemma <ref> above. Set
\[\thetabt_\star=\frac{1}{\sqrt{H}}\concat\left(\thetab_\star\big(\thetab_1^\pare{1}\big),\ldots,\thetab_\star\big(\thetab_H^\pare{1}\big)\right).\]
Let any $(\X,y)$ from <ref>. Then, with probability at least $1-\delta$ over the randomness of $\alpha_h, h\in[H]$ it holds

\[
y\left\langle\nabla \Phit\left( \X; \thetabt^\pare{1} \right), \thetabt_\star\right\rangle \geq \gammastar - 2\left(2R^3T(S+P)+\sqrt{T}R\right)\sqrt{\frac{2\log(1/\delta)}{H}} \, .
\]

We start by expanding the empirical margin:

\begin{align}\label{eq:expand empirical margin}
y\left\langle \nabla \Phit\left( \X; \thetabt^\pare{1} \right), \thetabt_\star\right\rangle
&=  \frac{1}{H} \sum_{h \in [H]} y \inp{\nabla_{\thetab} \Phi\left(\X; \thetab_{h}^\pare{1} \right)}{\thetab_\star(\thetab_h^\pare{1})} =:
\frac{1}{H}\sum_{h \in [H]} X_h \,.
\end{align}

Note that each summand $X_h$ defined above depends only on $\alpha_h, h\in[H]$. Thus,  $\{X_h\}_{h\in[H]}$ are random variables because $\{\alpha_h\}_{h\in[H]}$ are random variables. Moreover, $X_h$ is almost-surely bounded satisfying

\begin{align}
|X_h|\leq \|\nabla_{\thetab} \Phi\left(\X; \thetab_{h}^\pare{1} \right)\| \|\thetab_\star(\thetab_h^\pare{1})\| \leq \sqrt{2T} \, R \, \left(2 \, R^2 \, \norm{\thetab_h^\pare{1}}_F + 1 \right) \leq 2 \, (2R^3T(S+P)+\sqrt{T}R) \, .
\end{align}

where the second inequality follows by Proposition <ref> and by the assumption $\|\thetab_\star(\thetab_h^\pare{1})\|_2=\sqrt{2}$ for all $h\in[H]$, and the last inequality follows because $\|\thetab_{h}^\pare{1}\|\leq \frac{\zeta}{2}\sqrt{T}\|\ub_\star\|_2+\sqrt{T}\|\pb\|_2\leq \sqrt{T}(S+P)$.

Finally, note that $\E[X_h]\geq \gammastar$ from Lemma <ref>.
Given these the desired claim follows by applying Hoeffding's inequality (see Fact <ref>) to (<ref>).

Define $\zbar := \zmu \maxi \znu$ and

\begin{align}
\gammastar := \frac{T(1-\zeta)\zeta}{4\sqrt{2(M+1)}}\big(\zeta S^4-7\zbar S^2-12\zbar^2 -16\frac{\zbar^3}{S^2}\big) -PT^{5/2}(S+Z)^3 + \frac{S\,\sqrt{T}}{\sqrt{2}}\, \Big(\zeta- 2(1-\zeta) \frac{\zmu}{S^2}\Big)\,.
\end{align}

\[
\sqrt{H} \geq 4 \cdot \frac{2R^3T(S+P)+\sqrt{T}R}{\gammastar}\cdot\sqrt{2\log(n/\delta)}\,.
\]

Then, with probability $1-\delta\in(0,1)$,
P3 holds with $\gamma=\gammastar/2$.

The proof is straightforward by using union bound and plugging the condition of $H$ in the result of Lemma <ref>.

§.§ Proof of Lemma <ref>

The proof of the lemma follows directly by      combining the two lemmas below and using $\zeta\leq 1$.

Fix any $h\in[H]$. Suppose zero initialization $\thetab_{h}^\pare{0}=0$ and consider first gradient step as in (<ref>).
It then holds that $\W_h^\pare{1} = 0$ and $\Ub_h^\pare{1}\in\R^{T\times d}$ has identical rows all equal to

\begin{align}\label{eq:U1 formula}
\frac{\zeta\alpha_h}{2}\ub_\star+\frac{\zeta\alpha_h}{2}\underbrace{\Big(\frac{1}{n}\sum_{i\in[n_1]}y_i\mub_{y_i} - \ub_\star\Big)}_{\pb_1} + \frac{\alpha_h(1-\zeta)}{2}\underbrace{\frac{1}{n}\sum_{i\in[n_1]}y_i\frac{1}{(1-\zeta)T}\sum_{t\in\Rcc_i}(\nub_{j_t}+\z_{i,t})}_{\pb_2} \, .
\end{align}

where recall that $\ub_\star=\mub_+-\mub_-$.

We start by computing

\begin{align*}
&= \sum_{i\in[n_1]}\nabla_{\thetab_h}\ell(y_i\Phit(\X;\thetab_{h}^\pare{0}))
= \sum_{i\in[n_1]}y_i\ellp(y_i\Phit(\X;\thetab_{h}^\pare{0}))\nabla_{\thetab_h}\Phit(\X;\thetab_{h}^\pare{0})
\\
&=  \ellp(0)\,\sum_{i\in[n_1]}y_i\nabla_{\thetab_h}\Phit(\X;\thetab_{h}^\pare{0})
= \frac{\ellp(0)}{\sqrt{H}}\,\sum_{i\in[n_1]}y_i\nabla_{\thetab_h}\Phi_h(\X;\thetab_{h}^\pare{0}) \\
&= \frac{1}{2\sqrt{H}}\,\sum_{i\in[n_1]}y_i\nabla_{\thetab_h}\Phi_h(\X;\thetab_{h}^\pare{0})
\end{align*}

where we used that $\Phi_h(\X;\thetab_{h}^\pare{0})=\Phit_h(\X;\thetab_{h}^\pare{0})=0$ because $\Ub_h=\mathbf{0}$ and also $\ellp(0)=1/2$ for the logistic loss.  Now, recall that $\Phi_h(\X;\theta_h)=\inp{\Ub_h}{\attn_h(\X;\W_h)}$. Hence, $\nabla_{\W_h}\Phi_h(\X;\thetab_{h}^\pare{0})=\mathbf{0}$, which gives us $\W_h^\pare{1}=\W_h^\pare{0}$. Also,

\[
\nabla_{\Ub_h}\Phi_h(\X;\thetab_{h}^\pare{0}) = \attn(\X;\W_{h}^\pare{0}) = \sft{\mathbf{0}}\X = \frac{1}{T}\ones_T \ones_T^\top \X \,.
\]

Hence, the $\tau$-th column of $\Ub_h^\pare{1}$ becomes for all $\tau\in[T]$:

\begin{align*}
[\Ub_h^\pare{1}]_{:,\tau} &= \frac{1}{2nT}{\alpha_h}\sum_{i\in[n_1]}y_i\sum_{t\in[T]}\x_{i,t}
\\
&= \frac{\zeta}{2n}{\alpha_h} \sum_{i\in[n_1]}y_i \mub_{y_i} + \frac{1}{2nT}{\alpha_h} \sum_{i\in[n_1]} y_i \sum_{t\in\Rcc_i} (\nub_{j_t} + \z_{i,t}) \, .
% \\
% &=\frac{\zeta\alpha_h}{2}\ub_\star+\frac{\zeta\alpha_h}{2}\underbrace{\Big(\frac{1}{n}\sum_{i\in[n]}y_i\mub_{y_i} - \ub_\star\Big)}_{\pb_1} + \frac{\alpha_h(1-\zeta)}{2}\underbrace{\frac{1}{n}\sum_{i\in[n]}y_i\frac{1}{(1-\zeta)T}\sum_{t\in\Rcc_i}(\nub_{j_t}+\z_{i,t})}_{\pb_2}
\end{align*}

The claim of the lemma follows by rearranging the above.

Suppose labels are and equal probable, i.e. $y_i\sim\Rad{\pm1}$. Then for the two terms $\pb_1, \pb_2$ in (<ref>) it holds with probability at least $1-2\delta$ and absolute constant $C>0$ over the randomness of labels that

\begin{align}\label{eq:p1 bound}
\|\pb_1\|\leq CS\Big(\sqrt{\frac{d}{n_1}} + \sqrt{\frac{\log(1/\delta)}{n_1}}\Big)\,,
\end{align}

\begin{align}\label{eq:p2 bound}
\|\pb_2\|\leq C(S+Z)\Big(\sqrt{\frac{d}{n_1}} + \sqrt{\frac{\log(1/\delta)}{n_1}}\Big)\,.
\end{align}

For arbitrary $\|\vb\|=1$, let $X_v=\inp{\vb}{\frac{1}{n}\sum_{i\in[n_1]}y_i\mub_{y_i}}$.

\begin{align*}
\tsub{X_v} = \frac{1}{n}\tsub{\sum_{i\in[n_1]}y_i\inp{\vb}{\mub_{y_i}}} \leq \frac{C}{n}\sqrt{\sum_{i\in[n_1]}\tsub{y_i\inp{\vb}{\mub_{y_i}}}^2} \leq \frac{C\,S}{\sqrt{n_1}} \, .
\end{align*}

where in the second inequality we used approximate rotation invariance of subgaussians and in the last step we used that for all $i\in[n_1]$, $|y_i\inp{\vb}{\mub_{y_i}}|\leq S$, thus they are  subgaussians with parameter $CS$. Further note that Note that $\E[X_v]=\inp{\vb}{\ub_\star}$. Thus, by centering property of subgaussians $\inp{\vb}{\pb_1}$ is also subgaussian with same constant $CS/\sqrt{n_1}$. Since this holds for all $\vb$ on the sphere, we conclude that $\pb_1$ is $CS/\sqrt{n_1}$-subgaussian.
From this, we can directly apply Fact <ref> for concentration of Euclidean norm of random vectors to arrive at (<ref>).

We can follow exactly same steps to prove (<ref>) for $\pb_2$. The only difference is noting that for all $i\in[n_1]$ and unit norm $\vb$:

\abs{y_i\frac{1}{(1-\zeta)T}\sum_{t\in\R_i^c}\inp{\nub_{j_t}+\z_{i,t}}{\vb}} \leq S+Z\,.

§ OPTIMAL MODEL

We first restate the optimal parameters $\thetab_\text{opt} = (\Ub_\text{opt}, \W_\text{opt})$:

\begin{align}
\Ub_\text{opt} &:= \frac{1}{S\sqrt{T}}\,\ones_d(\mubp-\mubn)^\top\,,\label{eq:Ustar}
\\
\W_\text{opt} &:= \frac{1}{S^2\sqrt{2(M+1)}}\Big(\mub_+\mub_+^{\top} + \mub_{-}\mub_{-}^{\top} + \sum_{\ell\in[M]}\nub_\ell(\mub_++\mub_{-})^{\top}\Big)\,,\label{eq:Wstar}
\end{align}

normalized so that $\|\thetab_\text{opt}\|_F = \sqrt{2}$.

The following lemma about saturation in softmax scores is used to prove Proposition <ref>.

Let relevance-scores vector $\rb=[r_1,\ldots,r_T]\in\R^T$ be such that for some $L\in[T]$ and $A,B\in\R$:

\[
r_1\geq r_2 \geq \ldots\geq r_L \geq A
\qquad\text{and}\qquad
B\geq \max\{ r_i\,|\, i=L+1,\ldots,T \}.
\]

Further assume $A>2B$. Fix any $\eps>0$ and

\[
\Gamma\geq \frac{2}{A}\log\Big(\frac{T/L-1}{\eps}\Big)\,.
\]

Then, for the attention weights $\ab=[a_1,\ldots,a_T]:=\sft{\Gamma\rb}\in\R^T$
it holds that

\begin{align}\label{eq:attn eps sandwich}
0\leq 1-\sum_{i\in[L]} a_i  = \sum_{i=L+1}^{T} a_i\leq \eps\,.
\end{align}

For convenience denote $D:=\sum_{j\in[T]}e^{\Gamma r_j}$. Note that  $D \geq L e^{\Gamma A}.$

Consider any $i>L$. Then,

\begin{align}\nn
a_i = \frac{e^{\Gamma r_i}}{D} \leq \frac{e^{\Gamma B}}{D}
\leq \frac{e^{\Gamma B}}{Le^{\Gamma A}} = \frac{1}{Le^{\Gamma (A-B)}}
\leq \frac{1}{Le^{\Gamma A/2}}\,.
\end{align}

Suppose $\Gamma\geq \frac{2}{A}\log\big(\frac{C}{\eps}\big)$, which ensures that

\[
e^{\Gamma A/2} \geq {C}/{\eps}
\]

Setting $C=\frac{T-L}{L}=\frac{T}{L}-1$, and combining the above two displays yields the desired:

\begin{align*}
% a_i &\geq \frac{\left(1+\eps\right)^{-1}}{L},\qquad i\in[L]
% \\
a_i &\leq \frac{\eps}{T-L},\qquad i>L\,.
\end{align*}

Thus, $\sum_{i>L}a_i \leq \eps $. The proof is complete by recalling
that $\sum_{i\in[T]}a_i=1$, hence $\sum_{i\in[L]}a_i\geq 1-\eps$.

§.§ Proof of Proposition <ref>

First, we  compute the attention matrix when the parameter $\W$ is set to the value below:

\begin{align}\label{eq:W not normalized}
\W=\mub_+\mub_+^{\top} + \mub_{-}\mub_{-}^{\top} + \sum_{\ell\in[M]}\nub_\ell(\mub_++\mub_{-})^{\top}\,.
\end{align}

For convenience, use the notation

$\rb_{t}^\top := \x_t^{\top}\W\X^{\top} \in \R^T$ for the $t$-th row of matrix used to find attention scores.  Similar to the proof of Lemma <ref> we consider two cases where row corresponds to a signal relevant of noisy token. We denote the $t'\in[T]$ entry of $\rb_t$ as $[\rb_t]_{t'}\in\R.$

Case 1. Relevance scores of signal tokens:
Consider signal token $t\in\Rc$ so that $\x_t=\mub_y$. Then for weights $\W$ in (<ref>), and using orthogonality in Assumption <ref> we can compute for all $t'\in[T]$

\begin{align*}
t \in \Rc: [\rb_t]_{t'} =
\begin{cases}
S^4 & \text{,} \, t' \in \Rc\,, \\
S^2(\mub_y^{\top}\z_{t'}) & \text{,} \, t' \in \Rc^c\,. \\
\end{cases}
\end{align*}

Therefore, again using Assumption <ref>,

\begin{align} \label{eq:relevance W signal}
t \in \Rc: [\rb_t]_{t'}
\begin{cases}
=S^4 & \text{,} \, t' \in \Rc\,, \\
\leq S^2\zmu & \text{,} \, t' \in \Rc^c\,. \\
\end{cases}
\end{align}

Case 2. Relevance scores of noisy tokens: Similar to the calculations above, using Assumption <ref> for parameters $\W$ as in (<ref>) it holds for noisy tokens $t\in\Rcc$ that

\begin{align*}
t \in \Rc^c: [\rb_t]_{t'} =
\begin{cases}
S^4 + S^2(\mub_y^{\top}\z_{t}) + S^2(\sum_\ell \nub_\ell^{\top}\z_t) & \text{,} \, t' \in \Rc \\
(\mub_{+}^{\top}\z_{t})(\mub_{+}^{\top}\z_{t'}) +(\mub_{-}^{\top}\z_{t})(\mub_{-}^{\top}\z_{t'}) & \text{,} \, t' \in \Rc^c \\ + \sum_\ell (\nub_\ell^{\top}\z_t)(\mub_{+}^{\top}\z_{t'})
+ \sum_\ell (\nub_\ell^{\top}\z_t)(\mub_{-}^{\top}\z_{t'}) + S^2(\mub_{+}+\mub_{-})^{\top}\z_{t'}\\
\end{cases}
\end{align*}

Therefore, using the noise bound assumptions, we have

\begin{align} \label{eq:relevance W noise}
t \in \Rc^c: [\rb_t]_{t'}
\begin{cases}
\geq S^4 - S^2(\zmu+\znu) & \text{,} \, t' \in \Rc \\
\leq 2\zmu^2+2\zmu\znu+2S^2\zmu
& \text{,} \, t' \in \Rc^c \,.
\end{cases}
\end{align}

Combining the above two cases, specifically Equations (<ref>) and (<ref>), we find that for all $t\in[T]$ the relevance-score vectors $\rb_t$ are such that

\begin{align}
t \in [T]: [\rb_t]_{t'}
\begin{cases}
\geq S^4 - S^2(\zmu+\znu):=A & \text{,} \, t' \in \Rc \\
\leq 2(\zmu^2+\zmu\znu+S^2\zmu):=B
& \text{,} \, t' \in \Rc^c \,,
\end{cases}
\end{align}

where we defined the parameters $A$ and $B$ for convenience. Note from assumption that

\[\zmu=\znu\leq \frac{S^2}{8}
%\leq  \big(\frac{\sqrt{17}-3}{8}\big) S^2
\implies A \geq \frac{3}{4} S^4 > \frac{1.25}{4} S^4 \geq 2B\,.\]

Thus, the conditions of Lemma <ref> hold for $L=|\Rc|=\zeta T$.
Applying the lemma we can immediately conclude that for

\[
\Gamma_* \geq \frac{8}{3S^4}\log\left(\frac{\zeta^{-1}-1}{\eps}\right) \geq \frac{2}{A}\log\left(\frac{\zeta^{-1}-1}{\eps}\right) \, .
\]

it holds:

\begin{align}\label{eq:attn bound for W}
\forall t\in[T]\,:\, 0\leq 1-\sum_{t'\in\Rc} \left[\sft{\Gamma_*\rb_t}\right]_{t'} =\sum_{t'\in\Rcc} \left[\sft{\Gamma_*\rb_t}\right]_{t'} \leq \eps \, .
\end{align}

Now, recall that
\[
\Wstar=\widetilde\Gamma\W\qquad\text{for}\quad \widetilde\Gamma=\Gamma/(S^2\sqrt{2(M+1)}) \, .
\]

Thus, it holds for all $t\in[T]$ that
\sft{\x_t^\top\Wstar\X^T} = \sft{\widetilde\Gamma\rb_t}\,.
Combining this with (<ref>) and the proposition's assumption on $\Gamma$ (satisfying $\widetilde{\Gamma}\geq \Gamma_*$), we have found that

\begin{align}\label{eq:attn bound for Wstar}
\forall t\in[T]\,:\, 0\leq 1-\sum_{t'\in\Rc} \left[\sft{\x_t^\top\Wstar\X^T}\right]_{t'} =\sum_{t'\in\Rcc} \left[\sft{\x_t^\top\Wstar\X^T}\right]_{t'} \leq \eps\,.
\end{align}

In the rest of the proof, we use (<ref>) to lower-bound the margin:

\begin{align*}
y\Phi(\X;\thetabstar) &= y\inp{\Ubstar}{\attn(\X;\Wstar)} = \frac{y}{S\sqrt{2T}}\sum_{t\in[T]}\sum_{t'\in[T]}\left[\phi(\x_t^\top\Wstar\X^T)\right]_{t'}(\mubp-\mubn)^\top\x_{t'} =:\frac{y}{S\sqrt{2T}}\sum_{t\in[T]} \psi_t \, ,
% \\
% &=y\sum_{t\in[T]}\sum_{t'\in\Rc}\left[\phi(\x_t^\top\Wstar\X^T)\right]_{t'}(\mubp-\mubn)^\top\x_t
\end{align*}

where we defined $\psi_t, t\in[T]$ for convenience.
For any $t\in[T]$, we have

\begin{align*}
\psi_t &= \sum_{t'\in\Rc}\left[\phi(\x_t^\top\Wstar\X^T)\right]_{t'}yS^2
\\&\geq yS^2\sum_{t'\in\Rc}\left[\phi(\x_t^\top\Wstar\X^T)\right]_{t'}
\\&\geq yS^2(1-\eps)
-2\eps\zmu \, .
\end{align*}

Putting the last two displays together and using $y^2=1$ completes the proof of the proposition.

§ EXPERIMENTS

In this section we provide some experiments discussing the role of number of heads $H$ in the training dynamics on synthetic data models.

Data Model <ref> We set the number of tokens $T=10$ and sparsity level $\zeta = 0.1$. We set $\{\mubp,\mubn,\nub_1, \nub_2, ..., \nub_M\}$ as the canonical basis vectors in $\R^d$, with $d=4, M=2$ and signal strength $S=2$. Noisy tokens $\z$ are sampled from a Gaussian $\Nc(0, \sigma^2\Iden_d)$, with $\sigma = 0.1$. We use $n=100$ training samples in each experiment and evaluate on a test set of size $300$ (total 5 trials). All models are initialized as $\thetabt^\pare{0} = \mathbf{0}$.

Figure <ref> shows the effect of increasing the number of heads when running GD with constant step-size $\eta=1.0$ and data generated from data model <ref>. Notice that rate of train loss decay reduces as we increase $H$, highlighting a potential downside of overparameterization. A similar observation has been recently noted by <cit.> when optimizing with GD to learn a single neuron with ReLU activation under Gaussian input. We also observe that at least for smaller $H$, GD indeed achieves margin $\gamma_\text{attn}$. It is worth noting that these observations do not contradict our theoretical findings in Theorems <ref> and <ref> which guarantee training convergence and generalization decay given sufficiently large number of heads $H$, without making an explicit connection between the rates of convergence as we increase $H$. We also test how these rates change if we scale step-size as $\eta = \order{\sqrt{H}}$ for GD (Figure <ref>, left) or optimize with Adam (Figure <ref>, right) using constant step-size $\eta = 0.06$. It is interesting to observe that in both these cases, the convergence speeds up for larger $H$, especially when optimizing with Adam, but somewhat strangely the margin attained by GD for larger $H$ continues to fall away from $\gamma_\text{attn}$. Further, note that our theory only covers step-size $\eta = \order{1}$ and the trends observed in Figure <ref> with $\eta = \order{\sqrt{H}}$ for GD fall outside this regime. In essence, we believe that it would be interesting future work to see how well these observations generalize to different datasets and develop theory that explains the relation of overparameterization to rates of convergence.

(33,68)GD with $\eta = \order{1}$

For data model <ref>. Effect of number of heads $H$ on convergence rates when trained with GD for constant step-size $\eta = \order{1}$. The average $\norm{\cdot}$ illustrates $1/H$ and $1/\sqrt{H}$ average for $\W$ and $\Ub$ across heads, respectively. Attn-score denotes the softmax scores for the relevant tokens averaged across all train samples and heads. The average $\norm{\W}$ indicates the saturation of softmax scores and consequently the token-selection (attn-score), and the average $\norm{\Ub}$ controls the loss behaviour. Results demonstrate that overparameterization slows down GD with constant step-size.

(29,68)GD with $\eta = \order{\sqrt{H}}$

(32,68)Adam with $\eta = \order{1}$

For data model <ref>. Effect of number of heads $H$ on convergence rates when (left) trained with GD when scaling step-size as $\eta = \order{\sqrt{H}}$; (right) trained with Adam with constant step-size  $\eta = \order{1}$.
Quantities plotted are same as in Figure <ref>. Results demonstrate that overparameterization speeds-up with train and test loss convergence in both the scenarios.

Planted data model Fix some $\W^* \in \R^{d \times d} , \, \Ub^* \in \R^{T \times d}$. Entries within $\X$ are sampled $X_{ij} \sim \Nc(0, 1), \,\, \forall i \in [T], \, j \in [d]$. Given such an $\X \in \R^{T \times d}$, generate the label $y$ using $\W^*$ as the attention matrix and $\Ub^*$ as the projection classifier:

\begin{align} \label{model planted}\tag{DM2}
y = \sign{\Phi(\X; \W^*, \Ub^*)} = \sign{\inp{\Ub^*}{\sft{\X\W^*\X^{\top}}\X}} \, .
\end{align}

Data generated using model <ref> is used to train a multi-head self-attention model as given in equation (<ref>). Such a teacher-student setting (train the student network to learn the ground truth parent) has been well explored in the past in the context of neural networks <cit.>. We set $d=5,\, T=10$. The train set contains $n=1000$ samples in each experiment and we evaluate on a test set of size 3000. Each result is averaged over 5 trials. For numerical ease, while generating (example, label) pairs we drop the samples for which $\abs{\text{output logit}} \leq \gamma_\text{attn}$, where we call $\gamma_\text{attn} > 0$ to be margin for the data model. We set $\gamma_\text{attn} = 0.2$ in all the experiments. All models are initialized as $\thetabt^\pare{0} = \mathbf{0}$. From Figure <ref> we observe that overparameterization somewhat improves convergence speeds for GD with step-size $\order{\sqrt{H}}$, similar to tokenized mixture model (Figure <ref>, left). Addition of momentum significantly helps speeding up convergence (see Figure <ref>, left) and so does optimizing with Adam (Figure <ref>, right). Interesting to note that all the models reach the expected margin $\gamma_\text{attn}$ which was not the case for large $H$ for the tokenized mixture model. Further, we can observe that initializing at $\thetabt^\pare{0} = \mathbf{0}$, all the optimizers find the planted model.

(33,68)GD with $\eta = \order{\sqrt{H}}$

For data model <ref>. Effect of number of heads $H$ on convergence rates when trained with GD when scaling step-size as $\eta = \order{\sqrt{H}}$. See Figure <ref> caption for to get more context on average $\norm{\cdot}$. Alignment of $\W$ with the planted-head $\Wt^\star$ at any iteration $k$ is given by $\frac{\inp{\Wt_k}{\Wt^\star}}{\norm{\Wt_k}\norm{\Wt^\star}}$, where $\Wt^\star:=\concat\big(\{\W^\star\}_{h\in[H]}\big)$ contains  $\W^\star$ repeated $H$ times. Alignment between $\Ubt$ and $\Ubt^\star$ is computed similarly.

(20,68)GD with $\eta = \order{\sqrt{H}}$ + momentum

(33,68)Adam with $\eta = \order{1}$

For data model <ref>. Effect of number of heads $H$ on convergence rates when trained with (left) GD + momentum where step-size scales as $\eta = \order{\sqrt{H}}$; (right) Adam with constant step-size $\eta = \order{1}$. Quantities plotted are same as in Figure <ref>. Results demonstrate that overparameterization speeds up convergence in both scenarios.


Kink in the dilaton-axion theory with a potential
possessing dilaton shift symmetry

O. V. Kechkin

Department of General Nuclear Physics, Faculty of Physics, M.V. Lomonosov
Moscow State University, 1(2), Leninskie gory, GSP-1, Moscow 119991, Russian
Federation
E-mail<EMAIL_ADDRESS>
The sigma model with dilaton and axion is generalized by including in it a
potential that is invariant under the global transformation of the dilaton
shift. In the (1 + 1)-dimensional case, a soliton is constructed, which turned
out to be an axion-type kink (anti-kink) with a nontrivial distribution of the
dilaton field. The solution is obtained for an arbitrary value of the dilaton-
axion coupling constant, and its mass is found.

Keywords: sigma model with potential, dynamic symmetry, soliton.

PACS: 05.45.Yv, 11.10.Lm, 11.30.-j.

## Introduction

Lagrangian systems with dilaton and axion appear when considering various
regimes of superstring theory [1] \- [2]; they have non-trivial extended
symmetry groups [3] \- [5]. The dilaton-axion sector of such systems is
described by the Lagrangian

${\cal
L}_{0}=\frac{1}{2}\left[\left(\partial\phi\right)^{2}+e^{2\alpha\phi}\left(\partial\kappa\right)^{2}\right],$
(1)

where $\phi=\phi(x)$ and $\kappa=\kappa(x)$ are the dilaton and axion fields,
and $\alpha$ is the dilaton-axion coupling constant (for example,
$\alpha=\frac{1}{2}$ for the case of low-energy dynamics in the heterotic
string theory). We consider a $(1+d)$-dimensional system on a flat background
with the Minkowski metric $g_{\mu\nu}=g^{\mu\nu}=diag(1;-1,…,-1)$ in Cartesian
orthogonal coordinates $x=\\{x^{\mu}\\}=\\{x^{0};\,x^{k}\\}$, where
$\mu=0,\dots,d$ and $k=1,\dots,d$. It was set
$\left(\partial\phi\right)^{2}=g^{\mu\nu}\partial_{\mu}\phi\partial_{\nu}\phi$
in (1) for compactness (and similarly for $\left(\partial\kappa\right)^{2}$).

Dilaton shift transformation

$\phi\rightarrow\tilde{\phi}=\phi+\epsilon,\qquad\kappa\rightarrow\tilde{\kappa}=\kappa
e^{-\alpha\epsilon}$ (2)

is the global isotopic symmetry of the theory with the Lagrangian (1) for an
arbitrary real value of the constant parameter $\epsilon$. We use this
symmetry, which plays an important role in string theory, as the basis for
further consideration.

##  1\. Potential with dilaton shift symmetry

It is easy to see that the combination of fields

$I=e^{\alpha\phi}\kappa$ (3)

is an invariant of transformation (2). Consider now a dilaton-axion system
with a Lagrangian

${\cal L}={\cal L}_{0}-{\cal V},$ (4)

where ${\cal L}_{0}$ is given by Eq. (1) and the potential ${\cal V}$ is
defined as a function of the indicated invariant, i.e. $V=V(I)$. It is clear
that the system (4) is symmetric with respect to the dilaton shift
transformation (2). In this paper, the potential is chosen in the Higgs form:

${\cal V}=\frac{\lambda}{4}\left(I^{2}-v^{2}\right)^{2},$ (5)

where $\lambda$ and $v$ are positive constants.

Performing a standard analysis of this theory, we come to the conclusion that
the set of vacuum states consists of pairs of constants
$\\{\phi_{v},\,\kappa_{v}\\}$, related to each other by the transformation (2)
and satisfying the restriction

$e^{2\alpha\phi_{v}}\kappa_{v}^{2}=v^{2}.$ (6)

Indeed, it is easy to verify that the Euler-Lagrange equations corresponding
to the system (1), (3), (4) and (5) are identically satisfied, and that the
Noetherian integral of the energy of the system reaches its minimal (trivial)
value on each of the representatives of this set.

Let us now calculate the mass spectrum of the system under consideration. To
do this, we set

$\phi=\phi_{v}+\xi,\qquad\kappa=\kappa_{v}+\eta;$ (7)

then for ${\cal L}_{2}$ – the part of the Lagrangian quadratic in perturbation
fields $\xi$ and $\eta$ over the chosen vacuum, we get:

${\cal
L}_{2}=\frac{1}{2}\left[\left(\partial\xi\right)^{2}+e^{2\alpha\phi_{v}}\left(\partial\eta\right)^{2}\right]-\lambda
v^{2}e^{2\alpha\phi_{v}}\left(\eta+\alpha\kappa_{v}\xi\right)^{2}.$ (8)

Diagonalization of Eq. (8) is achieved by substitution

$\xi=\frac{\theta+\alpha
v\chi}{\sqrt{1+\alpha^{2}v^{2}}},\qquad\eta=e^{-\alpha\phi_{v}}\frac{\chi-\alpha
v\theta}{\sqrt{1+\alpha^{2}v^{2}}};$ (9)

its result

${\cal
L}_{2}=\frac{1}{2}\left(\partial\theta\right)^{2}+\frac{1}{2}\left(\partial\chi\right)^{2}-\frac{m_{\chi}^{2}}{2}\chi^{2},$
(10)

where

$m_{\chi}=\sqrt{2\lambda\left(1+\alpha^{2}v^{2}\right)}\,v$ (11)

shows that the system has a massless Nambu-Goldstone mode $\theta$ and a Higgs
field $\chi$ with mass $m_{\chi}$ represented in Eq. (11).

##  2\. Stationary case. Integrals of motion

Consider now the system (1), (3), (4) and (5) in the stationary case. We are
interested in solitons, i.e. solutions that have a finite value of the energy
integral [7]. According to Derrick’s theorem [6], there is no prohibition on
their existence only for $d=1$, that is, for the original $(1+1)$-dimensional
theory, which will be assumed below. Thus, starting from here $\phi=\phi(x)$
and $\kappa=\kappa(x)$, where $x=x^{1}$.

The equations of motion then have the following form:

$\phi^{\prime\prime}-\alpha
e^{2\alpha\phi}\left[\left(\kappa^{\prime}\right)^{2}+\lambda\left(e^{2\alpha\phi}\kappa^{2}-v^{2}\right)\kappa^{2}\right]=0,$
(12) $\left(e^{2\alpha\phi}\kappa^{\prime}\right)^{\prime}-\lambda
e^{2\alpha\phi}\left(e^{2\alpha\phi}\kappa^{2}-v^{2}\right)\kappa=0,$ (13)

where the prime means taking the derivative with respect to $x$. It is easy to
check that the equations (12), (13) are the Euler-Lagrange equations for the
Noetherian energy integral of the system (1), (3), (4) and (5) for $d=1$ in
the stationary case. This integral is defined as

${\cal E}=\int_{-\infty}^{+\infty}dx\,T_{0}^{0},$ (14)

where

$T_{0}^{0}=\frac{1}{2}\left[\left(\phi^{\prime}\right)^{2}+e^{2\alpha\phi}\left(\kappa^{\prime}\right)^{2}\right]+\frac{\lambda}{4}\left(e^{2\alpha\phi}\kappa^{2}-v^{2}\right)^{2}$
(15)

is the corresponding component of the energy-momentum tensor.

The energy functional (14) admits a simple Noetherian analysis, which makes it
possible to integrate the system of nonlinear equations (12), (13). Namely, it
has explicit symmetry with respect to the shift

$x\rightarrow\tilde{x}=x+\varepsilon$

for $\varepsilon={\rm const}$ and invariant fields $\phi$ and $\kappa$. The
corresponding Noetherian charge, i.e. the integral of motion of the system
(12), (13), is the formal “energy” for the “mechanical system” with
generalized coordinates $\phi$ and $\kappa$ in the formal “time” $x$:

${\cal
F}=\frac{1}{2}\left[\left(\phi^{\prime}\right)^{2}+e^{2\alpha\phi}\left(\kappa^{\prime}\right)^{2}\right]-\frac{\lambda}{4}\left(e^{2\alpha\phi}\kappa^{2}-v^{2}\right)^{2}$
(16)

In this case, the integral of motion (16) is the result of the Legendre
transformation from the formal “Lagrangian” $T^{0}_{0}$ to the formal
“Hamiltonian” ${\cal F}$ according to the standard formula

${\cal F}=\frac{\partial
T^{0}_{0}}{\partial\phi^{\prime}}\phi^{\prime}+\frac{\partial
T^{0}_{0}}{\partial\kappa^{\prime}}\kappa^{\prime}-T^{0}_{0}.$ (17)

Then, the invariance of the functional (14), (15) with respect to the dilaton
shift transformation (2) gives one more integral of the system motion. Namely,
applying the Noetherian formalism, we arrive at the integral of motion

${\cal G}=\frac{\partial
T^{0}_{0}}{\partial\phi^{\prime}}\left.\frac{\partial\tilde{\phi}}{\partial\epsilon}\right|_{\epsilon=0}+\frac{\partial
T^{0}_{0}}{\partial\kappa^{\prime}}\left.\frac{\partial\tilde{\kappa}}{\partial\epsilon}\right|_{\epsilon=0}$
(18)

which after taking into account Eqs. (2) and (15) turns out to be equal to

${\cal G}=\phi^{\prime}-\alpha e^{2\alpha\phi}\kappa\kappa^{\prime}.$ (19)

By a direct check, one can make sure that ${\cal F}^{\prime}={\cal
G}^{\prime}=0$ when Eqs. (12), (13) are satisfied, as it should be for
integrals of motion.

##  3\. Soliton solution: axion kink (antikink)

Using the requirements for the convergence of the energy integral (14), (15),
we obtain the following conditions on the asymptotics of the fields:
$\lim_{x\rightarrow\pm\infty}\phi(x)=\phi_{\pm\infty},\qquad\lim_{x\rightarrow\pm\infty}\kappa(x)=\kappa_{\pm\infty},$
where the (constant) limit values of $\phi_{\pm\infty}$ and
$\kappa_{\pm\infty}$ are bounded by a relation of the form (6):

$e^{2\alpha\phi_{\pm\infty}}\kappa_{\pm\infty}^{2}=v^{2}.$ (20)

Substituting these limit values into (16) and (19) shows that the integrals of
motion corresponding to solitons are trivial:

${\cal F}={\cal G}=0.$ (21)

This fact allows us to integrate the system explicitly by solving the system
of first-order ordinary differential equations (16), (19), (21) with the
boundary condition, for example, for $x\rightarrow+\infty$:

$\lim_{x\rightarrow+\infty}\phi(x)=\phi_{0},\qquad\lim_{x\rightarrow+\infty}\kappa(x)=\kappa_{0}.$
(22)

Here, for compactness, we put $\phi_{+\infty}=\phi_{0}$,
$\kappa_{+\infty}=\kappa_{0}$; of course, the introduced quantities satisfy
the condition

$e^{2\alpha\phi_{0}}\kappa_{0}^{2}=v^{2}.$ (23)

Namely, the equation ${\cal G}=0$ is easily integrated and leads, when (22) is
taken into account, to the following result:

$e^{-2\alpha\phi}+\alpha^{2}\kappa^{2}={\rm
const}=e^{-2\alpha\phi_{0}}+\alpha^{2}\kappa_{0}^{2}.$ (24)

Then, using the relations ${\cal G}=0$, (23) and (24), we reduce the equation
${\cal F}=0$ to the following form:

$\left(\kappa^{\prime}\right)^{2}=\frac{\lambda
v^{2}\left(1+\alpha^{2}v^{2}\right)}{2\kappa^{2}_{0}}\left(\kappa^{2}_{0}-\kappa^{2}\right)^{2}.$
(25)

It can be seen that Eq. (25) is a relation that determines the axion kink
(antikink) type solution

$\kappa=\kappa_{0}\tanh\left(\frac{x-x_{0}}{r_{0}}\right)$ (26)

in the case with $\kappa^{2}_{0}-\kappa^{2}>0$ centered at $x_{0}$ and with
characteristic size (radius)

$r_{0}=\sqrt{\frac{2}{\lambda v^{2}\left(1+\alpha^{2}v^{2}\right)}}.$ (27)

Here, in the case of $\kappa_{0}>0$, a kink is obtained, but if
$\kappa_{0}<0$, it is an antikink. Obviously, in both cases
$\kappa_{\pm\infty}=\pm\kappa_{0}$, from which, taking into account (20), we
immediately obtain that $\phi_{\pm\infty}=\phi_{0}$. The calculation of the
dilaton field using (24) and (26) leads, when (23) is taken into account, to
the following result:

$\phi=\phi_{0}-\frac{1}{2\alpha}\log\left[1+\frac{\alpha^{2}v^{2}}{\cosh^{2}\left(\frac{x-x_{0}}{r_{0}}\right)}\right].$
(28)

Finally, the energy integral ${\cal E}={\cal E}(\alpha)$ for the constructed
solution—the mass of the kink (antikink)—is indeed finite. Namely,
substitution of the obtained solution into the defining relations (14) and
(15) gives:

${\cal E}(\alpha)=\Pi(\alpha){\cal E}(0),$ (29)

where

$\Pi(\alpha)=\frac{3}{4\alpha^{3}v^{3}}\left[\left(1+2\alpha^{2}v^{2}\right)\log\left(\sqrt{1+\alpha^{2}v^{2}}+\alpha
v\right)-\alpha v\sqrt{1+\alpha^{2}v^{2}}\right],$ (30)

and

${\cal E}(0)=\frac{4}{3}\sqrt{\frac{\lambda}{2}}\,v^{3}.$ (31)

Eqs. (29)–(31) show how the dilaton-axion interaction “forms” the mass-energy
of the found soliton. Indeed, ${\cal E}(0)$ is the mass of the kink (antikink)
in the absence of this interaction, that is, at $\alpha=0$. In this case, the
constructed solution describes a constant dilaton $\phi=\phi_{0}$, which does
not contribute to the soliton mass, and a standard kink/antikink for an axion
with $\kappa_{0}=\pm v$ (in this case, $\phi_{0}$ is a free parameter). In the
presence of an interaction, the mass is “renormalized”, turning into ${\cal
E}(\alpha)$, and the “renormalization coefficient” $\Pi(\alpha)$ does not
depend on the asymptotics and has an unexpectedly nontrivial form.

## Conclusion

The dilaton-axion system with a Higgs-type potential with dilaton shift
symmetry is interesting because of its closeness to applications of
superstring theory. It has soliton solutions, which are generalizations of the
standard kink and antikink in the standard scalar field theory with the Higgs
potential. These solutions can form the basis for the study of dilaton-axion
worlds on a brane [7], the decay of the false dilaton-axion vacuum [8], and a
number of other studies in the corresponding areas of classical and quantum
field theory.

The methods developed in this article can be applied to construct soliton
solutions in dilaton-Maxwell electrodynamics (see [9]) with an appropriately
selected potential. Interesting results related to the construction and study
of kink-type solitons in other theories can be found, for example, in [10],
[11].

## References

  * 1. Polchinski J. // String Theory. Cambridge University Press, 2005.
  * 2. Becker K., Becker M., Schwarz J. H. //String Theory and M-Theory. A Modern Introduction. Cambridge University Press, 2007.
  * 3. Maharana J., Schwarz J. H.  // Nucl. Phys. В. 1993. 390, P. 3.
  * 4. Herrera-Aguilar A., Kechkin O. V.  // Phys. Rev. D. 1999. 59, P. 124006.
  * 5. Kechkin O. V.  // Phys. Rev. D. 2002. 65, P. 066006.
  * 6. Derrick G. H. // J. Math. Phys. 1964\. 5, P. 1252.
  * 7. Rubakov V. // Classical Theory of Gauge Fields. Princeton University Press, 2002.
  * 8. Coleman S. // Phys. Rev. D. 1977. 15, P. 2929.
  * 9. Kechkin O. V., Mosharev P. A. // Int. J. Mod. Phys. A. 2016. 31, P. 1650169.
  * 10. Blinov P.A., Gani T.V., Malnev A.A., Gani V.A., Sherstyukov V.B. // Chaos, Solitons and Fractals. 2022. 165, P. 112805.
  * 11. Khare A., Duzgun A., Saxena A. // Int. J. Mod. Phys. B. 2021 355, P. 2150324 .


11institutetext: Paul Scherrer Institut, Villigen, Switzerland

# Medical Physics Commissioning

D. Meer

###### Abstract

Medical commissioning is an important step in bringing a particle gantry into
clinical operation for tumour treatments. This involves the parametrization
and characterization of all relevant systems including beam delivery, patient
table, imaging systems and connection to all required software components.
This article is limited to the necessary tasks for the beam delivery system of
a pencil beam scanning system. Usually, commissioning starts with the
characterization of the unscanned beam and calibration of the beam energy. The
next steps are the parametrization of the scanning system, the commissioning
of the beam position monitoring system and characterization of the spot size,
which all require precision better than 1 mm. The commissioning effort for
these tasks also depends on the gantry topology. Finally, calibration of the
dose measurement system ensures that any dose distribution can be delivered
with an absolute precision better than 1%.

###### keywords:

Proton therapy; pencil beam scanning; commissioning; calibration; medical
physics.

##  0.1 Introduction

Proton therapy systems are complex and commissioning is an important step in
bringing such a facility into clinical operation. Medical physics
commissioning is the intermediate step between technical commissioning and
medical acceptance. Technical commissioning includes the installation and
functional testing of system components, the beam line magnets or the rotating
mechanical gantry structure.

Within _medical physics commissioning_ , the main focus is on integral system
tests and parametrization of the machine characteristics. The correct
interaction between different system components is verified and important
machine parameters are measured. This task not only consists of checking the
beam scanning system and monitoring system, but also includes checking the
mechanical systems (gantry rotation), the patient positioning systems, the
different imaging systems and the transformation of coordinate system between
them, as well as the software connection to the treatment planning software
(TPS) and oncology information system. Finally, a properly configured safety
system ensures a safe operation. After this important step, the system must be
able to precisely deliver a predefined dose distribution with a given dose to
a target.

What follows is medical acceptance of the whole treatment unit when operated
by a responsible end-user. The system specifications are validated by numerous
tests, performed by medical physicists. Many of these tests have end-to-end
character and successful completion is a prerequisite before clinical
operation can be started.

There is no strict separation between technical and medical commissioning, and
responsibilities between producer and end-user might shift from one
installation to another. It also makes a difference whether an established
commercial product is installed in a clinical environment or a system is
developed for clinical application at a research institute. In this report we
look at the latter case, mainly reporting on the experience gained in the
commission of the second gantry at the Paul Scherrer Institut (PSI). Gantry 2
[1] is an active scanning system and we therefore only concentrate on pencil
beam scanning (PBS) rather than passive scattering. We mainly focus on the
dose delivery system, although medical commissioning comprises much more than
this, as mentioned before. In the first section, we discuss the commissioning
of the unscanned beam. The following section explains the lateral beam
spreading with the scanning system and the last section describes the
commissioning tasks for the dose-monitoring system.

##  0.2 Commissioning the unscanned beam

Almost all particle therapy systems use beam transfer lines to transport the
particles from the source (the accelerator) to the treatment rooms. Dipole and
quadrupole magnets along the beam line help to direct and focus the particle
beam to the isocentre. Additionally, a collimator, mechanical slits or
scattering foils help to create the required phase space at the treatment
location. Today, the vast majority of PBS systems do not use local energy
modulation right in front of the patient but rely on variable beam energy from
the accelerators or on upstream energy degrading systems.

###  0.2.1 Beam line tuning

To cover the full clinical range in depth, at least 50 different beam energies
must be transported from the accelerator to the isocentre, and a beam line
usually contains more than 30 devices. To optimize the beam line settings,
well proven tools like TRANSPORT[2] or Mad-X[3] are available.

From a particular solution for the magnet settings for a given energy, beam
line settings for other energies can be obtained by scaling the magnet
currents with the particle momentum. This approach also gives smooth behaviour
of the beam characteristics over the entire energy range. However, a linear
model, as used in TRANSPORT, cannot describe the entire beam line, which also
contains vacuum windows, collimators and drift distances in air or energy-
absorbing materials. In order to obtain a full beam line description and a
qualitative number for the beam line transmission, these models must be
combined with Mote Carlo methods. Several beam profile monitors along the beam
line can help to validate the predictions from such models.

To complete the beam tuning, some manual optimization with steering magnets is
necessary in order to centre the beam. It is of particular importance that the
beam is well centred on the mechanical rotation axis of the gantry. This
procedure can be simplified if a position monitor on the still horizontal but
rotating part of the gantry is available and data from this monitor can be
read while the beam is turned on and the gantry is rotating. If a circle is
visible on this position monitor, the beam centring can still be improved as
shown in fig:BeamCentering.pdf.

Figure 1: Beam centring at the beginning on the rotating part of the gantry.
The beam position is recorded continuously while the gantry rotates. The
initial beam positions (a) and beam positions after proper centring (b) are
shown. Maximum deviation is about 350$\mu$m after optimization.

Steering magnets on the gantry also help to centre the beam. If necessary,
gantry angle dependent solutions must be found.

###  0.2.2 Energy calibration

The range of the particles in matter (water) is one of the crucial parameters
for TPS and needs to be adjusted with an absolute precision better than
±0.5mm. Nevertheless, the calculations of beam tuning tools are based on beam
momentum and, due to several factors, the comparison of the converted particle
range with the measured range at the beam line often shows not negligible
differences. Therefore, energy-to-range calibration is necessary and is
performed with help of Bragg peak measurements. The range of a particle beam
is defined as the 80% point of the maximum ($R_{80}$) in the distal fall-off
of the Bragg peak. By this definition, the range is insensitive to different
momentum spread [4].

The preferred tools for measuring range in water are so-called range scanners.
They consist of two dose-measurement chambers (plain parallel ionization
chambers (ICs)): one is fixed at the entrance to the measurement phantom and
the second one moves at depth. The ratio between the two chambers is
calculated at different depths to obtain the depth–dose curve. Commercial
measurement tools such as the PEAKFINDER Water Column from PTW-Freiburg,
Germany, are available for such measurements. The typically requested
precision for range measurements is better than 0.2mm, which corresponds
approximately to a resolution of 0.1MeV. At PSI, we have developed a water
scanner which only consists of the movable chamber. We have access to the last
dose monitor on the gantry which is used for data normalization.

For determination of the $R_{80}$, it is sufficient to only measure the region
around the Bragg peak with high resolution. However, the integral depth–dose
profile is an important input for the TPS and must also be measured with high
precision. As shown in fig:DepthDoseCurve.pdf, an adaptive granularity can
help to accelerate measurements.

Figure 2: Measured integral depth–dose curves for 10 different beam energies.
The adaptive measurement granularity provides high resolution and efficient
data acquisition.

###  0.2.3 Beam line supervision

In systems with variable beam energy, the beam line settings are changed with
high frequency and online supervision helps to verify the correct settings. Of
particular importance are the dipoles of the energy-selection system,
collimators and slits. The correct settings for the dipoles can be verified by
an independent measurement of the magnetic field with Hall probes. The signal
of the Hall probe is proportional to the $B$ field

$V_{\mathrm{HallProbe}}\propto B\propto
pc=\sqrt{E_{\textrm{kin}}^{2}+2E_{\textrm{kin}}E_{0}}\leavevmode\nobreak\ ,$
(1)

and can be parametrized as a function of beam energy $E$,

$V_{\mathrm{HallProbe}}=a_{1}\sqrt{(E-a_{2})^{2}+2E_{0}(E-a_{2})}+a_{3}$ (2)

with three parameters $a_{1}$, $a_{2}$ and $a_{3}$. At PSI, the stability of
the electronics limits the energy resolution of the beam line supervision to
about 1MeV, corresponding to approximately 2mm range uncertainty in water.

##  0.3 Calibration of the scanning system

Once the unscanned beam has been tuned, commissioning of the scanning system
can be started. First, the impact of the gantry topology on the calibration of
the scanner magnets is discussed.

###  0.3.1 Up-stream and down-stream scanning

The location of the scanner magnets on a PBS gantry has significant
implications for both gantry design and commissioning. In a conventional
design, the scanner magnets are the last devices in the beam line as shown in
fig:UpStreamScanning.pdf(b). In this so-called _down-stream scanning_ , the
beam direction is divergent while the beam is scanned laterally. The scanner
magnets are placed at least 2m away from the isocentre since a small source-
to-axis distance (SAD) is unfavourable, leading to an increased skin dose, for
example. To first order there is a linear correlation between the spot
position at the isocentre and the scanner magnet current. Another advantage is
that the spot shape is unaffected for different scan positions. On the other
hand, the calibration of the scanning system is sensitive to the longitudinal
alignment of the device for the position measurement in the case of a finite
SAD.

Figure 3: Two possible gantry topologies with up-stream (a) and down-stream
(b) scanning

_Up-stream scanning_ is the other gantry topology in which the scanner magnets
are installed before the last bending dipole as shown in
fig:UpStreamScanning.pdf(a). In this configuration, a parallel beam (infinite
SAD) can be obtained by properly optimizing the beam optics and the last
bending dipole. A parallel scanned beam at the isocentre simplifies treatment
planning and measurements for the quality assurance program. However, there
are also additional technical challenges: beam-energy dependent
inhomogeneities and fringe fields of the dipole can affect spot shape at the
isocentre and beam focus needs to be dynamically adjusted when scanning the
beam in the divergent plane. Consequently, higher-order corrections for
position-to-current conversion need to be considered as well. PSI Gantry 2 [5]
is one of the few upstream scanning systems, and the calibration of that is
discussed in the following section.

###  0.3.2 Scanner magnet calibration

The important prerequisite for accurate calibration of the scanner magnets is
the availability of an appropriate device to measure the beam position at the
isocentre for the full scanning range and different gantry angles. We decided
to place a copy of the nozzle strip chamber on a rotatable support at the
isocentre (see fig:NozzleChamberCalibration.pdf (right)). As a consequence of
a parallel beam, the scan range at the isocentre is the same as in the nozzle,
and we could therefore use a spare position monitor from the nozzle. This has
the additional advantage that the standard read-out from the therapy control
system (TCS) can be used, and data from the isocentre measurement can be
integrated easily into the TCS logging file. The rotational support can turned
remotely and integrates a beam dump which greatly simplifies the measurement
at different gantry angles.

An alternative setup is to use a measurement device that can be attached to
the gantry nozzle. In that case, the deformation of the supporting structure
for different gantry angles needs to be considered.

The result of the sweeper calibration is two functions which provide the
scanner magnet current

$I_{\textrm{S}}^{x}=f^{x}(E,x,y)\ ,\ I_{\textrm{S}}^{y}=f^{y}(E,x,y)$ (3)

as a function of beam energy and lateral position at the isocentre for both
scanner magnets. These functions can be look-up tables with additional
interpolation or an analytic function. We decided to use polynomial functions
that would be less sensitive to measurement errors and to give a smooth
position dependency. The functions are calculated for every 10MeV and
interpolation is used for intermediate energies. Unfortunately, it was not
possible to find one single polynomial expression for the full scanning range
because each corner of the scan range showed substantial and specific
deviation from the linear position to current correlation. To better consider
the peculiarity of each corner, the global polynomial model was expanded with
a local solution for each quadrant:

$\begin{split}I_{\textrm{S}}^{x}(x,y)&=\overbrace{g_{1}x+g_{2}y+g_{3}x^{2}+g_{4}y^{2}+g_{5}y^{3}+g_{6}y^{4}}^{\text{Global
function}}+\\\
&\sum_{q=1}^{4}\underbrace{l_{1}^{q}xy+l_{2}^{q}x^{2}y+l_{3}^{q}xy^{2}+l_{4}^{q}x^{3}y+l_{5}^{q}x^{2}y^{2}+l_{6}^{q}xy^{3}+l_{7}^{q}x^{4}y+l_{8}^{q}x^{3}y^{2}+l_{9}^{q}x^{2}y^{3}+l_{10}^{q}xy^{4}}_{\text{Local
function for each quadrant}}\end{split}$ (4)

and an analogue expression for $I_{\textrm{S}}^{y}$.

The ansatz with only mixed terms in $x$ and $y$ for the local solution ensures
that the local contribution is vanishing when $x\rightarrow 0$ or
$y\rightarrow 0$, and this provides a smooth function of position. The
position to current calibration with this model requires 46 parameters for one
beam energy. This is still much less data than a look-up table to deliver beam
spots with a precision in the order of 100$\mu$m.

###  0.3.3 Position projection to the isocentre

During dose delivery with active scanning, the position of the pencil beam is
continuously monitored with a position monitor. Segmented ICs are the monitors
of choice for minimizing the additional amount of material in the beam path.
However, they measure the beam in the nozzle approximately 1m before the
isocentre and therefore the measured position needs to be projected to the
isocentre. This requires profound knowledge of the beam angle for the full
scanning range and all beam energies. An uncertainty in the beam angle of
approximately 1mrad already leads to a position measurement error of 1mm (see
fig:NozzleChamberCalibration.pdf (left-hand side)). The same measurement setup
as for the scanner magnet calibration can be used.

Figure 4: Precise data on the beam angle is required to predict the position
at the isocentre from the measured position in the nozzle chamber (left-hand
side). At PSI, a second strip chamber was positioned at the isocentre to
measure the correlation (right-hand side). The same setup is also used for the
calibration of the scanner magnets.

###  0.3.4 Spot size measurements

In addition to the depth–dose curve, the spot size is the other important
input parameter for the TPS. The TPS can work with a single spot size for a
given energy as long as spot size variations are negligible over the entire
scan range. To assess this assumption a two-dimensional spot shape measurement
needs to be performed at the isocentre over the full scan and proton energy
range. At PSI, we are using a system which is based on a CCD camera looking at
a scintillating foil, which provides high spatial resolution (see
fig:SpotSize.pdf (right-hand side)). Similar commercial equipment is
available, such as the Lynx detector from IBA Dosimetry in Schwarzenbruck
(Germany), for example. The spot size can be extracted from a delivered spot
pattern as shown in fig:SpotSize.pdf (left-hand side).

Figure 5: Typical spot pattern for spot size determination for a beam energy
of 150MeV (left-hand side). The device for this measurement is based on a
scintillating screen and a CCD camera (right-hand side).

Multiple spots in one single data acquisition increase the measurement
efficiency as long as the spots are well separated and the tails of the spots
are not overlapping. A two-dimensional Gaussian is fitted for each spot to
determine the spot size, and the required parameters for the TPS are
extracted, depending on the beam model in the TPS.

##  0.4 Dose monitor calibration

For a correct dose distribution, the exact dose per spot determination is as
essential as the spatial location of the spot, and the number of delivered
protons must be detected with high accuracy. ICs with a minimal amount of
material are again the preferred measurement devices because the dose monitor
stays permanently in the beam path. The collected charge from the IC

$Q_{\mathrm{IC}}=G_{\mathrm{IC}}\cdot S(E)\cdot\frac{1}{k_{Tp}}\cdot
q_{e}\cdot n_{p}$ (5)

is proportional to the number of protons $n_{p}$ and the stopping power $S(E)$
for a given energy. The factor $k_{Tp}$ corrects for temperature and pressure
other than standard conditions. The theoretical calculated value for the IC
gain $G_{\mathrm{IC}}$ based on chamber geometry has large uncertainties and
therefore experimental determination is required. At PSI, we use a Faraday cup
for this calibration. The charge measured with the Faraday cup is
$Q_{\mathrm{FC}}=q_{e}n_{p}$ and can be used for direct gain determination

$G_{\mathrm{IC}}=\frac{Q_{\mathrm{IC}}\cdot k_{Tp}}{S(E)\cdot
Q_{\mathrm{FC}}}\leavevmode\nobreak\ .$ (6)

There is also an alternative calibration procedure based on a measurement with
a small IC [6].

In general, dose distributions are calculated with pencil beam models which
predict the dose per incident proton [7]. The IC calibration is needed to
convert the number of protons from a pencil beam to the expected IC signal.
The validity of the IC calibration is verified by measuring the dose delivered
to a water phantom with a certified thimble IC following a code of practice
[8]. At PSI, we found an agreement of 2–3% between the dose calculated with
the beam model and absolute dosimetry measurements. An additional empirical
correction factor reduces this difference to less than 0.5%.

##  0.5 Summary and conclusion

It is far beyond the scope of this article to give a complete description of
medical commissioning. Important topics such as calibration of the imaging
systems or the patient table were not touched. The goal of this text is to
illustrate the methodology for some of the major tasks.

After successful medical commissioning, the system is validated by acceptance
tests with similar characteristics and is then ready for patient treatment.
After clinical operation has started, the performance of the system is
continuously monitored by quality assurance (QA) tests. These tests have
different repetition periods ranging from daily to yearly tests. The careful
execution of a QA program is necessary to guarantee clinical operation with
constant high quality over many years.

## Acknowledgements

The author would like to thank all members of the Center for Proton Therapy at
PSI who contributed to this work.

## References

  * [1] E. Pedroni _et al._ , _Z. Med. Phys._ 14 (2004) 25. http://dx.doi.org/10.1078/0939-3889-00194
  * [2] K.L. Brown, _et al._ , Transport, a Computer Program for Designing Charged Particle Beam Transport Systems, CERN 80-04 (1980). http://dx.doi.org/10.5170/CERN-1980-004
  * [3] W. Herr and F. Schmidt, in Proceedings of the CAS-CERN Accelerator School: Intermediate Course on Accelerator Physics, Zeuthen, Germany, 15–26 september 2003, CERN-2006-002 (CERN, Geneva, 2004), pp. 505-528, http://dx.doi.org/10.5170/CERN-2006-002.505
  * [4] W.C. Hsi _et al._ , _Med. Phys._ 36 (2009) 2297. http://dx.doi.org/10.1118/1.3132422
  * [5] E. Pedroni, _et al._ , _Eur. Phys. J. Plus_ 126 (2011) 66. http://dx.doi.org/10.1140/epjp/i2011-11066-0
  * [6] C. Gomà _et al._ , _Phys. Med. Bio._ 59 (2014) 4961. http://dx.doi.org/10.1088/0031-9155/59/17/4961
  * [7] E. Pedroni, _et al._ , _Phys. Med. Biol._ 50 (2004) 4961. http://dx.doi.org/10.1088/0031-9155/50/3/011
  * [8] P. Andreo _et al._ , Absorbed Dose Determination in External Beam Radiotherapy, International Atomic Energy Agency, Technical Reports Series No. 398 (2000)