Data-Driven Track Following Control for Dual Stage-Actuator Hard Disk Drives

First]Nikhil Potu Surya Prakash
First]Joohwan Seo
Second]Alexander Rose
First]Roberto Horowitz

[First]University of California, Berkeley, Berkeley, CA, 94720 USA

[Second]Leibniz University Hannover, Germany,
Hanover, Germany

e-mail: {nikhilps, joohwan_seo, rose1993<EMAIL_ADDRESS>
In this paper, we present a frequency domain data-driven feedback control design methodology for the design of tracking controllers for hard disk drives with two-stage actuator as a part of the open invited track 'Benchmark Problem on Control System Design of Hard Disk Drive with a
Dual-Stage Actuator' in the IFAC World Congress 2023 (Yokohoma, Japan). The benchmark models are

Compared to the traditional controller design, we improve robustness and avoid model mismatch by using multiple frequency response plant measurements directly instead of plant models.

Disturbance rejection and corresponding error minimization is posed as an $H_2$ norm minimization problem with $H_\infty$ and $H_2$ norm constraints.

$H_\infty$ norm constraints are used to shape the closed loop transfer functions and ensure closed loop stability and $H_2$ norm constraints are used to constrain and/or minimize the variance of relevant.

Data-driven control, robust control, hard disk drives, convex optimization.

§ INTRODUCTION

Currently, two main types of drives are used for data storage: Hard Disk Drives (HDD) and Solid State Drives (SSD).

While SSDs are advantageous for use in Personal Computers (PC) due to high read/write speed and size, HDDs are dominant for data storage devices in data centres due to their high robustness and comparatively low cost ([Chen et al. , 2022]).
Cost and performance of a HDD are strongly connected to the storage capacity, which highly depends on the spacing between successive tracks on which the data is stored.

This spacing is limited by the sensitivity of the controller to external disturbances, measurement noises and track runout.

The major sources of disturbance in HDDs are the fan-induced vibrations from the cooling systems and the rotational vibrations from other HDDs in a storage box at the data centers.

These disturbances have a wide bandwidth with high-frequency components up to 10 kHz.

Thereby, low-frequency disturbance compensation is done by a Voice Coil Motor (VCM) with higher strokes and high-frequency disturbance compensation is realized by a low stroke Piezo-electric actuators (PZT). Both these actuators together form the Dual Stage Actuation (DSA) system of an HDD.

PZTs can commonly reach one to three tracks.

Furthermore, due to manufacturing tolerances and due to the dependence on the temperature, the plants of each HDD are different.

Therefore, robust and precise controllers are needed to handle the disturbances for variations of plants.

Data-driven feedback controllers are capable of handling this complex task.

Data-driven feedback control design approaches from frequency response measurements [[Galdos et al., 2010, Karimi et al., 2017, Karimi et al., 2016]] for dual stage systems have been developed in [Bagherieh , 2017, Bagherieh et al , 2018] to suppress the disturbances.

Controllers for disturbance rejection and data-driven $H_\infty$ synthesis controllers for triple-stage actuator systems have also been developed in [Prakash et al., 2022].

Though these controllers are robust to the variations in the plant models, a common controller might not be the optimal controller for each individual HDD as there might be variations in the plants and the disturbances.

To address these issues, data driven feedforward controllers have been developed in [Shah et al , 2020, Shah et al , 2019, Shah et al , 2021] based on the frequency response measurement of disturbance processes on top of the robust feedforward controllers and another add-on adaptive feedforward controller has been developed in [Chen et al. , 2022] on top of
the robust feedback controller to account for plant variations.

In this paper, we present a data-driven mixed $H_2/H_\infty$ synthesis framework for designing robust feedback controllers for a DSA HDD with a VCM and a PZT actuator. The design is performed on the benchmark HDD actuator models from [Atsumi et al., 2020] as a art of the open invited track session 'Benchmark Problem on Control System Design of Hard Disk Drive with a
Dual-Stage Actuator' at IFAC World Congress 2023 to be held in Yokohoma, Japan.

For this purpose we use multiple plant measurements for a benchmark DSA HDD.

The disturbance rejection controller design will be posed as a convex optimization problem with the objective to minimize the average $H_2$ norm of the output due to various process noises subject to constraints on $H_\infty$ and $H_2$ norms of appropriate closed loop transfer functions.

The $H_\infty$ constraints will be used to shape the closed loop transfer functions using numerically shaped weights in the frequency domain and guarantee stability of the closed loop system ([Bagherieh , 2017]).

By the $H_2$ norm constraints the variances of various signals of interest are treated.

In section <ref>, we introduce the plant of the DSA HDD as block diagram, give the controller factorization with the frequency response of the VCM & PZT, define the closed-loop transfer functions and show the frequency responses of the disturbance.

Section <ref> describes our data-driven control design using $H_2$- and $H_\infty$-norm.

For both, we define the constraints.

Following from the $H_2$-objective, the minimum $H_2$-norm controller is defined.

In section <ref>, we show the simulation results of the controller for nine different plants.

§ PRELIMINARIES

In this paper, we use the discrete time domain (z-domain).

Note that the generalization to the case of continuous time (s-domain) is straightforward.

The DSA HDD plant model $G(z)$
is a Multiple Input Single Output (MISO) system with two inputs and determines the displacement of the head from the output of the two controllers.

In the following we use the shorthand notation with the dependence on $z$.

We use a set of frequency responses $G\left(e^{j\omega T_s}\right)$, with $\omega\!\in\!\Omega\!=\!(-\frac{\pi}{T_s},\frac{\pi}{T_s})$ and sampling time $T_s$ to characterize the plant.

§.§ Control structure of DSA HDDs

Control block diagram for a Dual Stage Actuator HDD (top) and control block diagram with lumped disturbance (bottom).

The block diagram of the DSA is shown in Fig. <ref>.

$r$ is the reference that needs to be tracked and

$r$ is set to zero for track following purposes.

The position error signal $e$ results from the reference, the sampled total head position $y_d$ and the repeatable runout $d_{RRO}$ (see Fig. <ref>), which is the track offset from the ideal track.

This error feeds into the discrete time controllers $C$ for the actuators of the two plants, the VCM plant and the PZT plant, of the MIMO system.

The path for PZT has subscript $dp$ and the path for VCM has subscript $dv$.

With the interpolation block $I_p$ the output signal of the controller is sampled up to a higher frequency for the multi-rate filters $F_{mp}$ and $F_{mv}$ for PZT and VCM respectively (see Fig.<ref>).

Repeatable runout used for the simulation.

Multi-rate filters for VCM (blue) and PZT (red) actuators

The zero order hold (ZOH) process $H_m$ converts the discrete signal into a continuous signal.

For both the paths, same $H_m$ is applied.

While the rotational vibrations $d_f$ caused by other HDDs in the storage box only affect the VCM path, the disturbances from the fans of the cooling system $d_p$ act on both.

For the design, we assume that $d_f$ and $d_p$ are generated by zero mean white noise of unit variance $\bar{d}_f$ and $\bar{d}_p$, and filtered by transfer function $D_f$ and $D_p$, respectively.

Both disturbances can be combined to $d \!=\! d_p \!+\! P_{cv}d_f$ using block diagram manipulations.

$y_{cp}$ denotes the displacement of the PZT actuator which is of special interest as it can only have a finite stroke to stay within the linear regime.

The total head position $y_c$ is sampled with $S$ to the discrete time signal $y_d$.

Frequency responses of the VCM plant (top) and PZT plant (bottom)

Frequency responses from input of $F_{mv}$ to output of $P_{cv}$ (top) and input of $F_{mp}$ to output of $P_{cp}$ (bottom)

The magnitude frequency responses of the benchmark model for the VCM and
PZT actuators are shown in Fig. <ref>. Some typical resonance frequency variations are also shown. The combined responses with the multi-rate filters are also presented in Fig. <ref>.

The variability in the resonance frequencies usually limit the attainable bandwidth of the servo system.

Therefore, piggy-back PZT actuators, which can only achieve a stroke of a few tracks but have higher resonance modes than the VCM, are used to increase the overall disturbance attenuation of the servo system.

In the model used for our case study, the PZT actuator has a resonance mode with a nominal resonance frequency of 40 kHz.

In the production of HDDs, manufacturing tolerances lead to slightly different plants of the DSA systems.

Therefore, the dynamics of the actuators and also the frequency responses differ between the drives, see Fig.<ref>.
Moreover, even for a specific actuator, the dynamics may vary due to environmental variations such as temperature.

In large data centers, millions of enterprise HDDs are stacked in the server boxes. If several HDDs are placed next to each other, vibrations due to neighboring drives or cooling fans can affect the hard disks differently.

It is common in industry to incorporate the effects of track run-out, windage, internal, and external disturbances in a disturbance filter $D(z)$.

A typical example of a frequency response of $D(z)$ is given in Fig. <ref>.

Since the influence of $D(z)$ on the position error $e$ is much larger than measurement noise $n$ and runout $d_{RRO}$, we focus on $D(z)$. However, we note that
$n$ and $d_{RRO}$ can be easily incorporated into our framework.

Typical frequency spectrum of the disturbances $d_f$ (top) and $d_p$ (bottom) used in Figs. <ref>. Boxes A and B show the resonant components

§.§ Controller Factorization

The vector controller transfer function can be factorized as $K \!=\! XY^{-1}$ following the convention from [Karimi et al., 2017] as

\begin{equation}
\begin{aligned}
X(z) &= \mathcal{X}_pz^p+\mathcal{X}_{p-1}z^{p-1}+\dots+\mathcal{X}_0, \nonumber \\
Y(z) &= z^p+\mathcal{Y}_{p-1}z^{p-1}+\dots+\mathcal{Y}_0,
\end{aligned}
\end{equation}

where $p$ is the controller order and the controller parameters are $\{\mathcal{Y}_{p-1},\dots, \mathcal{Y}_0 \} \!\in\! \mathcal{R}$ and $\{\mathcal{X}_p,\mathcal{X}_{p-1},\dots, \mathcal{X}_0 \} \!\in\! \mathcal{R}^{n\times 1}$. For the remainder of the paper, the convention $X_i = \mathcal{X}_p(i)z^p+\mathcal{X}_{p-1}(i)z^{p-1}+\dots+\mathcal{X}_0(i)$ will be used to denote the $i^{th}$ row of $X(z)$.

With the controller factorization presented above, various closed loop transfer functions can be calculated as follows

\begin{equation}\label{eq:SensitivityCF}
\begin{aligned}
&S_{d \rightarrow e}  &= \frac{Y}{Y+GX} , \;
&U_{d \rightarrow u_{VCM}} &= \frac{X_1}{Y+GX}, \\
&U_{d \rightarrow u_{PZT}} &= \frac{X_2}{Y+GX}, \;
&Y_{d \rightarrow y_{PZT}} &= \frac{G_{PZT}X_2}{Y+GX}
\end{aligned}
\end{equation}

where $G = [P_{cv} \; P_{cp} ]$ is the DSA plant.

§ DATA-DRIVEN CONTROL DESIGN

In this section, various objectives and constraints that will be used in the disturbance rejection controller design will be formulated in such a way that a convex optimization solver can obtain a solution. It's important to pose the problem as a convex optimization problem as some of the constraints involve matrix inequalities and any nonlinearities in the matrix inequalities cannot be handled by most of the existing optimization solvers. The conversion to a convex optimization problem is an attractive method

for robust controller design as $H_2$ constraints and objectives can also be handled by the framework. The formulation of the constraints and objectives will be demonstrated for one of the closed loop transfer functions, but the same procedure can be followed for any of the closed loop transfer functions.

The $H_\infty$ constraints are used to ensure stability and shape the closed loop transfer functions and $H_2$ constraints are used to constrain the variance of signals of interest. In the data-driven design, the constraints will only be enforced at the frequencies at which the responses are available. It is assumed that the data set is rich enough i.e., the frequencies at which the data is collected represent the key characteristics of the plants. Otherwise there is a possibility of obtaining controllers that might not stabilize the plant. Some such cases are

* when the controller order is comparable to the number of data points, the obtained solution might be an overfit to the data and the real system might not be stabilized by the controller,

* when the key characteristics like the resonant peaks are not captured by the data, again the obtained solution might not stabilize the actual plant.

§.§ Constraints and Objectives

§.§.§ $H_\infty$ Constraints:

$H_\infty$ constraints are typically used to shape various closed loop transfer functions with appropriate weighting transfer functions. In the data driven control framework, a feasible controller that satisfies the $H_\infty$ constraints is guaranteed to stabilize the closed loop system [[Bagherieh , 2017]]. A typical $H_\infty$ constraint on the weighted sensitivity of the DSA from the disturbance $d$ to the position error signal $e$ can be formulated as $||W(\omega)S_{d \rightarrow e}(e^{j\omega})||_\infty \!\leq\! \gamma$, where $W(\omega)$ is a numerically shaped weight for all the frequencies $\omega\!\in\!\Omega \!=\! (-\frac{\pi}{T_s},\frac{\pi}{T_s})$.

Using the definition for sensitivity in (<ref>) with the controller factorization, this constraint can be equivalently written as (Ch 5 of [Bagherieh , 2017])

\begin{equation}\label{eq:HinfConstraint1}
\gamma^{-1}|W(\omega)Y(e^{j\omega})| < Re[Y(e^{j\omega})+G(\omega)X(e^{j\omega})].
\end{equation}

Note that for a MISO system, this constraint is a convex constraint as it can be expressed as a Second Order Cone constraint. Any of the available Second Order Conic Program (SOCP) solvers can handle these constraints. For practical purposes, as strict inequality constraints cannot be handled by optimization solvers, the strict inequalities are replaced with slack inequalities with the help of a small tolerance.
The $H_\infty$ constraints that are used in the current design are as follows:

\begin{align}
&\mbox{Single Stage (VCM):}\nonumber \\
||W_{S^{VCM}_{d \rightarrow e}}S^{VCM,i}_{d \rightarrow e}||_\infty &< 1, \\
||W_{U^{VCM}_{d \rightarrow u_v}}U^{VCM,i}_{d \rightarrow u_v}||_\infty &< 1,
\end{aligned}\\[10pt]
&\mbox{Dual Stage (DSA):}\nonumber \\
||W_{S^{DSA}_{d \rightarrow e}}S^{DSA,i}_{d \rightarrow e}||_\infty &< 1, \\
||W_{U^{DSA}_{d \rightarrow u_v}}U^{DSA,i}_{d \rightarrow u_v}||_\infty &< 1.
\end{aligned}
\end{align}

Here the superscript VCM refers to the case when only the VCM is operating i.e., when $G = [P_{cv} \; 0]$ and the superscript DSA refers to the case when both the VCM and PZT are operating i.e., when $G = [P_{cv} \; P_{cp}]$. The superscript $i$ is used to represent the $i^{th}$ plant i.e., the plant's frequency response $G$ would be of the $i^{th}$ set of VCM and PZT plants.

The first set of constraints on just the single stage are imposed to ensure that the system remains stable even PZT actuator fails. The weights on the VCM's input for the both the stages are designed in such a way that the resonant modes of the VCM actuator are
not excited.
$H_\infty$ constraints on the PZT's inputs are not imposed as the multi-rate filters are already designed in a way that the PZT's resonant modes are not excited. But, if multi-rate filters are not used, $H_\infty$  constraints can also be introduced on the PZT's inputs. The inverse weights used for the design are shown with dashed blue lines in Figs. <ref> and <ref>.

§.§.§ $H_2$ Constraints:

$H_2$ constraints are used to constrain the variances of various signals. Parseval's theorem states that the time domain and frequency domain $L_2$ norms of a signal are equivalent. This means that the $H_2$ norm of a transfer function is equal to the $L_2$ norm of the transfer function's impulse response (Note that $H_2$ norm is used for a transfer function whereas $L_2$ norm is used for a signal).

The convolution of the impulse response with the input signal gives us the system's response to the input signal. For a normally distributed random variable, the confidence of finding the random variable within three standard deviations from the mean is $99.7\%$. Though hard limits on the magnitudes of the signals cannot be enforced in the frequency domain design unlike in state space techniques like Model Predictive Control (MPC), we can constrain the signal's variance to increase the confidence that the signal does not go beyond the limit $99.7 \%$ of the time. This can be done by restricting the $L_2$ norm of the zero mean signal to be always less than a third of the limit.

The constraints can be applied on various signals like the inputs to the actuators, the stroke limits of the actuators, the overall position error signal etc. In case of multiple plants, we can either constrain the $L_2$ norms of the signals for each of the plants or constrain the average $L_2$ norm  of the signals for all the plants. A PZT actuator's displacement needs to be restricted for it to behave linearly. This forms a classic example of an $H_2$ norm constraint. The constraint in this case can be formulated as constraining the average squared $H_2$ norm of the output of the PZT due to various process noises. This would constrain the average variance of the output of the PZT actuator. The maximum value of the variance can be set based on the confidence with which we want the output of the PZT to stay within the desired limits. Typically the stroke of a PZT is set to be around one to three tracks. For the current design, the average stroke is limited to be $\mu = 50 nm$.

This constraint can be formulated using Parseval's theorem as

\begin{equation}\label{eq:H2Constraints1}
\begin{aligned}
||y^i_{cp}||^2_2 =\, &||Y^i_{d \rightarrow y_{cp}}D_p||^2_2\\
+&||Y^i_{d \rightarrow y_{cp}}P^i_{dv}D_f||^2_2\\
\implies\! \frac{1}{l}\sum_{i=1}^{l}||y^i_{cp}||^2_2 =\, & \frac{1}{l}\sum_{i=1}^{l}(||Y^i_{d \rightarrow y_{cp}}D_p||^2_2\\
&+||Y^i_{d \rightarrow y_{cp}}P^i_{dv}D_f||^2_2)\leq \mu,
\end{aligned}
\end{equation}

where the subscript $i$ corresponds to the $i^{th}$ plant.
The $H_2$ norm for this PZT output transfer function is given by

\begin{equation}\label{eq:H2Constraints2}
\hspace{-5pt}||Y^i_{d \rightarrow y_{cp}}D_p||^2_2 \!=\! \frac{T_s}{2\pi} \!\int\limits_{\Omega} \!tr[(Y^i_{d \rightarrow y_{cp}}D_p)^*Y^i_{d \rightarrow y_{cp}}D_p]d\omega.
\end{equation}

where $*$ denotes the complex conjugate.

Using slack variables $\Gamma^{i}_{y_{cp}}$ and $\Lambda^{i}_{y_{cp}}$, the $H_2$ constraint in Eq. (<ref>) can be reformulated into two constraints as,

\begin{align}
\label{eq:H2Constraints3}
&\frac{1}{l}\sum_{i=1}^{l}\frac{T_s}{2\pi} \int\limits_{\Omega}\! tr[\Gamma^{i}_{y_{cp}}(\omega)+\Lambda^{i}_{y_{cp}}(\omega)]d\omega \leq \mu,\\
&(Y^i_{d \rightarrow y_{cp}} D_p)^*  Y^i_{d \rightarrow y_{cp}}D_p \preceq \Gamma^{i}_{y_{cp}}.  \\
%&D_p^*Y_{d \rightarrow %y_{cp,i}}^*Y_{d %\rightarrow y_{cp,i}}D_p %\preceq \Gamma^{y_{cp}}_i.
\end{aligned} \\
&(Y^i_{d \rightarrow y_{cp}}P^i_{dv} D_f)^*   Y^i_{d \rightarrow y_{cp}}P^i_{dv}D_p \preceq \Lambda^{i}_{y_{cp}}.  \\
%&D_p^*Y_{d \rightarrow %y_{cp,i}}^*Y_{d %\rightarrow y_{cp,i}}D_p %\preceq \Gamma^{y_{cp}}_i.
\end{aligned}
\end{align}

Since the responses are available only at a finite set of frequencies, the integral in Eq. (<ref>) will be evaluated approximately using a trapezoidal rule by replacing the function $\Gamma^{i}_{y_{cp}}(\omega)$ with a finite set of $\Gamma^{k,i}_{y_{cp}}$ for $k\!=\!1\mbox{ to } N$ where $N$ is the number of frequency responses available per plant. The matrix inequalities in Eq. (<ref>) & (<ref>) can be reformulated using controller factorization and the Schur complement lemma as

\begin{align}
\begin{bmatrix}
\Gamma^{k,i}_{y_{cp}} & X_1D_p \\
(X_1D_p)^* & (Y+G^iX)^*(Y+G^iX)
\end{bmatrix}(\omega_k) &\succeq 0.\\
\end{aligned}\\
\begin{bmatrix}
\Lambda^{k,i}_{y_{cp}} & X_1P^i_{dv}D_p \\
(X_1P^i_{dv}D_p)^* & (Y+G^iX)^*(Y+G^iX)
\end{bmatrix}(\omega_k) &\succeq 0.\\
\end{aligned}
\end{align}

Notice that this  is a non convex constraint, as ${P^i}^*P^i \!=\! (Y\!+\!G^iX)^*(Y\!+\!G^iX)$ is quadratic. However, this quadratic part can be linearized using a first order Taylor series expansion around a nominal controller $K_c \!=\! X_c/Y_c$, as shown in <cit.>.

\begin{equation}
(P^i)^*P \approx {P^i_c}^*P^i_c+(\Delta P^i)^*P^i_c+{P^i_c}^*(\Delta P^i),
\end{equation}

where $P^i_c = Y_c+G^iX_c$ and $\Delta P^i = P^i-P^i_c$

This leads to

\begin{align}
\label{eq:H2Constraints7a}
\begin{bmatrix}
\Gamma^{k,i}_{y_{cp}} & X_1D_p \\
(X_1D_p)^* & (P^i_c)^*P^i_c\!+\!(\Delta P^i)^*P^i_c\!+\!{P^i_c}^*(\Delta P^i)
\end{bmatrix}\!(\omega_k) \!\succeq\! 0.
\end{aligned}\\
\label{eq:H2Constraints7b}
\begin{bmatrix}
\Lambda^{k,i}_{y_{cp}} & X_1P^i_{dv}D_p \\
(X_1P^i_{dv}D_p)^* & (P^i_c)^*P^i_c\!+\!(\Delta P^i)^*P^i_c\!+\!{P^i_c}^*(\Delta P^i)
\end{bmatrix}\!(\omega_k) \!\succeq\! 0.
\end{aligned}
\end{align}

§.§.§ $H_2$ Objective:
$H_2$ objectives can be used in cases when the variances of signals need to be minimized. Hence $H_2$ objective becomes an ideal choice for disturbance rejection problems. The objective can be formulated to minimize the variance of the tracking error i.e., the position error signal $e$ due to various process noises. The position error signal due to the disturbance $\bar{d}_f$ and $\bar{d}_p$ is given by  $S_{d \rightarrow e} D\bar{d}$.

The optimization problem in it's raw form can be written as

\begin{equation}\label{eq:H2Obj1}
\begin{aligned}
& \underset{X,Y}{\text{min}}
& & \frac{1}{l}\sum_{i=1}^{l}(||S^{DSA,i}_{d \rightarrow e}D_p||^2_2+||S^{DSA,i}_{d \rightarrow e}P^i_{dv}D_f||^2_2) \\[5pt]
& \text{s.t.}
& & H_\infty \mbox{ and } H_2 \mbox{ constraints}.
\end{aligned}
\end{equation}

Using a similar approach shown for the $H_2$ constraints, we can formulate the optimization problem using slack variables $\Gamma^i_{S_{d \rightarrow e}}$ and $\Lambda^i_{S_{d \rightarrow e}}$ as follows

\begin{equation}\label{eq:H2Obj2}
\begin{aligned}
& \underset{X,Y}{\text{min}}
& &  \frac{1}{l}\sum_{i=1}^{l}\int_{\Omega} tr[\Gamma^i_{S_{d \rightarrow e}}(\omega)+\Lambda^i_{S_{d \rightarrow e}}(\omega)]d\omega \\[5pt]
& \text{s.t.}
& & \begin{bmatrix}
\Gamma^i_{S_{d \rightarrow e}}(\omega) & YD_p \\
(YD_p)^* & (Y+G^iX)^*(Y+G^iX)
\end{bmatrix} \succeq 0, \\
& & & \begin{bmatrix}
\Lambda^i_{S_{d \rightarrow e}}(\omega) & YP^i_{dv}D_p \\
(YP^i_{dv}D_p)^* & (Y+G^iX)^*(Y+G^iX)
\end{bmatrix} \succeq 0, \\
%& & & \begin{bmatrix}
%\Gamma_{S_{d \rightarrow %e}}(\omega) & YD \\
%D^*Y^* & %(Y+G_iX)^*(Y+G_iX)
%\end{bmatrix} \succeq 0, \\
& & & H_\infty \mbox{ and } H_2 \mbox{ constraints}.
\end{aligned}
\end{equation}

§.§ Minimum $H_2$ norm controller

As was shown previously, the $H_2$ norm minimization is a non-convex problem. The quadratic part can be linearized around a nominal controller to make the constraint convex or more precisely a Linear matrix Inequality (LMI). The final controller can be designed by first obtaining the nominal  controller using just $H_\infty$ constraints without any objective and then linearizing around it to obtain a better controller with the $H_2$ constraints and $H_2$ objective. The detailed algorithm is shown algorithm <ref>.

Iterative algorithm for minimum $H_2$ norm controller
[1]
$N_{iter} \in \mathcal{Z^+}$ number of iterations required for satisfactory convergence
Define $H_\infty$ Constraints
Optimize($H_\infty$ Constraints)
Obtain initial controller $K_0 \!=\! X_0/Y_0$
Define linearized $H_2$ Constraints with $X_0,Y_0$
$k\leq N_{iter} $
Optimize($H_2$ Objective, $H_\infty , H_2$ Constraints)
Obtain $K_k \!=\! X_k/Y_k$
Define linearized $H_2$ Constraints with $X_k,Y_k$

§ RESULTS

The optimization problem with all the constraints presented in the previous section was set up using MOSEK and YALMIP in MATLAB R2021a. The optimization problem was solved on a Windows 11 Lenovo Thinkpad PC with AMD Ryzen 7 4700U 2.00 GHz processor once with 300 equally spaced frequency data points and once with 10000 equally spaced frequency data points  from 0 to 25200 Hz. The design process took on average 1040 seconds when the data points were 300 and 7320 seconds for 10000 data points when the controller order was 12 and the number of iterations used for the minimization of $H_2$ norm for both the cases were 2. The obtained controller was able to stabilize the closed loop system.

The closed loop disturbance to error sensitivity and the disturbance to input of VCM along with the corresponding inverse weights (dotted lines) used in the $H_\infty$ constraints are shown in Fig. <ref> and Fig. <ref> for both the single stage and the dual stage systems. The plots show that the $H_\infty$ constraints are respected. It can also be seen that the low frequency disturbance attenuation characteristics of the designed controller are suitable to meet our design requirements. A bandwidth of 1 kHz was achieved for the closed loop system.

Bode plots of closed loop disturbance to error sensitivities for the single stage $S^{VCM}_{d \rightarrow e}$ (left) and the dual stage $S^{DSA}_{d \rightarrow e}$ (right) for all the nine cases. Dotted blue lines show their corresponding inverse weights.

Bode plots of closed loop disturbance to VCM input transfer functions for the single stage $U^{VCM}_{d \rightarrow u_v}$ (left) and dual stage $U^{DSA}_{d \rightarrow u_v}$ (right) for all the nine cases. Dotted blue lines show their corresponding inverse weights.

The closed loop system with the designed controller was simulated with uniformly distributed noises for 9 different plants to evaluate the performance and robustness of the controller. The total output of the DSA system and the output of the PZT actuator with the designed controller are plotted in Fig. <ref>. The maximum values of the output of the PZT actuator and the $3\sigma$ values of the total output of the DSA system for all the 9 cases are shown in Fig. <ref>. It can be seen that the output of the PZT actuator remains within the desired limits for all the 9 cases. The worst case 3$\sigma$ value of the output of DSA was found to be 18.8% of the the track width.

Moving means of total head position of the DSA system (bottom) and the output of the PZT actuator (top) in nanometers for all the 9 cases.

Maximum value of the output of PZT in each case in nm (left) and $3\sigma$ values of the total head position as a percentage of track width in each case (right)

§ CONCLUSIONS

In this paper, a data driven control design framework has been used to design a robust track following controller for a benchmark dual stage actuator hard disk drive.

The disturbance rejection problem has been formulated as a $H_2$ norm objective minimization problem with $H_\infty$ constraints for stability and shaping the closed loop transfer functions and $H_2$ norm constraints to limit the variances of signals.

The performance of the controller was evaluated in simulation with uniformly distributed random noises and the results have been presented.

[Bagherieh , 2017]
O. Bagherieh,
Estimation, identification and
data-driven control design for hard disk drives,
PhD thesis, University of California, Berkeley, 2017. https://escholarship.org/uc/item/0998n097

[Prakash et al., 2022]
N.P.S.Prakash and R. Horowitz
Data-Driven Robust Feedback Control Design for Multi-Actuator hard Disk Drives
Modeling Estimation and Controls Conference, 2022.

[Atsumi et al., 2020]
T. Atsumi and S. Yabui
Quadruple-Stage Actuator System for Magnetic-Head Positioning System in HDDs
The IEEE Transactions on Industrial
Electronics, volume 67(11), pages 9184-9194, 2020.

[Eguchi et al. , 2020]
T. Eguchi, Y. Asai, K. Ichikawa, and M. Takada Airborne and Structure-Borne Transmission of High Frequency Fan Vibration in a Storage Box Proceedings of the ASME 2017 Conference on Information Storage and Processing
Systems, paper No. 5406, 2017.

[Galdos et al., 2010]
G. Galdos, A. Karimi and R. Longchamp
controller design for spectral mimo models by convex
Journal of Process Control, volume 20(10), pages 1175–1182, 2010.

[Karimi et al., 2017]
A. Karimi and C. Kammer
A data driven approach to robust
control of multivariable systems by convex optimization”.
Automatica, 2017.

[Karimi et al., 2016]
A. Karimi, A. Nicoletti and Y. Zhu
Robust hinfinity
controller design using frequency domain data via
convex optimization.
International Journal of Robust and
Non Linear Control, volume 20(10), 2016.

[Shah et al , 2020]
P. Shah and R. Horowitz,
Data Driven Feedforward Control Design and Input Shaping Techniques for Multi Actuator Drives,
ASME 2020 Conference on Dynamics Systems and Control, 2020.

[Shah et al , 2020]
P. Shah, Z.Chen and R. Horowitz,
Application of MIMO Data Driven Feedback Control Design to Dual Stage Hard Disk Drives
ASME 2020 Conference on Dynamics Systems and Control, 2020.

[Chen et al , 2020]
Z. Chen, P. Shah and R. Horowitz,
Head-Positioning Control in Triple-Stage-Actuator Hard Disk Drives Using Mixed  Synthesis Methodologies
ASME 2020 Conference on Dynamics Systems and Control, 2020.

[Shah et al , 2019]
P. Shah and R. Horowitz
Active Vibration Rejection in Multi Actuator Drives – Data Driven Approach
ASME Conference on Dynamics Systems and Control , 2019.

[Bagherieh et al , 2018]
O. Bagherieh, P. Shah and R. Horowitz,
Application of Mixed  Data Driven Control Design to Dual Stage Hard Disk Drives
ASME Conference on Dynamics Systems and Control, 2018.

[Shah et al , 2021]
P. Shah and R. Horowitz
Data Driven Feedforward Control Design and Input Shaping Techniques for Multi Actuator Drives.
ASME 2021 Letters in Dynamics Systems and Control, 2021.

[Prakash et al. , 2022]
N.P.S Prakash, Z. Chen and R. Horowitz.
Data-Driven Strictly Positive Real System Identification with prior
System Knowledge
American Controls Conference, June 2022.

[Chen et al. , 2022]
Z. Chen, N.P.S Prakash and R. Horowitz,
Adaptive Feedforward Reference Design for Active Vibration Rejection in Multi-Actuator Hard Disk Drives
American Controls Conference, June 2022.

[Mosek ApS , 2019]
Mosek Aps,
The MOSEK optimization toolbox for MATLAB manual. Version 9.0,
2019, http://docs.mosek.com/9.0/toolbox/index.html

[Lofberg , 2004]
J. Lofberg.
Yalmip: A toolbox for modeling and optimization
in matlab.
IEEE International Symposium on Computer Aided Control Systems
Design, pages 284–289, 1954.

[Prakash et al. , 2021]
N.P.S Prakash, Z. Chen and R. Horowitz.
System Identification in Multi-Actuator Hard Disk Drives with Colored Noises using Observer/Kalman Filter Identification (OKID) Framework, ArXiv, volume 2109.12460, 2021.


# EXTRACTER: Efficient Texture Matching with Attention and Gradient Enhancing
for Large Scale Image Super Resolution

###### Abstract

Recent Reference-Based image super-resolution (RefSR) has improved SOTA deep
methods introducing attention mechanisms to enhance low-resolution images by
transferring high-resolution textures from a reference high-resolution image.
The main idea is to search for matches between patches using LR and Reference
image pair in a feature space and merge them using deep architectures.
However, existing methods lack the accurate search of textures. They divide
images into as many patches as possible, resulting in inefficient memory
usage, and cannot manage large images. Herein, we propose a deep search with a
more efficient memory usage that reduces significantly the number of image
patches and finds the $k$ most relevant texture match for each low-resolution
patch over the high-resolution reference patches, resulting in an accurate
texture match. We enhance the Super Resolution result adding gradient density
information using a simple residual architecture showing competitive metrics
results: PSNR and SSMI.

Index Terms—  Reference based super-resolution, Texture transfer, Transformer,
Cross-attention, Gradient density features

##  1 Introduction

The paradigm Image Reference-Super Resolution aims to recover high-resolution
Images by transferring accurate textures from a reference image (with a
centrain similarity degree) reducing burred and artifacts. In recent years,
vision transformers have improved super-resolution results. For example,
TTSR[1] introduces attention to Ref-Super Resolution by successfully
transferring textures from the Ref image. They use a learnable VGG pre-trained
feature extractor to obtain attention matrices $Q,K,V)$ to perform a cross-
attention mechanism to find the best features for the SR reconstruction.

Lin et al. [2] proposed a novel low-resolution backbone capable of extracting
a best feature representation and adding a branch to refine the low-resolution
and reference features. Some other works [3, 4] claim that a better texture
search is required in order to obtain less blurred images and use multiple
reference images for a more accurate pattern search. Gou et al. [5] enhance
memory efficiency by using low-resolution dimensions to find correlations and
filtering patch matches for enhancing the final result and adding gradient
information using a pre-existing SISR model for the final result.

To address the above problems, we propose a search stra-tegy to efficiently
split the images into patches, find the $top_{k}$ HR matches for each LR
patch, and add structural information for enhancing the Super-Resolution
result. Specifically, we first extract deep features from a VGG19-based
architecture. Different from [1] and most of the recent methods, we split
images into patches using a $6\times 6$ window (instead of $3\times 3$) for
the deepest feature level, resulting in a more memory efficient usage that can
allow us to use large-scale images. Second, we propose a research strategy but
different from [5], we use $top_{k}$ matches between the low-resolution and
ref patches instead of the max feature for each low-resolution patch. Finally,
we merge textures at different scales and add gradient density information
form a better spatial reconstruction using a simple residual network.

The primary contributions of this paper are. First. we introduce a Search and
Transfer module to identify correlations between low-resolution and reference
patches; we use larger window in with state-of-the-art (SOTA) methods. This
significantly reduces the dimensionality of the correlation matrix and allows
to use the top-$k$ matches to enhance texture transfer. Second, we introduce a
Gradient Density-Enhancing Module (GDE) to improve the merging of textures
from different deep levels while considering gradient density information.
This module is implemented by a straightforward recurrent network. And third,
we conduct extensive experiments on benchmark datasets that provide us strong
evidence that the proposal overcomes SOTA methods.

##  2 Related Work

In recent years, Single Image Super Resolution (SISR) improved super-
resolution methods by using residual blocks[6] and designing deeper networks.
These methods use $\mathcal{L}_{1}$ and $\mathcal{L}_{2}$ losses as the
training objective functions that have demonstrated nonaccuracy for human
perception [1]. To solve this, novel methods use a GAN strategy[7] resulting
in better satisfying results or adopt classic computer vision transformation
such as gradient mapping [8].

Since the appearance of vision transformers, vision tasks has been improved.
For example, TTSR[1] introduces cross-attention to Ref-Super Resolution for
transferring textures: a patch matching based technique robust to miss-
alignment problems [9, 10]. Based on TTSR, Lin et al. [2] add channel-wise
attention. [3, 4] and use multiple image patches for transferring textures,
resulting in better results. In this direction, cross-attention mechanisms are
used and better memory usage is required. Gou et al. [5] enhance memory
efficiency by using low-resolution dimensions to find correlations and use
classical vision transformation for structural reconstruction, such as
gradient density flow.

##  3 Method

In this section, we proposed Efficient Texture Matching with Attention and
Gradient Enhancing for Image Super Resolution (EXTRACTER). It consists of four
modules: Deep Feature Extractor (DFE), Search and Transfer Module (STM),
Cross-Scale Feature Integration (CSFI), and Gradient Density Enhancing Module
(GDE). The main scheme is shown in Fig. 1.

$Lr$$Lr_{u}$$Ref_{du}$$Ref$$F_{g}$$g$$F$$T_{1}$$T_{2}$$T_{3}$$S$$x_{TT}$$T^{\prime}_{3}$$T^{\prime}_{2}$$T^{\prime}_{1}$$Sr$$CSFI$$GFE$$\begin{matrix}DFE\\\
+\\\ ST\end{matrix}$ Fig. 1: Efficient Texture Matching with Attention Scheme:
We input the low-resolution image, reference down-upsampled, and the reference
image, pass it through the Deep Feature Extractor (DFE), and obtain the $k$
relevance texture and score matrices at multiple levels with the
SearchAndTransfer (ST). Then we merge the simple features $F$ with the
attention textures using a Cross Scale Feature Integration (CSFI). Finally, we
refine the partial super-resolution result $x_{TT}$ adding gradient features
$F_{g}$ extracted from the Gradient Density map $g$ to obtain the final Super-
Resolution image.

The model produces a $4\times$ super-resolution image. It inputs
$(Lr_{u},Ref_{du},Ref)$. $Lr_{u}$ represents the bicubic upsampled low-
resolution image and $Ref_{du}$ represents a bicubic downsampling-upsampling
concatenation operation over $Ref$ image. We produce $Q,K,V$ feature maps and
find the correlation matrix ($R$) using $Q,K$ normalized inner product between
patches. Then we filtered the best patches based on correlation matrix $R$ and
then we take the $top_{k}$ matches for each patch. We integrate the obtained
features at three different scales using a Cross Scale Feature Integration[1]
and finally, we add gradient density from the LR image to improve structural
information and create the Super-Resolution image.

###  3.1 Deep Feature Extractor

We transform the data into a new representation with more evident complex
characteristics at different resolutions. For this, we use the VGG19 [11]
backbone (previously trained with ImageNet[12]). Let $(Lr_{u},Ref_{du},Ref)$
be input to our Deep Feature Extractor(DFE). The output of DFE can be
formulated as

$\displaystyle Q_{i}$ $\displaystyle=$ $\displaystyle DFE_{i}(Lr_{u})$ (1)
$\displaystyle K_{i}$ $\displaystyle=$ $\displaystyle DFE_{i}(Ref_{du})$ (2)
$\displaystyle V_{i}$ $\displaystyle=$ $\displaystyle DFE_{i}(Ref)$ (3)

where $i$ denotes the feature level of the $DFE$. We take three scales of
features from VGG19 with output channels $[64,128,256]$ and reduce the image
$2\times$ to the original scales at each level.

$Ref$$Ref_{du}$$Lr_{u}$$Lr$$DFE$$Search$$Transfer$$FeatureMerge$$Q$$K$$V$$F$$S$$T$$T_{out}$
Fig. 2: Feature extraction and texture search: The model inputs
$Lr_{u},Ref_{du},Ref$, pass it through a Deep Feature Extractor (DFE) to
perform patch Correlation Search. We use the result as index to select the
best $k$-textures by Transfer. Finally, with textures $T$ and soft-attention
matrices $S$, we merge them with simple features $F$ from $Lr$ to create
$T_{out}$.

###  3.2 Search and Transfer Module

Let is omit the $i$ index from (1) for notation simplification. The following
calculations are made for a single level of DFE, is is depict at Fig. 2. We
infer correlations between $LR_{u}$ and $Ref_{du}$ using attention via $Q$ and
$K$ at two stages. First we divide $Q,K$ into overlaping patches
$q_{i}:i\in[1,2,\ldots,H_{LR}\times W_{LR}/s^{2}]$ and
$k_{j}:j\in[1,2,\ldots,H_{Ref}\times W_{Ref}/s^{2}]$, respectively, where $s$
is the stride setput for patch displacement. In experiments, we use a window
of $6$ and stride $s=2$. The correlation matrix is computed as the normalized
inner product

$c^{\prime}_{i,j}=\left\langle\frac{q_{i}}{||q_{i}||},\frac{k_{j}}{||k_{j}||}\right\rangle.$
(4)

Next, we keep the best score indices of the $k_{j}$ patches for each of
$q_{i}$ $H^{\prime}=\arg\max_{j}(C^{\prime})$ . Using the $H^{\prime}$ matrix
as index, we extract the most relevance patches of $K$ as
$K^{\prime}=K_{H^{\prime}}$. Following, we use a re-search strategy by keeping
the best score indices of the $k^{\prime}_{j}$ normalized patches for each of
$q_{i}$ using the $top_{u}$ largest matches

$H,S=top_{u}(C)\textup{ with
}c_{i,j}=\left\langle\frac{q_{i}}{||q_{i}||},\frac{k^{\prime}_{j}}{||k^{\prime}_{j}||}\right\rangle$
(5)

with $S,H$ tensors containing the $u$-maximum scores and index for $C$; i.e.,

$H_{0}=\arg\max_{j}C_{ij}\textup{ ,}\hskip 5.69054ptS_{0}=\max_{j}C_{ij}$ (6)

and $H_{1},S_{1}$ be the second maximum indices and scores matches, etc. Now,
we select the best textures from $V$ using the $H_{i}$, $i=1,\dots,u$
matrices: $T_{i}=V_{H_{i}}$. So that, we extract the best matches using the
hard attention matrix as index. Finally, for an output of the Initial Feature
Extractor (IFE) of LR image, denote as $F=IFE(x)$. Hence, we integrate the
found features $T_{i}$:

$F_{TT}=F+\sum_{i=1}^{u}Conv_{i}(Concat(F,T_{i}\otimes S_{i}))\otimes S_{i};$
(7)

where $\otimes$, $Conv_{i}(\cdot)$ and $Contat(\cdot)$ denotes element-wise
multiplication, convolutional $3\times 3$ and concatenation blocks,
respectively.

###  3.3 Cross-Scale Feature Integration

Inspired by SoTA methods for style/texture transferring [13, 14, 1], We
integrate the previous attention results at different scales following [1];
this can be modeled as

$x_{TT},T_{1},T_{2},T_{3}=CCFI(\\{F_{TT}^{(i)}\\}_{1=1,2,\ldots});$

where $x_{TT}$ is the merged super-resolution texture and $T_{1}$, $T_{2}$,
$T_{3}$ are the syntetized textures.

###  3.4 Gradient Enhancing Density Module

To give more information about the structure of the low-resolution image, some
work has been done [8, 6]. We incorporate a Gradient Enhancing module for
adding structural and edge information to the partial output of the
$CSFI(\cdot)$. First, we extract the Gradient Density for each of the RGB
image channels we convolve the Image with $3\times 3$ Sobel filters kernels
[15] from $x$ and $y$ derivative directions; $K_{x}$ and $K_{y}$,
respectively. and calculate Gradient Density as

$GD(I)=\sqrt{(K_{x}*I)^{2}+(K_{y}*I)^{2}}.$

Now, we pass the image gradient density $g$ through a residual feature
extractor: $F_{g}=GFE(g)$. Finally, using the output from $CSFI(\cdot)$ :
$x_{TT},T_{1},T_{2},T_{3}$, the SR image is formulate as

$\displaystyle x_{1g}$ $\displaystyle=$ $\displaystyle
RB_{1}(Conv(Concat(F_{g},T_{3})))$ $\displaystyle x_{2g}$ $\displaystyle=$
$\displaystyle RB_{2}(Conv(Concat(x_{1g}\uparrow,T_{2})))$ $\displaystyle
x_{3g}$ $\displaystyle=$ $\displaystyle
RB_{3}(Conv(Concat(x_{3g}\uparrow,T_{3})))$ $\displaystyle SR$
$\displaystyle=$ $\displaystyle Conv(Concat(x_{3g},x_{TT}))$

where $RB(\cdot)$ represents a residual scheme and $\uparrow$ is $2\times$
bicubic upsampling.

###  3.5 Loss Function

The overall loss is

$\mathcal{L}_{total}=\lambda_{1}\mathcal{L}_{rec}+\lambda_{2}\mathcal{L}_{perc}+\lambda_{3}\mathcal{L}_{grad}+\lambda_{4}\mathcal{L}_{adv}$
(8)

where

$\mathcal{L}_{rec}=(chw)^{-1}||SR-HR||_{1},$

with $c,h,w$ the channel, height, weight of the $HR$ image. In the aim of
enhacing the similarity of the feature space representation of the generated
image and the $SR$ image using the $vgg19$ feature space [16, 17], we use

$\mathcal{L}_{perc}=(c_{i}h_{i}w_{i})^{-1}||vgg19_{i}(SR)-vgg19_{i}(HR)||_{1},$

with $c_{i},h_{i},w_{i}$ the channel, height, weight at the correspoinding $i$
level. For structural similarity enhacing, we introduce Gradient Density Loss
using (3.4)

$\mathcal{L}_{grad}=(chw)^{-1}||GD(SR)-GD(HR)||_{1},$

with $c_{i},h_{i},w_{i}$ the channel, height, weight at the correspoinding $i$
level. Similar to [1, 18], we use a WGAN-GP for more stable training. This
loss is described as

$\displaystyle\mathcal{L}_{D}$ $\displaystyle=$
$\displaystyle\mathbb{E}_{\tilde{x}\sim\mathbb{P}_{g}}\left[D(\tilde{x})\right]-\mathbb{E}_{x\sim\mathbb{P}_{r}}\left[D(x)\right]$
$\displaystyle+\lambda\mathbb{E}_{\hat{x}\sim\mathbb{P}_{\hat{x}}}\left[\left\|\nabla_{\hat{x}}D(\hat{x})\right\|_{2}-1)^{2}\right],$
$\displaystyle\mathcal{L}_{G}$ $\displaystyle=$
$\displaystyle-\mathbb{E}_{\tilde{x}\sim\mathbb{P}_{g}}\left[D(\tilde{x})\right].$

###  3.6 Implementation Details

The window size for extracting patches is set as $k=6$ with padding $p=2$ and
a stride of $s=2$. In experiments, we explore other configurations. The
architecture for the CSFI model is $[16,8,4]$, $[9,9,9]$ for GDE and $4$
residual blocks for IFE’s. For the correlation matrix, we use only the deepest
feature extractor level to perform matrix multiplication. We use data
augmentation for training by randomly flipping up-down and left-right followed
by a random rotation of $90^{\circ},180^{\circ},270^{\circ}$ with a batch
fixed to $9$. The weights of the loss coefficients are
$1,1e^{-2},1e^{-3},1e^{-3}$ in the same order of equation (8). An Adam
optimizer with $lr=1e^{-4}$, $\beta_{1}=0.9$, $\beta_{2}=0.999$ and default
$\epsilon=1e^{-8}$. All the experiments were performed in a single GPU NVIDIA
GeForce RTX 3090 using the pytorch framework.

##  4 EXPERIMENTS AND RESULTS

Following the recent work, we use two metrics to evaluate the results: Peak
Signal to Noise Ratio (PSNR) and Structure Similarity Index (SSIM) [19]. We
conduct the training using CUFED5 Dataset [20]. It contains 11,871 pairs
consisting of an input and reference image. There are 126 testing images, each
having 4 reference images with different similarity levels. We also evaluate
our method using different text sets such as Sun80 [21], Urban100 [22], and
Set14[23]. Sun80 contains 80 natural images, each of them paired with several
reference images. Urban100 and Set14 do not have reference images so we took
it randomly from the same dataset. All the SR results are evaluated of PSRN
and SSIM on the Y channel of YCbCr space. Following the SOTA methods, we train
our model using the train set from CUFED5 and test it on the CUFED5 test set,
Sun80, Urban100, and Set14. Two versions of our model were trained, the first
one trained only using reconstruction loss and the second using all losses.
EXTRACTER-rec outperforms recent methods despite using a bigger window size,
as we can see in Table 1. We observe better visual results when all losses
were used, Fig. 1 illustrates some visual results with other novel models. We
study different configurations for our model. Table 3 shows the number of
parameters and the correlation matrix shape during the training phase for the
CUFED5 dataset. We found that our method reduces $4\times$ the shape from the
attention mechanism. Table 3 shows the effectiveness of changing the kernel
size for the test phase using large image size datasets such as Sun80 and
Urban100.

Method | CUFED5 | Sun80 | Urban100 | Set14
---|---|---|---|---
SRNTT | 25.61 / .764 | 27.59 / .756 | 25.09 / .774 | 26.73 / .731
SRNTT-rec | 26.24 / .784 | 28.54 / .793 | 25.50 / .784 | 27.68 / .766
TTSR | 25.63 / .765 | 28.59 / .774 | 24.69 / .748 | 26.88 / .748
TTSR-rec | 27.03 / .802 | 30.02 / .814 | 25.88 / .784 | 28.10 / .782
SSEN-rec | 26.78 / .791 | - | - | -
DPFSR | 25.23 / .749 | 28.59 / .774 | 24.35 / .734 | -
DPFSR-rec | 27.25 / .808 | 30.10 / .815 | 26.03 / .787 | -
$C^{2}$\- Matching | 27.16 / .805 | 29.75 / .799 | 25.52 / .764 | -
Extracter | 26.40 / .789 | 29.02 / .789 | 24.72 / .752 | 26.50/.740
Extracter-rec | 27.29 / .811 | 30.02 / .816 | 26.04 / .785 | 28.09 / .782
Table 1:  Quatitative metrics of the generated images using PSNR / SSIM. The 2-highest scores are denoted in black. Method | Params. (M) | Kernel size | corr. matrix shape
---|---|---|---
TTSR | 6.73 | $3\times 3$ | $1600\times 1600$
DPFSR | 6.91 | $3\times 3$ | $1600\times 1600$
Extracter | 9.31 | $6\times 6$ | $800\times 800$
Table 2: Model parameters and shape of the training correlation matrix. Our method reduce significantly the matrix multiplication cost by extracting larger patches. Kernel Size | Sun80 | Urban100
---|---|---
$3\times 3$ | OFM | OFM
$6\times 6$ | 30.02 / .816 | 26.04 / .785
$12\times 12$ | 29.98 / .814 | 25.74 / .781
Table 3: Kernel size when obtaining patches for PSNR / SSIM metrics with our
model. All comparations where made on single GPU. Models using $3\times 3$
kernel like TTSR and DPFSR produces Out of Memory (OFM) due the large image
dimensions on Sun80 and Uban100 datasets.

Input | HR | SRNTT
---|---|---
Reference | TTSR | EXTRACTER

Input | HR | TTSR
---|---|---
| Reference | EXTRACTER

Fig. 3: Experimental results: we compare our model with available testing
models online.

##  5 CONCLUSIONS AND FUTURE WORK

In this paper, we propose a novel deep texture search with more efficient
memory usage for RefSR. The proposed model consists of a learnable Deep
Feature Extractor, a Search and Transfer Module that uses the top-$k$ matches
between the Lr and Ref patches for transferring textures in a more efficient
memory usage way than SOTA methods by using larger windows, a Cross Scale
Feature Integrator and, finally, a Gradient Enhancing Density module. Our
experiments demonstrate the competitive performance of EXTRACTER over the
recent attention mechanisms for RefSR using PSRN and SSIM metrics. The
ablation studies demonstrate the efficiency of managing larger windows when
using large-scale images, resulting in a non-out-of-memory as other recent
methods. In the future, we would like to enhance our model by changing the
CSFI for a simpler network to reduce training time, using the transferring
mechanisms to refine generative models, and exploring RefSR real-world
applications, such as satellite super-resolution and movie super-resolution.

Acknowledges. Work supported by Conacyt, Mexico (Grant CB-A1-43858) and E.
Reyes Scholarship.

## References

  * [1] F. Yang, H. Yang, J. Fu, H. Lu, and B. Guo,  “Learning texture transformer network for image super-resolution,”  in Proc. CVPR, 2020, pp. 5791–5800.
  * [2] Ruirong Lin and Nanfeng Xiao,  “Dual projection fusion for reference-based image super-resolution,”  Sensors, vol. 22, no. 11, pp. 4119, 2022.
  * [3] Xu Yan, Weibing Zhao, Kun Yuan, Ruimao Zhang, Zhen Li, and Shuguang Cui,  “Towards content-independent multi-reference super-resolution: Adaptive pattern matching and feature aggregation,”  in Computer Vision – ECCV. 2020, pp. 52–68, Springer Int. Pub.
  * [4] Yanchun Xie, Jimin Xiao, Mingjie Sun, Chao Yao, and Kaizhu Huang,  “Feature representation matters: End-to-end learning for reference-based image super-resolution,”  in Computer Vision – ECCV, Cham, 2020, pp. 230–245, Springer Int. Pub.
  * [5] Kehua Guo, Liang Chen, Xiangyuan Zhu, Xiaoyan Kui, Jian Zhang, and Heyuan Shi,  “Double-layer search and adaptive pooling fusion for reference-based image super-resolution,”  ACM Trans. Multimedia Comput. Commun. Appl., 2023.
  * [6] Ziyu Liu, Ruyi Feng, Lizhe Wang, and Tieyong Zeng,  “Gradient prior dilated convolution network for remote sensing image super-resolution,”  IEEE Jou. Selected Topics in Applied Earth Observations and Remote Sensing, vol. 16, pp. 3945–3958, 2023.
  * [7] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi,  “Photo-realistic single image super-resolution using a generative adversarial network,”  in Proc. CVPR, 2017, pp. 4681–4690.
  * [8] Jian Sun, Zongben Xu, and Heung-Yeung Shum,  “Gradient profile prior and its applications in image super-resolution and enhancement,”  IEEE Trans. on image process., vol. 20, pp. 1529–42, 11 2010.
  * [9] Huanjing Yue, Xiaoyan Sun, Jingyu Yang, and Feng Wu,  “Landmark image super-resolution by retrieving web images,”  IEEE Trans. on Image Process., vol. 22, no. 12, pp. 4865–4878, 2013\.
  * [10] Haitian Zheng, Mengqi Ji, Haoqian Wang, Yebin Liu, and Lu Fang,  “Crossnet: An end-to-end reference-based super resolution network using cross-scale warping,”  in Proc. CVPR, 2018, pp. 88–104.
  * [11] K Simonyan and A Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  in 3rd International Conference on Learning Representations (ICLR 2015). Computational and Biological Learning Society, 2015.
  * [12] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei,  “Imagenet: A large-scale hierarchical image database,”  in Proc. IEEE/CVF CVPR. Ieee, 2009, pp. 248–255.
  * [13] L. Gatys, A. Ecker, and M. Bethge,  “A neural algorithm of artistic style,”  Jou. of Vision, vol. 16, no. 12, pp. 326–326, 2016.
  * [14] Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo,  “Learning pyramid-context encoder network for high-quality image inpainting,”  in Proc. CVPR, 2019, pp. 1486–1494.
  * [15] N. Kanopoulos, N. Vasanthavada, and R.L. Baker,  “Design of an image edge detection filter using the sobel operator,”  IEEE Jou. of Solid-State Circuits, vol. 23, no. 2, pp. 358–367, 1988.
  * [16] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee,  “Enhanced deep residual networks for single image super-resolution,”  in Proc. CVPR Workshops, 2017, pp. 136–144.
  * [17] Jianbo Wang, Huan Yang, Jianlong Fu, Toshihiko Yamasaki, and Baining Guo,  “Fine-grained image style transfer with visual transformers,”  in Proc. ACCV, 2022, pp. 841–857.
  * [18] Reyes-Saldana Esteban and Rivera Mariano,  “Deep variational method with attention fo high-definition face generation,”  in Pattern Recognition: 14th Mexican Conference, MCPR 2022, Ciudad Juárez, Mexico, June 22–25, 2022, Proceedings, Berlin, Heidelberg, 2022, p. 116–126, Springer-Verlag.
  * [19] Alain Horé and Djemel Ziou,  “Image quality metrics: Psnr vs. ssim,”  in ICPR, 2010, pp. 2366–2369.
  * [20] Zhifei Zhang, Zhaowen Wang, Zhe Lin, and Hairong Qi,  “Image super-resolution by neural texture transfer,”  in Proc. CVPR, 2019, pp. 7982–7991.
  * [21] Libin Sun and James Hays,  “Super-resolution from internet-scale scene matching,”  in 2012 IEEE Int. Conf. Computational Photography, 2012, pp. 1–12.
  * [22] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja,  “Single image super-resolution from transformed self-exemplars,”  in Proc. CVPR, 2015, pp. 5197–5206.
  * [23] Roman Zeyde, Michael Elad, and Matan Protter,  “On single image scale-up using sparse-representations,”  in Curves and Surfaces, Berlin, Heidelberg, 2012, pp. 711–730, Springer Berlin Heidelberg.