Quantum-Classical Multiple Kernel Learning

Ara Ghukasyan
Agnostiq Inc., 325 Front St W, Toronto, ON M5V 2Y1
University of Waterloo, 200 University Ave W, Waterloo, ON N2L 3G1

Jack S. Baker
Agnostiq Inc., 325 Front St W, Toronto, ON M5V 2Y1

Oktay Goktas
Agnostiq Inc., 325 Front St W, Toronto, ON M5V 2Y1

Juan Carrasquilla
University of Waterloo, 200 University Ave W, Waterloo, ON N2L 3G1
Vector Institute,  661 University Ave Suite 710, Toronto, ON M5G 1M1

Santosh Kumar Radha
Agnostiq Inc., 325 Front St W, Toronto, ON M5V 2Y1

As quantum computers become increasingly practical, so does the prospect of using quantum computation to improve upon traditional algorithms. Kernel methods in machine learning is one area where such improvements could be realized in the near future. Paired with kernel methods like support-vector machines, small and noisy quantum computers can evaluate classically-hard quantum kernels that capture unique notions of similarity in data. Taking inspiration from techniques in classical machine learning, this work investigates simulated quantum kernels in the context of multiple kernel learning (MKL). We consider pairwise combinations of several classical-classical, quantum-quantum, and quantum-classical kernels in an empirical investigation of their classification performance with support-vector machines. We also introduce a novel approach, which we call QCC-net (quantum-classical-convex neural network), for optimizing the weights of base kernels together with any kernel parameters. We show this approach to be effective for enhancing various performance metrics in an MKL setting. Looking at data with an increasing number of features (up to 13 dimensions), we find parameter training to be important for successfully weighting kernels in some combinations. Using the optimal kernel weights as indicators of relative utility, we find growing contributions from trainable quantum kernels in quantum-classical kernel combinations as the number of features increases. We observe the opposite trend for combinations containing simpler, non-parametric quantum kernels.

§ INTRODUCTION

Research towards developing novel computational paradigms aims to overcome the limitations of classical computers. Quantum computing leverages the unique properties of quantum mechanics as a promising alternative for tackling some classically infeasible problems. New algorithms for noisy intermediate-scale quantum (NISQ) devices <cit.> continue to emerge, spurred on by recent demonstrations of quantum supremacy <cit.>. In the field of machine learning, kernel methods <cit.> are one promising application of this technology <cit.>.

Kernel methods, such as support-vector machines (SVMs) <cit.>, are prominent in classical machine learning due to theoretical guarantees associated with their convex loss landscapes. SVMs and related algorithms can perform non-linear classification to capture complex patterns in data. One limitation of kernel methods is that they require computing an $M \times M$ kernel matrix, where $M$ is the number of training samples. On the other hand, many quantum kernel methods <cit.> are immediately suitable for NISQ devices.

An overview of prospective multiple kernel learning methods in the era of quantum computing, emphasizing combinations of quantum (q) and/or classical (c) kernels, as well as quantum or classical optimization approaches.

Quantum kernel methods, like quantum support-vector machines (QSVMs) <cit.>, have shown promise in various applications <cit.>, including supernova classification in cosmology <cit.>, probing phase transitions in quantum many-body physics <cit.>, and detecting fraud in finance <cit.>. The crux of the QSVM is embedding input data into quantum states, which allows kernels to be estimated from the overlap of these states. Importantly, the quantum feature spaces accessed by quantum kernel methods can be tailored to data <cit.> by training parametric embeddings.

Multiple kernel learning (MKL) aims to enhance performance by combining different kernels into a single, more expressive kernel <cit.>, in order to learn a wider variety of decision functions. Here, tuning combinations in a data-driven way <cit.> provides an additional means tailoring kernels. MKL can facilitate feature selection and the incorporation of domain-specific knowledge, which are essential for addressing real-world problems <cit.>. MKL also encourages sparsity in the combination of kernels, effectively performing model selection in learning the optimal weights for component kernels. This can reduce the risk of overfitting and improve generalization performance <cit.>. To date, MKL has proven valuable in various applications, including image classification, natural language processing, bioinformatics, and drug discovery <cit.>.

As the NISQ era progresses, MKL may prove useful for near term applications that rely on both quantum and classical computing. These paradigms can be combined in six different ways, as illustrated in Fig. <ref>; utilizing classical-classical (c.c.), quantum-classical (q.c.), or quantum-quantum (q.q.) kernel combinations. Either classical or quantum techniques can also be applied to the subsequent optimization problem as well. For example, recent research has focused on using quantum annealers to solve the SVM problem for fully classical kernels <cit.>. Multiple quantum kernels have also been combined in the fully-quantum (q.q.) setting, using deterministic quantum computing with one qubit (DQC1) <cit.>, and solving the SVM problem on a classical computer.

A foremost goal of this paper is to systematically address c.c., q.q., and especially q.c. kernel combinations, of which the latter is noticeably lacking in existing literature. Using the classical EasyMKL algorithm <cit.> to optimize kernel weights, we consider pairwise combinations within a set of three quantum and three classical kernels. We also introduce another training step to fine-tune any parametric component kernels in our end-to-end learnable quantum-classical-convex neural network (QCC-net) method (also used for time-series analysis in Ref.  <cit.>). We report performance metrics for the various combinations in a supervised classification setting using SVMs, across synthetic datasets with a range of two to thirteen features.

The remainder of this paper is organized as follows: Sec. <ref> provides a brief overview of kernel methods, introduces the concepts behind quantum embedding kernels, and provides additional motivation for hybrid paradigms in MKL. We then present our experimental methodology in Sec. <ref>, including implementation details, descriptions of the base classical and quantum kernels, as well as details of data preparation and the QCC-net. Empirical results are discussed in Sec. <ref> in terms of performance metrics and optimal kernel weights. Finally, we draw insights from these results and suggest future directions in Sec. <ref>.

§ KERNEL METHODS

Kernel methods are machine learning algorithms that rely on a kernel function to compute numerical notions of pairwise similarity. Any function $k(\bm{x}, \bm{x}^{\prime})$ is a kernel if there exists a feature map $\bm{\Phi} : \mathbb{R}^d \rightarrow \mathcal{F}$, such that

\begin{align}
k(\bm{x}, \bm{x}^{\prime}) = \langle \bm{\Phi}(\bm{x}^{\prime}), \bm{\Phi}(\bm{x})\rangle
\label{eq:inner-product-kernel}
\end{align}

for all $\bm{x} \in X \subset \mathbb{R}^d$.

Kernels provide “shortcuts” into a feature space $\mathcal{F}$ via the kernel trick.
Consider a linear classifier

\begin{align}
y_\text{pred}(\bm{x}) = \text{sgn}(\langle \bm{w}, \bm{x} \rangle + b),
\label{eq:linear-decision}
\end{align}

which predicts binary labels $y_i \in \{-1, +1\}$ for points $\bm{x}\in\mathbb{R}^d$, using a separating hyperplane defined by the normal vector $\bm{w} \in \mathbb{R}^d$ and a distance to the origin $b \in \mathbb{R}$. If the classes in $X$ are not linearly separable, the algorithm can be attempted in an alternate Hilbert space, finding $\tilde{\bm{w}} \in \mathcal{F}$ and $\tilde{b} \in \mathbb{C}$ such that

\begin{align}
y_\text{pred}(\bm{x}) = \text{sgn}(\langle \tilde{\bm{w}}, \bm{\Phi}(\bm{x}) \rangle + \tilde{b}).
\label{eq:linear-decision-phi}
\end{align}

correctly classifies $\bm{\Phi}(\bm{x})$ (and thereby $\bm{x}$).

Instead of evaluating Eq. (<ref>) directly, the representer theorem <cit.> is applied to re-express $\tilde{\bm{w}}$ as a finite linear combination with real coefficients $\alpha_m$,

\begin{align}
\tilde{\bm{w}} = \sum_m \alpha_m \bm{\Phi}(\bm{x}^{(m)}),
\label{eq:representer}
\end{align}

which yields a classifier that operates on $\bm{\Phi}(\bm{x})\in\mathcal{F}$, without directly evaluating any inner products:

\begin{align}
y_\text{pred}(\bm{x}) = \text{sgn}\Bigg(\sum_m \alpha_m k(\bm{x}, \bm{x}^{(m)}) + b\Bigg)
\label{eq:linear-decision-representer}
\end{align}

Thus, the kernel trick enables non-linear classification with linear algorithms, using an implicit transformation of $X$ through a feature map $\bm{\Phi}$.

Kernel methods are more transparent to formal analysis (as compared to neural networks, for example) and often lead to convex problems that benefit from provable performance guarantees <cit.>. Their connection to supervised quantum machine learning (QML) models <cit.>, which can be re-formulated as kernel methods <cit.>, has stoked additional interest for near-term quantum applications. As a more distant prospect, kernel-based problems are especially well suited for fault-tolerant quantum computers <cit.>.

kernel name     kernel function     parameters

Linear     $\langle \bm{x}, \bm{x}^{\prime} \rangle$     none

Polynomial     $(\theta_0 \langle \bm{x}, \bm{x}^{\prime} \rangle + \theta_1)^3$     $\{\theta_0, \theta_1$}

RBF     $\exp(-\theta_2 ||\bm{x} - \bm{x}^{\prime} ||^2)$     $\{\theta_2$}

Classical kernels considered in this work. Default parameter values are chosen to be $(\theta_0, \theta_1) = (1/d, 1)$ and $\theta_2 = 1$.

[column sep=0.3cm]

[transparent, column sep=0.3cm]
H     R_z(x_1)     [2]ZZ_12 (x_1 x_2)     [3, label style=yshift=0.7cm]ZZ_13(x_1 x_3)

H     R_z(x_2)             [2]ZZ_23 (x_2 x_3)

H     R_z(x_3)

[transparent, column sep=0.3cm]
R_x(x_1)     [2]ZZ_12 (θ_1)     [3, label style=yshift=0.7cm]ZZ_13(θ_2)         R_y(θ_4)

R_x(x_2)             [2]ZZ_23 (θ_3)     R_y(θ_5)

R_x(x_3)                 R_y(θ_6)

kernel name     embedding unitary     Eqs.     parameters     embedding circuit $(U_{\bm{\theta}}(\bm{x}))$     ref.

RX     $\mathcal{R}_{\text{X}}(\bm{x})$     (<ref>)     -     !1cm     -

IQP     $V(\bm{x}) H^{\otimes N}$     (<ref>)     -     !1.5cm     <cit.>

QAOA     $W(\bm{\theta}) \mathcal{R}_{\text{X}}(\bm{x})$     (<ref>, <ref>)     $\bm{\theta} \in \mathbb{R}^{2N}$     !1.4cm     <cit.>

Quantum kernels considered in this work. Circuit diagrams provide a 3-qubit example of the embedding circuit (i.e. one half of the kernel circuit). Initial parameters for the QAOA circuit are chosen uniformly at random from $[0,\,2\pi]^{2N}$. Equations defining the embedding circuits are provided in Sec. <ref>.

§.§ Quantum Kernels

Quantum embedding kernels (QEKs) <cit.> implement a similarity measures via fidelity estimates between data-representing quantum states. QEKs are an attractive prospect insofar as classically-hard kernel functions can be realized with parametric operations that encode data points as quantum states <cit.>. To achieve efficient and expressive  QEKs, the quantum feature space should be readily parametrized by only a handful of gate variables (here denoted by $\bm{\theta}$).

To construct a QEK, one must choose a unitary transformation $U_{\bm{\theta}}$ to define the embedding,

\begin{align}
\ket{\bm{\Phi}(\bm{x})} = U_{\bm{\theta}}(\bm{x})\ket{0}.
\label{eq:quantum-embedding}
\end{align}

Here, a corresponding feature map is one that takes $\bm{x}$ to a density matrix $\rho(\bm{x}) = \ket{\bm{\Phi}(\bm{x})}\bra{\bm{\Phi}(\bm{x})}$. Note that any possibility for quantum advantage is lost if the embedding is too simple <cit.>, so $U$ must be chosen carefully. The QEK itself is defined by the Frobenius inner product $\langle \rho(\bm{x}^{\prime}), \rho(\bm{x}) \rangle$, or equivalently

\begin{align}
\text{Tr} \{ \rho(\bm{x}^{\prime}) \rho(\bm{x}) \} = | \braket{\bm{\Phi}(\bm{x}^{\prime})}{\bm{\Phi}(\bm{x})} |^2 = k_{\bm{\theta}}(\bm{x}, \bm{x}^{\prime}).
\label{eq:quantum-kernel}
\end{align}

Any of several existing methods for fidelity estimation <cit.> can be used to evaluate Eq. (<ref>) in practice, allowing a hybrid classifier to be implemented by calling $k_{\bm{\theta}}(\bm{x}, \bm{x}^{\prime})$ inside a classical linear algorithm (e.g. Eq. (<ref>)). More generally, the modular nature of kernel methods provides a convenient setting for high-level manipulation of both quantum and/or classical kernels.

While the kernel trick is simple to apply, it remains challenging to determine an effective kernel for a given classification problem. To this end, heuristic <cit.> or cost-based optimization <cit.> can be leveraged to improve the performance of parametric kernels. MKL is a complementary approach that considers different kernels in combination, and determines the weights for component kernels.

§.§ Multiple Kernel Learning

The goal in MKL is to improve a kernel method's performance by introducing a novel notion of similarity derived from multiple distinct kernels. Typical use cases for MKL include affecting feature selection <cit.>, enabling anomaly detection <cit.>, and enhancing expressivity <cit.>. Many kernel combination strategies have been proposed for MKL <cit.>, although weighted linear combinations are often effective <cit.>.

A combination function $f_{\bm{\gamma}} : \mathbb{R}^R \rightarrow \mathbb{R}$ and a base set of kernels $k_{\bm{\theta}}^{(r)} : \mathbb{R}^d \times \mathbb{R}^d \rightarrow \mathbb{R}$ define a combined kernel <cit.>

\begin{align}
k_{\bm{\theta},\bm{\gamma}}(\bm{x}, \bm{x}^{\prime}) := f_{\bm{\gamma}}\Big(\{k_{\bm{\theta}}^{(r)}(\bm{x}, \bm{x}^{\prime})\}_{r=1}^R\Big)
\label{eq:combined-kernel}
\end{align}

with adjustable parameters $\bm{\theta}$ and $\bm{\gamma}$ for $k_{\bm{\theta}}^{(r)}$ and $f_{\bm{\gamma}}$, respectively. For brevity of notation, $\bm{\theta}$ is assumed to contain all kernel parameters and each kernel $k_{\bm{\theta}}^{(r)}$ is assumed to utilize only a subset “$r$” of $\bm{\theta}$.

To be a valid kernel, the resulting $k_{\bm{\theta}, \bm{\gamma}}$ must be positive semi-definite, which means the inequality

\begin{align}
\bm{z}^T K_{\bm{\theta},\bm{\gamma}} \bm{z} \geq 0
\label{eq:positive-semi-definite}
\end{align}

must hold for all $\bm{z} \in \mathbb{R}^M$, where $M = \#\{\hat{Z}\}$ for any (improper) subset $\hat{Z}$ of the training set, $\hat{X} \subset X$. Here, $K_{\bm{\theta},\bm{\gamma}} \in \mathbb{R}^{M \times M}$ is the Gram matrix <cit.> of all pairwise kernels, $k_{\bm{\theta},\bm{\gamma}}(\bm{x}, \bm{x}^\prime)$, for $\bm{x},\bm{x}^\prime \in \hat{Z}$. If the Gram matrices of all $R$  base kernels satisfy Eq. (<ref>), then their additive or multiplicative combinations are also guaranteed to be valid kernels <cit.>.

Illustration of a hybrid kernel that combines inner products in $C_1$, a generic (“classical”) Hilbert space, with inner products from $Q_1$, the state space of a quantum system.

A convex linear combination of base kernels (Fig. <ref>) is constructed by taking

\begin{align}
f_{\bm{\gamma}} = \langle \bm{\gamma}, \cdot \rangle
\label{eq:f-gamma-linear}
\end{align}

with the following constraints on $\bm{\gamma}$:

\begin{align}
||\bm{\gamma}||_1 = 1,\quad \gamma_r \geq 0.
\label{eq:gamma-constraint}
\end{align}

As a weighted average, this form is both easy to interpret and computationally convenient. Using Eq. (<ref>), we can infer that the resulting kernel,

\begin{align}
k_{\bm{\theta}, \bm{\gamma}}(\bm{x}, \bm{x}^{\prime}) &:= \sum_{r=1}^R \bm{\gamma}_r k^{(r)}_{\bm{\theta}}(\bm{x}, \bm{x}^{\prime}),
\label{eq:kernel-linear-combination}
\end{align}

corresponds to a feature map

\begin{align}
\bm{\Phi}_{\bm{\theta},\bm{\gamma}}(\bm{x}) = [\sqrt{\gamma_1}\bm{\Phi}_{\bm{\theta}}^{(1)}(\bm{x}), ... , \sqrt{\gamma_R}\bm{\Phi}_{\bm{\theta}}^{(R)}(\bm{x})]^T.
\label{eq:combined-feature-map}
\end{align}

Here, data-driven optimization of $\bm{\gamma}$ effectively decides the importance of each component feature map. Once all the relevant kernels have been evaluated, the MKL technique does not distinguish between kernels of quantum and/or classical nature. Instead, the kernel values are simply weighted to maximize an optimization objective (Sec. <ref>) in view of the training data.

To contrast, the q.q. implementation by Vedaie et al. <cit.> involves preparing a parametrized mixed state,

\begin{align}
\rho_{\bm{\gamma}} = \sum_{r = 1}^{R} \gamma_{r} \ket{r}^{\otimes q}\bra{r}^{\otimes q},
\label{vedaie-mixed-state}
\end{align}

which represents an $R$-component kernel of the form in Eq. (<ref>); weighted by the classical probabilities of the mixture, and computed via the expectation value of an encoding operator. Here, each individual kernel is evaluated at the same time, using a single quantum circuit.

In our approach, the kernel (or Gram) matrices,

\begin{align}
[K^{(r)}_{\bm{\theta}}]_{\bm{x},\bm{x}^\prime} = k^{(r)}_{\bm{\theta}}(\bm{x}, \bm{x}^{\prime}),\quad\forall\bm{x},\bm{x}^\prime\in\hat{X}
\label{eq:gram-matrix}
\end{align}

are computed separately for each quantum or classical kernel. The combination parameters ($\bm{\gamma}_r$) are then determined classically, using these matrices and the training labels as inputs. This allows us to treat c.c., q.q., and q.c. kernel combinations identically, within a general optimization framework.

§.§.§ Quantum-Classical Kernel Combinations

Since computing common classical kernels adds no significant overhead compared to querying quantum computers or simulating quantum circuits, a useful q.c. combination could be an easy way to make the most of available NISQ hardware. Feature spaces associated with quantum kernels are unique insofar as (1) they can be classically-hard to simulate, and (2) are directly modelled by physical quantum states. The first point here is necessary for achieving quantum advantage <cit.> (though this does not per se guarantee better learning performance), while the second point means that the feature space is directly tunable in an efficient manner, unlike useful feature spaces in the traditional setting. Using an MKL strategy like Eq. (<ref>) creates a weighted concatenation of feature vectors, which leads to an overall kernel that interpolates between component kernels (Fig. <ref>). Adding a classical kernel to a complicated (and often periodic) quantum kernel also introduces a natural means of trainable regularization (Fig. <ref>), which can improve model performance beyond the training set. This effect is adjustable and could be eliminated entirely if the classical kernel is indeed superfluous. On the other hand, the optimal kernel weights could reveal that the quantum kernel is actually ineffective for the problem at hand.

Examples of trained SVM decision functions from the IQP (Tab. <ref>) and Linear kernels (Tab. <ref>). Top panels show each kernel individually and the bottom panel shows their linear combination, with weights $\gamma_1 = 0.4$ and $\gamma_2 = 0.6$ for the IQP and Linear kernels, respectively. The pictured training set represents a typical ($d=2$) instance of the datasets considered in this study. All horizontal and vertical axes range from 0 to $2\pi$.

§.§.§ Kernel Training and MKL Optimization

The process of training the kernel parameters ($\bm{\theta}$) typically represents a separate consideration with respect to optimizing the kernel combination weights ($\bm{\gamma}$). In this work, kernel parameter training relies on a stochastic, gradient-based method <cit.>, while weights optimization involves solving a convex quadratic problem based on the EasyMKL algorithm <cit.>. The QCC-net introduced in Sec. <ref> combines these processes in a fully differentiable way, with the MKL objective substituted as the training loss. As we shall demonstrate in Sec. <ref>, parameter training is necessary in some cases for the MKL algorithm to distinguish component kernels.

§ METHODOLOGY

§.§ Design

To explore c.c., q.c., and q.q. MKL methods side by side, we considered binary classification tasks using all pairwise combinations from a set of three classical (Tab. <ref>) and three quantum (Tab. <ref>) kernels. To quantify classification performance and characterize the kernels, we used the metrics in Tab. <ref>. Here, the base metrics $(T/F)P$ and $(T/F)N$ correspond to the true/false positive and true/false negative counts, respectively. These are used to compute the classification metrics in the lower half of Tab. <ref> according to the corresponding definition.

metric     definition

TP     $\#\{\bar{\bm{x}}\,\,\text{s.t.}\,\,y_{\text{pred}}(\bar{\bm{x}}) = \bar{y} = 1\}$

TN     $\#\{\bar{\bm{x}}\,\,\text{s.t.}\,\,y_{\text{pred}}(\bar{\bm{x}}) = \bar{y} = -1\}$

FP     $\#\{\bar{\bm{x}}\,\,\text{s.t.}\,\,y_{\text{pred}}(\bar{\bm{x}}) = 1, \bar{y} = -1\}$

FN     $\#\{\bar{\bm{x}}\,\,\text{s.t.}\,\,y_{\text{pred}}(\bar{\bm{x}}) = -1, \bar{y} = 1\}$

Accuracy     $\frac{TP + TN}{TP + TN + FP + FN}$

AUCROC     area under $(\frac{TP}{TP + FN}, \frac{FP}{FP + TN})\Big|_t$

Margin     $\min\{||\bm{\Phi}_{\bm{\theta}, \bm{\gamma}}(\bm{x}) - \bm{\Phi}_{\bm{\theta}, \bm{\gamma}}(\bm{x}^{\prime})||\,\,\text{s.t.}\,\,y \neq y^\prime\}$

Spectral Ratio     $\sum[K_{\bm{\theta},\bm{\gamma}}]_{ii} / \sqrt{\sum[K_{\bm{\theta},\bm{\gamma}}]_{ij}^2}$

Base metrics (top four), classification metrics (accuracy, AUCROC), and kernel metrics (margin, spectral ratio)  used for combined comparisons in Sec. <ref>. Classification metrics are computed from testing outcomes and kernel metrics are computed from training outcomes.

The reported accuracy is the number of correct predictions divided by total number of predictions. The area under the receiver operator characteristic (AUCROC) is computed across true/false positive rates over decreasing thresholds, $t$. These thresholds are obtained from $y_\text{pred}(\bar{\bm{x}})$ and $\bar{y}$ for all $\bar{\bm{x}}$ in the testing set $\bar{X} = X \setminus \hat{X}$, using the  function from <cit.>.

The final two metrics in Tab. <ref>, the margin and spectral ratio, are computed on the Gram matrix that represents the “kernelized” training data. These metrics characterize the kernel function and its embedding, unlike the accuracy and AUCROC, which are measures of classification performance. The margin is the smallest distance between points in feature space ($\mathcal{F}$) which belong to different classes. It is the maximization target for the SVM algorithm <cit.> and has a fixed upper bound for a given dataset $\hat{X}$ and feature map $\bm{\Phi}$.

The spectral ratio is the sum of the diagonal elements of $K_{\bm{\theta},\bm{\gamma}}$ divided by its Frobenius (or $L_{2,2}$) norm. For bounded kernels (i.e., RBF and all quantum kernels) the diagonal sum is always equal to $\#\{\hat{X}\}$ (i.e. the size of $\hat{X}$), because these kernels evaluate to unity on identical input pairs. Since off-diagonal terms correspond to non-identical input pairs, the maximum spectral ratio of 1 is obtained with a kernel

\begin{align}
k_{\bm{\theta},\bm{\gamma}}(\bm{x},\bm{x}^\prime) = \delta_{\bm{x},\bm{x}^\prime},
\end{align}

which is a Kronecker delta function on the domain $\bm{x},\bm{x}^\prime \in \hat{X}$. This limit corresponds to poor classification in general, as only identical points are considered “similar”. Conversely, the minimum spectral ratio $1/\#\{\hat{X}\}$ is obtained with the constant function

\begin{align}
k_{\bm{\theta},\bm{\gamma}}(\bm{x},\bm{x}^\prime) = 1,
\end{align}

which considers every pair of points to be maximally similar. Therefore, a reasonable spectral ratio corresponds to some value between these extremes.

For combinations containing unbounded kernels (Linear, Polynomial), we normalize the component Gram matrix (Eq. (<ref>)) before the sums over $K_{\bm{\theta},\bm{\gamma}}$ (Tab. <ref>) are computed. Normalization additionally helps to stabilize the kernel weighting algorithm (Sec. <ref>) in general. However, this results in $\sum [K_{\bm{\theta}}^{(r)}]_{ii} < \#\{\hat{X}\}$, which (unless $\gamma_r$ is zero) unnaturally lowers of the combined kernel's spectral ratio. Direct comparisons of spectral ratios are therefore not reliable between combinations that contain only bounded kernels versus those containing one or more unbounded kernels. Comparisons within these two groups, however, are fully justified.

§.§.§ Classical and Quantum Kernel Selection

Among the classical set, the Linear kernel corresponds to a feature map whose feature space is (trivially) the native data space ($\mathbb{R}^d$). Any prediction function learned with the Linear kernel is therefore strictly linear itself. We also consider a cubic Polynomial kernel and the radial basis function (RBF) kernel as more “powerful” examples of common classical kernels. Both of the latter two are parametric and produce non-trivial feature maps. Any feature space of the RBF kernel is in fact infinite-dimensional <cit.>, and the kernel itself is known to be universal <cit.>.

On the quantum side, we construct a QEK from each embedding unitary in Tab. <ref> as per Eqs. (<ref>,<ref>). The shorthand

\begin{align}
\mathcal{R}_{\text{A}} (\bm{z}) = \exp\Bigg(-\frac{i}{2} \sum_{p=1}^{m} z_p \text{A}_p \Bigg)
\label{eq:R_a}
\end{align}

is introduced here to represent single-qubit $\text{A}$-rotations, where $\text{A}_p$ are Pauli $X$, $Y$, or $Z$ gates acting on qubit $p$ of an $N$-qubit circuit; taking the rotation angle from component $p \leq m \leq N$ of the input vector $\bm{z} \in \mathbb{R}^m$.

The simplest quantum kernel that we consider is RX (Tab. <ref>), which encodes each component of $\bm{x}\in\mathbb{R}^d$ onto one of $N=d$ qubits using $\mathcal{R}_{\text{X}}(\bm{x})$ as the embedding unitary. The IQP <cit.> kernel is defined by the ansatz

\begin{align}
V(\bm{x}) = \Bigg\{\exp\Bigg(-\frac{i}{2}\sum_{p\neq q} x_p x_q Z_p Z_q \Bigg)\Bigg\} \mathcal{R}_\text{Z}(\bm{x}),
\label{eq:IQP-equation}
\end{align}

which uses of $\mathcal{R}_\text{Z}$ rotations followed by two-qubit gates on all pairs of qubits to encode data. Lastly, the variational ansatz,

\begin{align}
W(\bm{\theta}) = \mathcal{R}_\text{Y}(\bm{\theta}) \Bigg\{\exp\Bigg(-\frac{i}{2}\sum_{p\neq q} \theta_{pq} Z_p Z_q \Bigg)\Bigg\}.
\label{eq:qaoa-w}
\end{align}

defines the QAOA <cit.> kernel. While similar to IQP, the QAOA kernel features parametric transformations after a single set of encoding gates. Regarding all three embedding circuits, we utilize the minimum, single layer ansatz in every case, as shown in Tab. <ref>.

§.§.§ QCC-net Optimization for MKL

Given $M$ training samples from $\hat{X} \subset X \subset \mathbb{R}^d$ and their labels as a matrix $\hat{Y} = \text{diag}(y_1, ..., y_M)$, where $y_i \in \{-1, 1\}$, the combination weights are determined to maximize the total distance (in feature space) between positive and negative samples. Following <cit.>, this problem is formulated as

\begin{align}
\max_{||\bm{\gamma}|| = 1}\, \min_{\bm{\phi}}\, (1 - \lambda) \bm{\gamma}^T \bm{d}_{\bm{\theta}}(\bm{\phi}) + \lambda ||\bm{\phi}||^2_2,
\label{eq:easymkl-minmax-problem}
\end{align}

where the vector of distances, $\bm{d}_{\bm{\theta}}(\bm{\phi}) \in \mathbb{R}^R$, has components

\begin{align}
d_{\bm{\theta}}^{(r)}(\bm{\phi}) = \bm{\phi}^T\hat{Y} \hat{K}_{\bm{\theta}}^{(r)} \hat{Y} \bm{\phi},
\label{eq:separation-component}
\end{align}

and the variable $\bm{\phi} \in \mathbb{R}^M$ is subject to

\begin{align}
\sum_{i|y(\bm{x}) = 1} \bm{\phi}_i = \sum_{i|y(\bm{x}) = -1} \bm{\phi}_i = 1.
\label{eq:phi-constraint}
\end{align}

Considering $0 \leq \lambda < 1$, a solution $\bm{\gamma}^\star$ parallel to $\bm{d}_{\bm{\theta}}(\bm{\phi})$ is evident for the outer maximization problem:

\begin{align}
\bm{\gamma}^\star = \frac{\bm{d}_{\bm{\theta}}(\bm{\phi})}{||\bm{d}_{\bm{\theta}}(\bm{\phi})||_2}.
\label{eq:gamma-solution}
\end{align}

This reduces Eq. (<ref>) to the (convex) minimization problem,

\begin{align}
\mathcal{L}_{\bm{\phi}_{\text{min}}}(\bm{\theta}) = \min_{\bm{\phi}}\, (1 - \lambda) || \bm{d}_{\bm{\theta}} (\bm{\phi}) ||_2 + \lambda ||\bm{\phi}||^2_2,
\label{eq:easymkl-problem}
\end{align}

whereby the solution $\bm{\phi}_{\text{min}}$ leads to the optimal kernel weights via Eq. (<ref>), given fixed values of the kernel parameters $\bm{\theta}$.

We implemented the above algorithm, which describes the process of determining the kernel weights ($\bm{\gamma}^\star$), using the convex program solver CVXPY <cit.> together with the default splitting conic solver (SCS) <cit.>. In order to integrate optimization of the kernel parameters ($\bm{\theta}$), we expressed the problem in Eq. (<ref>) as a differentiable neural network layer using the CVXPY extension  <cit.>, which uses  <cit.> internally. This allowed the gradients

\begin{align}
\frac{\partial \mathcal{L}_{\bm{\phi}_{\text{min}}}(\bm{\theta})}{\partial \bm{\theta}} := \Bigg(\frac{\partial \mathcal{L}_{\bm{\phi}_{\text{min}}}(\bm{\theta})}{\partial \theta_0},\,...\,,\frac{\partial \mathcal{L}_{\bm{\phi}_{\text{min}}}(\bm{\theta})}{\partial \theta_{N-1}} \Bigg)
\label{eq:qcc-net-gradient}
\end{align}

to be obtained at intermediate steps in the overall computation. Access to these gradients, in turn, allows a gradient-based optimizer to be used for training the kernel parameters to maximize $\mathcal{L}_{\bm{\phi}_{\text{min}}}(\bm{\theta})$ with respect to $\bm{\theta}$. (We utilized the Adam optimizer <cit.> in this work.) As illustrated in FIG. 1 of Ref. <cit.>, the complete QCC-net algorithm proceeds as follows:

* Starting with $R$ base kernels, compute $\hat{K}_{\bm{\theta},\bm{\gamma}}$ assuming balanced $\bm{\gamma}$ (i.e. all components equal to $1/R$), and using random/default initial kernel parameters $\bm{\theta}_l = \bm{\theta}$, where $l = 0$ initially.

* Solve the cone problem in Eq. (<ref>) using CVXPY+SCS to determine $\bm{\phi}_{\text{min}}$ at fixed $\bm{\theta}_l$. If the optimizer's termination criteria are met, advance to (iv). Else, continue to (iii).

* Use the gradients in Eq. (<ref>) to update the kernel the parameters $\bm{\theta}_{l} \rightarrow \bm{\theta}_{l+1}$, maximizing loss function $\mathcal{L}_{\bm{\phi}_{\text{min}}}(\bm{\theta})$ with respect to $\bm{\theta}$, then return to step (i) with $\bm{\theta}_{l} := \bm{\theta}_{l+1}$.

* Set $\bm{\theta}^\star = \bm{\theta}_{l}$, obtain $\bm{\gamma}^\star$ from $\bm{\phi}_{\text{min}}$, and compute the final kernel matrix.

The QML framework Pennylane <cit.> was used to implement and simulate quantum circuits with a PyTorch- <cit.> compatible interface for gradient calculations and GPU acceleration. We also used Covalent (see [Covalent: <https://www.covalent.xyz>]) to facilitate distributed computations.

§.§ Dataset Preparation and Preprocessing

We utilized the  software package <cit.> to synthesize instances of generic, single-label datasets for binary classification [: <https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html>] (based the method in <cit.>). Each dataset instance consisted of 100 samples ($\bm{x} \in \mathbb{R}^d$), with $d$ informative features, split 50:50 into training and testing subsets. Samples belonging to each class were distributed evenly between two $d$-variate Gaussian clusters with randomly correlated features; for $d$ in our experiments ranging from 2 and 13. Our initial investigations determined that all kernel combinations performed very well on linearly separable datasets, as expected. Therefore, to reveal meaningful differences between kernel combinations, we used a “class separation” of 1.0 (see [1]), ensuring with high probability that the two clusters were not linearly separable. All features were scaled (using [: <https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html>]) into the interval $[0, 2\pi]$ in preprocessing. We also utilized an SVM implementation from  [: <https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html>].

A total 120 instances of this classification problem were computed for each of the three result types (Tab. <ref>), across $d = 2,3,...,13$ features, with 10 repetitions per value of $d$. The Supplementary Information [See Supplementary Information [publisher url] for examples of two-dimensional datasets.] contains representative examples for the easily visualized $d=2$ case, with one training set also visible in Fig. <ref>.

In the section that follows, performance metrics based on the outcomes of these experiments are used to demonstrate the QCC-net optimization approach. We also identify how kernel parameter optimization influences the selection of kernel weights in pairwise combinations of q.c., q.q., and c.c. kernels, and conclude with a brief discussion to reconcile the prior results.

result type     $\bm{\theta}$     $\bm{\gamma}$

(i)     default/random     uniform

(ii)     default/random     optimized

(iii)     trained     optimized

The three types of results considered in this section. Types are distinguished by the inclusion of kernel parameter ($\bm{\theta}$) training and/or weights ($\bm{\gamma}$) optimization.

§ RESULTS AND DISCUSSION

As shown in Tab. <ref>, the results considered in this section are subdivided into three distinct types: (i) non-optimized, (ii) semi-optimized, and (iii) fully optimized. Note that only type (iii) results include kernel parameter training. The other two types of results feature default (Tab. <ref>) or randomly chosen (Tab. <ref>) parameter values. For type (i) and type (ii) results, default kernel parameters are kept constant throughout, whereas random kernel parameters are uniquely generated for each instance of the problem.

Median kernel metrics over 120 instances, across datasets with $d=2$ to $13$ features, for every unique kernel combination. Colour scales are normalized among adjacent pairs in columns (i) and (iii), which are labelled according to the type of results contained therein. The final column contains the difference, (i) subtracted from (iii). Entries in the difference column are true values rounded to 2-digit precision.

§.§ Performance Comparisons

With the chosen MKL strategy, a combination of identical kernels is equivalent to that specific kernel individually, due to the symmetry present in Eq. (<ref>). That is, the base kernel is recovered when $k^{(r)}_{\bm{\theta}}(\bm{x}, \bm{x}^{\prime}) \equiv k^{(1)}_{\bm{\theta}}(\bm{x}, \bm{x}^{\prime})$ for all $r$:

\begin{align}
k_{\bm{\theta}, \bm{\gamma}}(\bm{x}, \bm{x}^{\prime}) &:= k^{(1)}_{\bm{\theta}}(\bm{x}, \bm{x}^{\prime})\sum_{r=1}^R \bm{\gamma}_r = k^{(1)}_{\bm{\theta}}(\bm{x}, \bm{x}^{\prime})
\label{eq:kernel-combination-collapse}
\end{align}

Hence, the diagonal entries in Fig. <ref> indicate the given metric for the lone base kernel. Equivalent values are expected for diagonal entries in columns (i) and (iii) if the base kernel is also non-parametric (Linear, RX, and IQP). In all other cases, however, the kernel combination (or a parametric base kernel) stands to benefit from parameter $(\bm{\theta})$ and/or kernel weights $(\bm{\gamma})$ optimization.

To provide an overall measure of optimization efficacy, we list the total differences between non-optimized (i) and fully optimized (iii) results in Tab. <ref>. The table is obtained by adding the differences in the final column of Fig. <ref> according to the following scheme: For a given metric, values for kernel combinations $A$-$B$ contribute to totals for both kernel $A$ and kernel $B$, whereas $A$-$A$ combinations contribute only once to the total for kernel $A$.

§.§.§ Accuracy

Looking at accuracy outcomes, the RBF- and RX- containing kernels exhibit the largest and second-largest scores, respectively, both with and without optimization. Meanwhile, the RBF- and QAOA- containing kernels exhibit the largest and second-largest improvements in accuracy, as compared to the other combinations. The greatest overall improvement in accuracy corresponds to the IQP-RBF kernel combination. Prior to optimization, and indeed after, the IQP-containing kernels exhibit the lowest overall accuracy, considering score totals in the same way as above. This improvement can be attributed to regularization via inclusion of the smooth RBF kernel together with IQP. This point was exemplified by Fig. <ref>, earlier in this text. Apart from RBF and QAOA, the Polynomial kernel (Tab. <ref>) is the only other parametric kernel. Compared to the related (and non-parametric) Linear kernel, the Polynomial kernel does not exhibit a significant difference in accuracy scores for the classification problem at hand, even after optimization.

Accuracy     AUCROC     Margin     Spectral Ratio

Linear     0.18     0.02     0.10     0.75

Polynomial     0.18     0.02     0.10     0.22

RBF     0.55     0.26     0.02     -1.76

RX     0.10     0.06     0.10     -0.22

IQP     0.14     -0.02     0.10     -0.34

QAOA     0.26     0.14     -0.06     -1.92

Total difference by metric, over all unique combinations containing each base kernel. Every entry is a sum of six elements from the difference grid for the given metric in final column of Fig. <ref>. Boldface entries indicate the kernel(s) showing the greatest total difference by magnitude, for each metric. Improvements in the accuracy, AUCROC, and margin correspond to positive values (i.e. a total increase). However, a total increase or decrease can both indicate an improvement in the spectral ratio, depending on initial values in the first column of Fig. <ref>.

§.§.§ AUCROC

Trends in the AUCROC values are broadly comparable to trends in the accuracy, as evident in the first two rows of Fig. <ref>. Here, again, the RBF- and QAOA-containing kernels show the largest overall improvement. Individually, the largest improvements are for the QAOA base kernel and the IQP-RBF kernel combination. A significant decrease in the AUCROC is seen here for the IQP-Linear and IQP-Polynomial kernel combinations, despite a net-zero change in the accuracy, when comparing (i) and (iii). Noting the low median AUCROC of the IQP base kernel (0.71), this suggests a stronger preference for the IQP kernel versus both the Linear and Polynomial kernels vis a vis the optimization target Eq. (<ref>), which is not necessarily aligned with the AUCROC metric. The same line of reasoning applies for the Linear and Polynomial kernels in combination with QAOA. Conversely, a stronger preference RBF in the IQP-RBF combination (in addition to optimizing $\theta_2$ (Tab. <ref>)) leads to a large increase (+0.08) in the AUCROC over the balanced IQP-RBF combination. It is also noteworthy that, despite a lower score overall, the base QAOA kernel exhibits a comparatively strong improvement in the median AUCROC with parameter optimization.

§.§.§ Margin

Outcomes for the margin metric also show the largest differences among combinations that pair low- and high-scoring base kernels. Excluding combinations with the two lowest-scoring base kernels (Linear, Polynomial), a slight overall decrease in the margin is evident among the remaining kernel combinations. Moreover, negative changes are associated with the RBF- and QAOA-containing combinations, and most strongly with the lone base kernels in either case. This suggests that parameter optimization tends to simultaneously reduce the margin, despite generally improving accuracy and AUCROC.

§.§.§ Spectral Ratio

We observe the largest differences overall in comparing type (i) and type (iii) results for the spectral ratio. Selection against the Linear and Polynomial kernels during $\bm{\gamma}$ optimization leads to a comparative increase of the spectral ratios here for related combinations, because the matching base kernels exhibit very low spectral ratios in general.

Density of optimized kernel weights ($\gamma_1^\star + \gamma_2^\star = 1$) with and without additional optimization of kernel parameters ($\bm{\theta}$), for $d \in \{2,6,13\}$ features. For a given $d$, distributions skewing right indicate a preference for the kernel on the right-hand side ($\gamma_2^\star > \gamma_1^\star$), and vice versa when skewing left ($\gamma_1^\star > \gamma_2^\star$). Outlined distributions correspond to fully optimized results (type (iii)) and filled distributions correspond to semi-optimized results (type (ii)). Alphabetic labels are provided for convenience.

Excluding Linear- and Polynomial-containing entries, all combinations among the RBF, QAOA, RX, and IQP kernels exhibit very high spectral ratios in the “no optimization” column, (i). Without tunable parameters, there is little expectation of any change for the RX and IQP base kernels, as well as the RX-IQP combination, because adjusting the kernel weights only interpolates between two already-large values. Indeed, the spectral ratio for RX- and IQP- containing kernels does not change significantly from column (i) to (iii). For combinations that contain the parametric RBF or QAOA kernels, on the other hand, a strong decrease in the spectral ratio toward more moderate values ($\approx 0.5$) is clearly observed. Moreover, the existence and severity of this trend for the lone RBF and QAOA kernels (third and final diagonal entries, respectively, in the bottom right grid of Fig. <ref>) confirms that parameter optimization is effective at balancing the spectral ratio.

§.§.§ Summary

Overall, these results show that kernel combinations, especially those involving parametric kernels, benefit significantly from the QCC-net optimization procedure. The RBF- and RX-containing kernels demonstrated the highest accuracy and AUCROC scores, with RBF- and QAOA-containing kernels exhibiting the largest improvements for these metrics. Specifically, the IQP-RBF kernel combination displayed the greatest overall improvement in accuracy, which we attribute to the regularization effect of the smoother RBF kernel. Conversely, the IQP, Linear, and Polynomial kernels showed lower overall performance. The results also highlight that parameter optimization can effectively balance the spectral ratio, which is particularly important for parametric kernels like RBF and QAOA.

§.§ Impact of Parameter Training on MKL

In order to separate the effects of kernel parameter training and kernel weights optimization, we proceed with a comparison between type (ii) and type (iii) results (Tab. <ref>). Recall that the latter uses both trained $\bm{\theta}^\star$ and optimized $\bm{\gamma}^\star$, whereas the former uses random or default $\bm{\theta}$ and optimized $\bm{\gamma}^\star$. We shall outline the comparison here in terms of the optimal kernel weights determined in either case. As illustrated in Fig. <ref>, we compare the distributions of $\bm{\gamma}^\star$ for data with $d=2$, $6$, and $13$ features to distinguish, also, any trends in $\bm{\gamma}^\star$ that depend on $d$.

§.§.§ Non-Parametric Combinations

To start, we confirm that type (ii) and type (iii) results are indeed very similar for kernel combinations that contain no parametric base kernels, namely those on rows (a) to (c). We can therefore discuss this subset of outcomes without distinguishing between the two types of results. A clear preference for the RX kernel over the Linear kernel is seen at all three values of $d$ on row (a) of Fig. <ref>. The same can be said for IQP, regarding the IQP-Linear combination on row (b). Neither result is surprising, since the Linear kernel is obviously not suited to the non-linearly-separable classification problem under consideration (see, for example, Fig. <ref>). Nonetheless, results from the previous section (Fig. <ref>) indicate that the RX-Linear and IQP-Linear combinations do in fact outperform the lone quantum kernels in either case, which may explain why the Linear kernel is not eliminated entirely (i.e. $\gamma_2 \neq 0$). On row (c), preference for the RX kernel is strong at $d=2$ for this non-parametric q.q. combination. However, a gradual shift toward balanced weights, and a narrowing of the weights distribution, is observed with increasing $d$. For the largest number of features, $d=13$, the RX and IQP kernels appear equally effective.

§.§.§ Parametric Kernel Combinations

In contrast to the above, results for the c.c. Polynomial-Linear combination, (d), show approximately balanced weights across all $d$, apart from a small proportion of the fully optimized outcomes (iii) that skew very strongly toward the Polynomial kernel. This suggests rare instances in which parameter training is highly successful for the Polynomial kernel. Results for the Linear-RBF combination immediately below, on row (e), may further support this: While the RBF kernel largely dominates the combination, a comparable proportion of the outcomes at the $d=13$ strongly favour the fully optimized Polynomial kernel. This is not to suggest, however, that the trained Polynomial kernel is particularly effective for the problem at hand, since its combination with the RX and IQP kernels, on rows (f) and (g), exhibits minimal difference between result types (ii) and (iii).

The QAOA-Polynomial combination on row (h) corresponds to the first unambiguous result as far as differentiating trained and random-parameter outcomes. Preference for the QAOA kernel over the Linear kernel is clear with and without parameter training, although much narrower distributions are observed for the former (type (iii)), especially at lower $d$. A similar trend is seen for the QAOA-Linear combination on row (j). With the Linear-RBF combination in between the prior two results, on row (i), the trend is again similar, except the semi-optimized distributions are more narrow here to start. Evidently, the initial value of the RBF scaling parameter, $\theta_0 = 1$, represents a reasonable choice for data with features scaled to $[0, 2\pi]$. No such choice can be made for the QAOA kernel, however, which is parametrized via periodic quantum gates. Thus, for the type (ii) results on rows (h) and (j), random-parameter outcomes for QAOA-containing combinations are more broadly distributed compared to the Linear-RBF combination.

§.§.§ Quantum-RBF and QAOA-Quantum Kernel Combinations

Rows (k) to (q) correspond to pairs among RBF and the three quantum kernels (Tab. <ref>). For the RX-RBF combination, (k), we note that $\bm{\gamma}^\star$ remains approximately balanced across $d$ without parameter optimization. For type (iii) results, however, the RBF kernel is preferred over RX, and more so with increasing $d$. Next, for IQP-RBF kernel combination on row (m), we note that $\bm{\gamma}^\star$ skews slightly toward RBF for both types of results at the two lower values of $d$. This is more severe for the fully optimized case (iii), in view of the $d=6$ result. At $d=13$, the semi-optimized case (ii) does not distinguish at all between IQP and RBF. (The distribution in Fig. <ref> row (m) is very narrow and obscured by the vertical grid line at $\gamma_1^\star = \gamma_2^\star = 0.5$.) In the fully optimized case, however, a preference for the RBF kernel is clearly visible. Here, and for the RX-RBF combination, (k), parameter training enhances the selectivity for weights tuning, especially at higher dimensions.

The final rows of Fig. <ref>, (n) to (q), contain the QAOA kernel in combination with RBF, RX, and IQP, respectively. Regarding the semi-optimized results for these three rows, we note that, while the random-parameter QAOA kernel (ii) is selected against for data with $d=2$ features, this trend gradually disappears for larger $d$. Indeed, at $d=13$, rows (n) to (q) illustrate virtually no preference between the random $\bm{\theta}$ QAOA kernel and the RBF, RX, or IQP kernels. The opposite trend is observed, however, for the fully optimized results on these combinations. That is, preference for the fully optimized QAOA kernel increases from $d=2$ to $13$. This result, too, supports the concluding claim in the previous paragraph. Operating upon data with $d=13$ features, the weights tuning algorithm (Sec. <ref>) does not discriminate between component kernels in neither q.q., nor RBF-containing q.c. combinations with the base kernels considered in this work.

§.§.§ Summary

These findings reveal that the RX, IQP, and RBF kernels surpass the Linear and Polynomial kernels for the studied classification problems. Furthermore, the fully optimized QAOA kernel is more strongly preferred as data dimensionality grows, suggesting that parameter training serves to enable kernel weights tuning in high-dimensional settings. The results also show limited discrimination between quantum and classical kernels in q.q. and RBF-containing q.c. combinations, suggesting that kernel and optimization choices should be tailored to the problem and dataset specifics.

Density contours illustrating distributions of performance metrics for QAOA-containing q.c. combinations for all three result types, using (i) random kernel parameters and equal kernel weights ($\bm{\theta}, \bm{\gamma}$); (ii) random kernel parameters and optimized weights ($\bm{\theta}, \bm{\gamma}^\star$); and (iii) trained kernel parameters and optimized kernel weights ($\bm{\theta}^\star, \bm{\gamma}^\star$). The dashed black lines in all subplots indicate $x=y$ and are provided to help orient to eye. Distributions include results for all feature sizes, $d=2$ to $13$. Note that matching results are expected for cases (i) and (ii) along the top row of either set of subplots, since these correspond to the lone QAOA base kernel.

§.§ Insights on QC Combinations

Generalizing upon results from the previous two sections, the RBF-containing q.c. combinations exhibit the best performance overall and the greatest improvements in performance metrics between result types (i) and (iii). When paired with the quantum kernels from Tab. <ref>, MKL optimization weighs the untrained RBF kernel equally or more heavily than all three quantum counterparts, including the untrained (random parameter) QAOA kernel. Based on the median metrics in Fig. <ref>, we can confirm that training $\theta_2$ improves the lone RBF kernel's performance. The fact that the trained QAOA kernel is weighted more heavily than the trained RBF kernel (Fig. <ref>, row (n)) then suggests an important and competitive contribution from QAOA. However, even in considering only $d=13$ feature data, median metrics for case (iii) closely resemble those of results across all $d$ in Fig. <ref>, where the lone RBF kernel consistently out-performs the QAOA-RBF combination.

One reason for this could be the shallow depth of the minimal ansätze utilized for QAOA and the other quantum kernels (Tab. <ref>). In the context of its original application <cit.>, multiple repetitions of the QAOA circuit are known to produce higher quality approximations to solutions for combinatorial problems. It may be the case that a similar relationship exists for QEKs that also utilize multiple, parametric QAOA layers for classification with kernel methods <cit.>. The related, though non-parametric, IQP kernel could also benefit from repetitions of the embedding circuit—such repetitions are, in fact, a requirement for rigorous classical “hardness” in this case <cit.>.

The observation of better performance metrics for the RX-containing q.c. combinations, in comparison to the QAOA- and IQP-containing combinations, is not necessarily contradictory to the above hypothesis. Considering that the RX kernel is effectively a quantum implementation of a classical cosine kernel <cit.>, and acknowledging the relative simplicity of the synthetic datasets under consideration, the utility of more complex QEKs may yet be demonstrable on higher-dimensional or specifically structured data, as in Refs. <cit.>.

In the present context, QAOA remains the best representative of a trainable and practical family of quantum kernels. The distributions of performance metrics for q.c. combinations that contain this kernel, and those of the base QAOA kernel, are illustrated in Fig. <ref>. Here, as also revealed by the median metrics in Fig. <ref>, the overall accuracy and AUCROC are only slightly improved, and the variance of outcomes only slightly reduced, in comparing result types (i) and (iii). Still more subtle is the accuracy and AUCROC difference between types (i) and (ii). The latter is unsurprising in view of Fig. <ref>, since without parameter training the optimized weights are seen to converge to the balanced (default) vector, $\bm{\gamma}^\star \rightarrow \bm{\gamma} = [0.5, 0.5]^T$, as $d$ increases.

Regarding the margin and spectral ratio outcomes for this subset of q.c. combinations (Fig. <ref> (b)), the type (ii) results suggest that weights optimization without parameter training may actually be detrimental for the QAOA-Linear and QAOA-Polynomial combinations, whereas the QAOA-RBF distribution is again not significantly altered. With the inclusion of parameter training in case (iii), the distributions for the QAOA kernel and all QAOA-containing q.c. combinations are seen to change more drastically. Specifically, the margin distribution is widened while the spectral ratios are reduced for type (iii) results across every row in Fig. <ref> (b). The type (iii) results here also converge to similar distributions for all four combinations considered, with the QAOA-RBF combination producing the best outcome overall.

§ CONCLUSION

In this work, we used modern software tools and a novel optimization procedure inspired by classical machine learning to systematically explore the utility of c.c., q.c., and q.q. kernel combinations in binary classification problems, over datasets with $d=2$ to 13 features.  Considering three quantum kernels and three canonical classical kernels in a comparative setting, we found that only the most complex and parametric quantum kernel (QAOA) attains higher a optimum weight in pairwise combination with the most performant classical kernel (RBF). Conversely, classification performance was not found to differ significantly between q.c. combinations featuring simpler classical kernels (Linear, Polynomial) in comparison to the lone quantum kernel. Regarding use of the QCC-net for training the kernel parameters and optimizing the combination weights, we found the kernel weighting step to be indecisive without the former training step when optimizing q.c. combinations that contain the parametric QAOA kernel. The simpler RX kernel and its q.c. combinations exhibited the best performance metrics among the quantum and quantum-containing kernels considered. More broadly, classical RBF kernel and its combinations performed the best overall.

We are able to recommend a number of directions for future work aimed at expanding the scope of this study and/or identifying an empirical advantage with quantum kernels. For example, the use of multi-layer embedding circuits for quantum kernels may prove more effective on higher-dimensional data, based on the trends we observed for QAOA-containing kernel combinations. Additionally, the EasyMKL algorithm is well suited for combining a far greater number of kernels and can therefore be used to explore the q.q, c.c., and q.c paradigms beyond pairwise combinations. On the other hand, alternate (non-linear) MKL strategies are also a worthwhile prospect for future work, assuming that combination weights can be computed efficiently. Finally, experiments on datasets with a different, less generic structure may provide the more promising results for q.q. and q.c. combinations, especially if the number of features is large.

§ ACKNOWLEDGEMENTS

Partial funding for this work was provided by the Mitacs Accelerate program.

§ SUPPLEMENTARY INFORMATION:
QUANTUM-CLASSICAL MULTIPLE KERNEL LEARNING

Ten examples of $d=2$ dimensional classification datasets corresponding to Sec. <ref>. Each pair of plots shows the equally-split training and testing subsets (left (a) and right (b), respectively). All horizontal and vertical axes range from 0 to $2\pi$. Square scatter points (red) belong one class and round scatter points (blue) belong to the other.