# Can Programming Languages Boost Each Other
via Instruction Tuning?

Daoguang Zan$\mathcal{y}$11footnotemark: 1 Ailun Yu$\mathcal{x}$ Bo
Shen$\mathcal{z}$ Jiaxin Zhang$\mathcal{z}$ Taihong Chen$\mathcal{z}$ Bing
Geng$\mathcal{z}$ Bei Chen¶
Jichuan Ji$\mathcal{z}$ Yafen Yao$\mathcal{z}$ Yongji Wang$\mathcal{y}$
Qianxiang Wang$\mathcal{z}$
$\mathcal{y}$Institute of Software, Chinese Academy of Science
$\mathcal{x}$Peking University
$\mathcal{z}$Huawei Co., Ltd.
¶Independent Researcher
<EMAIL_ADDRESS><EMAIL_ADDRESS>The first two authors contributed
equally to this work.

###### Abstract

When human programmers have mastered a programming language, it would be
easier when they learn a new programming language. In this report, we focus on
exploring whether programming languages can boost each other during the
instruction fine-tuning phase of code large language models. We conduct
extensive experiments of $8$ popular programming languages (Python,
JavaScript, TypeScript, C, C++, Java, Go, HTML) on StarCoder. Results
demonstrate that programming languages can significantly improve each other.
For example, CodeM-Python $15$B trained on Python is able to increase Java by
an absolute $17.95$% pass$@1$ on HumanEval-X. More surprisingly, we found that
CodeM-HTML $7$B trained on the HTML corpus can improve Java by an absolute
$15.24$% pass$@1$. Our training data is released at
https://github.com/NL2Code/CodeM.

_Keywords_ Large Language Model $\cdot$ Code Generation $\cdot$ Programmer
Language $\cdot$ Instruction Tuning

##  1 Introduction

Code large language models (code LLMs) are blooming recently (Zan et al.,
2023). A lot of code LLMs are released in succession, e.g., Codex (Chen et
al., 2021), AlphaCode (Li et al., 2022), PaLM-Coder (Chowdhery et al., 2022),
CodeGen (Nijkamp et al., 2023), CodeGeeX (Zheng et al., 2023), StarCoder (Li
et al., 2023), and Code Llama (Rozière et al., 2023). Owing to their amazing
code generation performance, code LLMs have attracted considerable attention
from both academic and industrial circles. Recent works (Ouyang et al., 2022)
have witnessed the instruction tuning technique that can teach LLMs how to
follow instructions. In the realm of code generation, WizardCoder (Luo et al.,
2023) and PanGu-Coder2 (Shen et al., 2023) also adopt this technique to elicit
their code generation capabilities. Although some code LLMs, such as CodeGen-
Multi Nijkamp et al. (2023) and StarCoder-base Li et al. (2023), are trained
on corpora spanning multiple programming languages, the interplay among these
languages remains unexplored. In programming practice, once a human programmer
has mastered a programming language, it would be easier to learn a new one due
to the homogeneity between programming languages. Motivated by this, we would
like to explore whether different programming languages can boost each other
during instruction fine-tuning of code LLMs.

To explore this idea, we craft the training corpus for each of $8$ popular
programming languages (Python, JavaScript, TypeScript, C, C++, Java, Go,
HTML), where each language includes about $9$K programming exercises. We train
StarCoder $7$B using the instruction tuning technique on each programming
language corpus separately, and test the performance of each fine-tuned model
across every programming language. Our findings reveal that programming
languages can significantly boost each other. Meanwhile, we found that the
improvement margin of different programming languages to each other is related
to the language similarity between them. For example, CodeM-JavaScript 7B
trained on JavaScript data can yield an absolute $11.80$% pass$@1$ improvement
in TypeScript. More interestingly, CodeM-HTML 7B trained on the markup
language HTML also can achieve an absolute $15.24$% pass$@1$ improvement in
Java.

In a nutshell, our contributions can be listed as follows: (1) Our findings
suggest that programming languages can significantly boost each other during
code LLMs’ instruction fine-tuning phase. (2) We glean valuable insights on
the correlation between multiple programming languages, paving the way for
future research on code generation. (3) We will make our training data
publicly available.

##  2 Methodology

###  2.1 Crafting Training Corpus of Eight Programming Languages

We select $8$ popular programming languages and construct their training data
separately. Our selected languages include Python, JavaScript, TypeScript, C,
C++, Java, Go, and HTML, covering diverse types such as procedure-oriented,
object-oriented, script, and even markup languages. For each programming
language, we construct its training data containing about $9$K data pairs.
Each pair includes both an instruction describing the programming problem and
its corresponding response. One practical example of HTML is shown in Figure
1.

Based on these selected languages, we construct a series of monolingual
datasets. We start from the dataset of CodeAlpaca
20K111https://huggingface.co/datasets/sahil2801/CodeAlpaca-20k, and extract
those Python-related data to form our seed instruction set. Then for each
selected programming language, we evolve existent instructions in the seed
instruction set to get corresponding new ones by prompting OpenAI’s
GPT-3.5222https://platform.openai.com/docs/models/gpt-3-5. For all the
selected languages except HTML, we adopt an in-depth evolution (Xu et al.,
2023), by asking GPT-3.5 to rewrite the seed instruction (Python) into a more
complicated version relevant to the target language (Python, JavaScript,
TypeScript, C, C++, Java, or Go). However, for HTML, we adopt in-breadth
evolution to produce a brand-new HTML-related instruction, since HTML (markup
language) is too different from other languages (non-markup languages).

# Instruction Design a web page that displays a message when loaded. The
message should be "Hello, World". Use HTML code to achieve this. # Response
<!DOCTYPE html>   <title>Hello World</title>

Hello, World
   Figure 1: A HTML training example of our
crafted instruction-answer pairs.

###  2.2 Instruction Tuning

Code pre-trained models such as Codex (Chen et al., 2021) and StarCoder (Li et
al., 2023) store a wealth of code knowledge. However, these models only
support left-to-right code generation based on context, as they are trained
solely on plain code snippets. Of late, the instruction tuning techniques
(Ouyang et al., 2022; Luo et al., 2023; Shen et al., 2023) are proposed, which
can enhance the model’s capabilities of following instructions so as to enable
chat features. During instruction tuning, we train StarCoder using the prompt
in Figure 2 to obtain our CodeM. We use DeepSpeed to accelerate the training
of CodeM with fp16 enabled. Additionally, we set the batch size to $2$ per
GPU, the learning rate to $2$e-$5$ with a cosine annealing schedule, the
gradient accumulation steps to $4$, and the warmup steps to $30$. After
instruction tuning, we use the prompt in Figure 3 to do the inference on
downstream tasks across various programming languages. For inference, we adopt
the greedy decoding strategy for sampling. Given that CodeM is a chat-style
model, the responses it generates often contain elements beyond just codes,
which typically makes them non-executable. So, we extract the code snippets
from the generated response to evaluate the performance of code generation.

Below is an instruction that describes a task, paired with an input that
provides further context. Write a response that appropriately completes the
request. ### Instruction: {problem} ### Response: {response} Figure 2: Prompt
format of instruction tuning. {problem} and {response} refer to the
instruction and answer obtained in Section 2.1. Below is an instruction that
describes a task. Write a response that appropriately completes the request.
### Instruction: Finish the {language} code for this problem: {problem} ###
Response: {signature} Figure 3: Prompt format of inference. {language},
{problem}, and {signature} represent the downstream programming language, the
given programming problem, and the function header, respectively.

##  3 Experiments

###  3.1 Evaluation Setup

####  3.1.1 Benchmarks and Baselines

We use HumanEval-X (Zheng et al., 2023) to evaluate the multilingual abilities
of models in Python, JavaScript, C++, Java, and Go. HumanEval-X is crafted by
adapting HumanEval (Chen et al., 2021) (Python) to other programming
languages. Following the same approach as HumanEval-X, we also create two new
versions of HumanEval: HumanEval-C and HumanEval-TypeScript. Note that
HumanEval can not directly be adapted to markup languages such as HTML, so our
downstream evaluation languages do not include HTML.

The primary baseline for all language versions of CodeM is their base model
StarCoder. We analyze whether CodeM trained on language A can improve language
B, in which case the baselines are CodeM directly trained on language B.

####  3.1.2 Metrics

We adopt pass$@1$ as our metric to evaluate all the models. Each model
generates one answer using the greedy decoding strategy for each programming
task, and the answer would be executed upon the given test cases. Only when
all the test cases are passed, the programming task can be considered solved
with the generated code. In this setting, pass$@1$ can be formulated as
$\frac{|P_{c}|}{|P|}$, where $|P|$ denotes the total number of programming
tasks in HumanEval and $|P_{c}|$ represents the number of solved tasks. In
essence, the pass$@1$ metric we use can be considered as the accuracy.

###  3.2 Results

####  3.2.1 Main Results

Table 1 shows the performance of CodeM, which are a series of models trained
on monolingual datasets of eight languages respectively, across different
language versions of HumanEval. As we can see, all CodeM models outperform
their base model StarCoder $7$B across all programming languages by a large
margin. Also, we found that programming languages can boost each other
significantly. For example, CodeM-Python trained solely on Python corpus is
able to improve HumanEval-Java by an absolute $14.03$% pass$@1$. This finding
reveals the inherent commonalities among different programming languages. More
surprisingly, CodeM-HTML boosts HumanEval-Java by an absolute $15.24$%
pass$@1$, even exceeding CodeM-Java. Similarly, CodeM-C++ beats CodeM-C on
HumanEval-C, and CodeM-JavaScript beats CodeM-TypeScript on HumanEval-
Typescript. Drawing upon these observations, we conjecture that the
improvement in multilingual code generation performance is predominantly due
to instruction tuning unlocking the model’s inherent potential, such as
natural or programming language understanding and following-instruction
capabilities, rather than merely incorporating new knowledge. In addition to
training CodeM on a monolingual training corpus, we further construct a $9$K
multilingual training set covering $8$ programming languages. Although each
language comprises only a small amount (~$1.2$K) of training instances,
experimental findings suggest that CodeM-Mixed excels in all languages, even
surpassing CodeM-Python on HumanEval-Python and CodeM-Java on HumanEval-Java.
This suggests that it is possible to yield superior code generation
performance by leveraging multilingual data in instruction tuning, without
harming the generalization of the model.

We also conduct experiments on StarCoder $15$B to verify the effectiveness of
CodeM. Specifically, we obtain $108$K Python training data following
WizardCoder (Luo et al., 2023), and finetune StarCoder $15$B to get CodeM-
Python. The results are shown in Table 2. CodeM-Python achieves state-of-the-
art performance on HumanEval-Python with $64.63$% pass$@1$, compared with
other models of the same scale. CodeM-Python also gets a tremendous
improvement in the generation of other programming languages. For instance, it
improves Java and JavaScript by an absolute $17.95$% and $16.77$% pass$@1$,
respectively.

####  3.2.2 Closer Analysis

We analyze the correlation between different programming languages. As
illustrated in Figure 4 (a), the improvement of code generation performance is
sensitive to training corpus of different programming languages. Moreover, we
found that C and C++ can boost each other more significantly, which is the
same for JavaScript and TypeScript. It is reasonable because these languages
are correlated to each other in language design, sharing some common syntax
and grammar. Figure 4 (b) shows that training on each programming language can
boost the code generation performance of all other languages. We can see that
the correlation values in Figure 4 (b) are mostly all positive, implying that
the improvement trend of different language brought by one monolingual
training corpus is relatively similar.

Table 1: Pass$@1$ (Accuracy) of StarCoder $7$B and CodeM trained on various programming languages. The numbers in red represent the absolute increase compared to StarCoder 7B. Model | HumanEval-Multilingual
---|---
Python | JavaScript | TypeScript | C | C++ | Java | Go
StarCoder 7B | 26.83 | 24.39 | 28.57 | 24.69 | 25.61 | 23.17 | 24.39
CodeM-Python | 38.4111.58 | 34.7610.37 | 33.544.97 | 29.014.32 | 34.158.54 | 37.2014.03 | 27.443.05
CodeM-JavaScript | 37.2010.37 | 40.2415.85 | 40.3711.80 | 27.783.09 | 32.937.32 | 34.7611.59 | 26.221.83
CodeM-TypeScript | 33.546.71 | 37.8013.41 | 37.278.70 | 30.255.56 | 30.494.88 | 28.054.88 | 25.611.22
CodeM-C | 39.6312.8 | 37.2012.81 | 32.303.73 | 32.107.41 | 35.379.76 | 38.4115.24 | 28.664.27
CodeM-C++ | 34.577.74 | 35.3710.98 | 32.303.73 | 34.579.80 | 39.0213.41 | 37.2014.03 | 28.053.66
CodeM-Java | 35.378.54 | 33.549.15 | 32.303.73 | 29.634.94 | 31.105.49 | 37.8014.63 | 27.443.05
CodeM-Go | 35.989.15 | 33.549.15 | 31.683.11 | 30.255.56 | 34.158.54 | 35.9812.81 | 32.327.93
CodeM-HTML | 31.714.88 | 33.549.15 | 32.303.73 | 25.931.24 | 28.663.05 | 38.4115.24 | 28.053.66
CodeM-Mixed | 43.2916.46 | 37.2012.81 | 37.899.32 | 32.107.41 | 37.8012.19 | 39.6316.46 | 29.274.88
Table 2: Pass$@1$ of StarCoder $15$B and CodeM-Python. The numbers in red denote the absolute improvement compared to StarCoder 15B. Model | HumanEval-Multilingual
---|---
Python | JavaScript | TypeScript | C | C++ | Java | Go
StarCoder 15B | 32.93 | 30.79 | 32.29 | 26.99 | 31.55 | 30.22 | 17.61
CodeM-Python | 64.6331.09 | 47.5616.77 | 39.757.46 | 35.199.20 | 43.8012.35 | 48.1717.95 | 34.7617.15

(a)

(b)

Figure 4: Correlations between different programming languages. We regard the
data in Table 1 as a matrix, and use “df.corr()” from the Pandas library to
compute the correlation between different programming languages. The
correlation results before and after “df.T” are presented in (a) and (b),
respectively.

##  4 Related Work

Codex (Chen et al., 2021) with 12-billion parameters is able to solve Python
programming problems automatically. This remarkable success triggered a
significant buzz in both the academic and industrial realms. Followed by
Codex, a plenty of code LLMs are proposed, including AlphaCode (Li et al.,
2022), PaLM-Coder (Chowdhery et al., 2022), CodeGen (Nijkamp et al., 2023),
InCoder (Fried et al., 2023), CodeGeeX (Zheng et al., 2023),
replit333https://huggingface.co/replit/replit-code-v1-3b, CodeT5 (Wang et al.,
2021, 2023), PyCodeGPT (Zan et al., 2022), SantaCoder (Allal et al., 2023),
StarCoder (Li et al., 2023), Code Llama (Rozière et al., 2023), and phi-1
(Gunasekar et al., 2023). These above models are trained on a large-scale code
corpus and achieve impressive code generation performance. During their pre-
training, some models are trained on datasets of multilingual programming
languages and then fine-tuned on a monolingual dataset to produce a more
powerful specialist version. As for the instruction fine-tuning phase,
WizardCoder (Luo et al., 2023), PanGu-Coder2 (Shen et al., 2023), and Phind-
CodeLlama444https://huggingface.co/Phind/Phind-CodeLlama-34B-v1 are proposed
to bolster the capability of following instructions and further boost the code
generation capability. Yet, none of these aforementioned models explore the
intricate interplay between different programming languages. In this report,
we therefore would like to investigate whether training code LLMs on
monolingual data can bolster performance in other programming languages.

##  5 Conclusion

Our findings reveal that a monolingual training corpus can enhance the
multilingual code generation capabilities of code LLMs via instruction tuning.
This highlights the intrinsic commonality and interconnectedness among
multiple programming languages. In our future work, we plan to delve into the
reasons why multiple languages can enhance each other. Also, we will explore
how to leverage our findings to elevate code generation capabilities for these
obscure or less-used programming languages by training on data from those
popular ones.

## Acknowledgements

We would like to thank our colleagues for their valuable feedback and
insights. Special thanks to An Fu (Huawei), Jingyang Zhao (Huawei), and Yuenan
Guo (Huawei) for their constructive help throughout this research.

## References

  * Zan et al. [2023] Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou.  Large language models meet NL2Code: A survey.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7443–7464, Toronto, Canada, July 2023. Association for Computational Linguistics.  URL https://aclanthology.org/2023.acl-long.411.
  * Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code.  _ArXiv_ , abs/2107.03374, 2021.
  * Li et al. [2022] Yujia Li, David H. Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom, Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de, Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey, Cherepanov, James Molloy, Daniel Jaymin Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de, Freitas, Koray Kavukcuoglu, and Oriol Vinyals.  Competition-level code generation with alphacode.  _Science_ , 378:1092 – 1097, 2022.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  PaLM: Scaling language modeling with pathways.  _ArXiv_ , abs/2204.02311, 2022.
  * Nijkamp et al. [2023] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.  CodeGen: An open large language model for code with multi-turn program synthesis.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Zheng et al. [2023] Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shanshan Wang, Yufei Xue, Zi-Yuan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang.  CodeGeeX: A pre-trained model for code generation with multilingual evaluations on humaneval-x.  _ArXiv_ , abs/2303.17568, 2023.
  * Li et al. [2023] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries.  StarCoder: may the source be with you!, 2023.
  * Rozière et al. [2023] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve.  Code Llama: Open foundation models for code, 2023.
  * Ouyang et al. [2022] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe.  Training language models to follow instructions with human feedback.  _ArXiv_ , abs/2203.02155, 2022.
  * Luo et al. [2023] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang.  WizardCoder: Empowering code large language models with evol-instruct.  _arXiv preprint arXiv:2306.08568_ , 2023.
  * Shen et al. [2023] Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, Yuenan Guo, and Qianxiang Wang.  PanGu-Coder2: Boosting large language models for code with ranking feedback, 2023.
  * Xu et al. [2023] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  WizardLM: Empowering large language models to follow complex instructions, 2023.
  * Fried et al. [2023] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis.  InCoder: A generative model for code infilling and synthesis.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Wang et al. [2021] Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi.  CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8696–8708, 2021.
  * Wang et al. [2023] Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi.  Codet5+: Open code large language models for code understanding and generation, 2023.
  * Zan et al. [2022] Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou.  CERT: Continual pre-training on sketches for library-oriented code generation.  In _International Joint Conference on Artificial Intelligence_ , 2022\.
  * Allal et al. [2023] Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Muñoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alexander Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, J. Poirier, Hailey Schoelkopf, Sergey Mikhailovich Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Franz Lappert, Francesco De Toni, Bernardo Garc’ia del R’io, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luisa Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Christopher Hughes, Daniel Fried, Arjun Guha, Harm de Vries, and Leandro von Werra.  SantaCoder: don’t reach for the stars!  _ArXiv_ , abs/2301.03988, 2023.
  * Gunasekar et al. [2023] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li.  Textbooks are all you need, 2023.


# A Weighted Solution to SVM Actionability and Interpretability

Samuel Marc Denton
<EMAIL_ADDRESS>Ansaf Salleb-Aouissi
<EMAIL_ADDRESS>

(Department of Computer Science
Columbia University
New York, NY 10025
)

###### Abstract

Research in machine learning has successfully developed algorithms to build
accurate classification models. However, in many real-world applications, such
as healthcare, customer satisfaction, and environment protection, we want to
be able to use the models to decide what actions to take.

We investigate the concept of actionability in the context of Support Vector
Machines. Actionability is as important as interpretability or explainability
of machine learning models, an ongoing and important research topic.
Actionability is the task that gives us ways to act upon machine learning
models and their predictions.

This paper finds a solution to the question of actionability on both linear
and non-linear SVM models. Additionally, we introduce a way to account for
weighted actions that allow for more change in certain features than others.
We propose a gradient descent solution on the linear, RBF, and polynomial
kernels, and we test the effectiveness of our models on both synthetic and
real datasets. We are also able to explore the model’s interpretability
through the lens of actionability.

##  1 Introduction

SVM models are one of the best performing predictive models in machine
learning, as they can effectively separate data into differing labels while
maximizing the margin of the decision function. Despite the progress achieved
with other predictive methods such as deep learning, SVM models remain
competitive and compelling in areas likes healthcare, where data will never
reach the large volumes required to train deep neural networks models. While
there is extensive research on SVMs, there is little to no information about
finding ways to use the margin to change labels of new datapoints. If we are
given a datapoint which, based on the decision function of the SVM, is most
likely an undesirable outcome (e.g., patient at risk of cardiovascular
disease), or a datapoint for which we know the class (e.g, a sick patient), we
would like to use the SVM boundary to move the datapoint to a desirable
outcome. Moving the datapoint can be complex and challenging depending on the
features themselves as well as the shape of the SVM boundary. Having this
functionality would be relevant to many applications such as medical care,
customer satisfaction, and environmental protection.

In this paper, we tackle the question of “What do I do with the prediction?”
or “How do I use a prediction model to act upon an undesirable label?”.
Previous research (e.g., [4, 12, 15, 16, 19]) has defined actionability as
giving a recommendation of the easiest way for a user to change an
"unsatisfactory situation" based on a set of rules. Unfortunately, while rule-
based classifiers allow for interpretability of the feature decisions, they
are not the best performing models. Additionally, previous papers focus on the
framework of defining costs of an action in rule-based classification and how
to maximize gain for a given cost. In the context of an SVM, this part of the
problem will be slightly simpler as we can assume we are in a Euclidean space
and thus we will frame our problem as a minimization of a weighted Euclidean
distance. To our knowledge, we are the first to propose a general approach for
actionability, in the context of support vector machines.

We illustrate our approach with atherosclerosis, a serious disease of the
arteries characterized by the deposition of plaques of fatty material on their
inner walls, causing serious complications like a reduced blood flow in the
big arteries, which increases the risk for heart attack and stroke.
Atherosclerosis has no known cure, however, prevention can play an important
role in reducing patient risk, by keeping under control the main risk factors:
high cholesterol and triglycerides, tobacco consumption, diabetes, high BMI,
high blood pressure, and lack of activity. The way these factors intertwine is
unknown, and hoping to reduce all the factors at once is unrealistic for most
patients. In atherosclerosis, as in many other health problems, one challenge
that a physician might face is to make important decisions about what should
be done to cure a patient or reduce his risk of a specific disease. The
purpose of actionability here is to suggest actions that are fine-grained
modifications of some actionable risk factors in order to decrease the risk of
atherosclerosis in patients.

We define an action as a change in the value of some feature (e.g., weight of
a patient, or his tobacco consumption). Our approach focuses on a framework
where a datapoint is framed as a vector of numerical features. We will then
use the vector to predict an outcome based on the given SVM model. The goal
will be to minimize the weighted distance of the action that shifts the
features to a point along the margin of the desired outcome. The weighted
distance will be some combination of the distance travelled in each feature
multiplied by a weight that represents the challenge of moving each feature,
that is how actionable each feature is (e.g., how hard it is to stop smoking).
The weight vector used to weight these distances is decided by the user. An
example of weighting, say in the application of assessing the risk of
atherosclerosis, may be that the amount of alcohol consumed (easier to change)
has a lower weight while BMI (harder to change) has a higher weight. The
action returned will then be the change in each actionable feature required to
achieve the desired outcome.

Through the lens of actionability, we also investigate interpretability, the
ability to explain machine learning models in a form that is understandable to
a human [2]. By setting the weight vector to all 1’s, we use the recommended
actions to rank-order the pertinent features in the model.

In Section 2, we describe the related work on actionability. In Section 3, we
describe the problem set up. In Section 4, we explore the linear SVM solution.
In Section 5, we extend the solution to fit the non-linear SVM problem. In
Section 6, we test the model on both simulated data and medical data. In
Section 7, we discuss the results and conclusions we can make about the
algorithm.

##  2 Related Work

In many real-world applications, a domain expert can act upon a prediction in
many ways, but ultimately would like to use the prediction and the model to
decide which actions to take. This type of knowledge falls under the umbrella
of actionability, an important but widely unexplored concept in machine
learning. A related topic is the interpretability of machine learning models,
for which there is a surge in research interest from the community (e.g., [3,
10, 18]). Understanding why a model is making a prediction increases our
confidence in using the models, helps detect bias, and communicate/explain
predictions to other parties. While interpretability aims to answer the
question “Why did the model make this prediction?”, actionability gives ways
to act upon those models and their predictions and address the related key
question of “What to do now?”.

Most research on learning actionable knowledge goes back to rule-based models,
that can be classified into three main lines of research: (1) use of
predefined set of simple actions that are mapped to classification rules or
deviations [1, 5, 7, 14]; (2) sift through a set of association or subgroup
discovery rules [6, 8, 9, 11] to identify descriptive actionable rules of the
form $\text{Condition}\to\text{Class}$, where Condition is a conjunction of
features describing the target Class; (3) elaborate actions by comparing pairs
of classification rules with contradicting classes but the same values for all
stable attributes [15, 16, 17, 21] to generate action rules. Given an
unsatisfactory situation, the work in [19, 20] discovers a set of action
recommendations to improve that situation using classification rules. Finally,
actionability is formalized as a case-based reasoning problem in [22, 23].
Given an example to reclassify, the authors propose to search for the closest
cases with the desirable class label, among training examples, centroids of
the clusters of the opposite class, or SVM support vectors. Choosing the
closest support vector will be used as a baseline to compare our proposed
approach. For a detailed literature survey on rule-based actionability see
[20].

##  3 Presenting the SVM Actionability Problem

Our model learns how to move a datapoint from the undesirable class to the
desirable class. We will provide our general framework for both a linear and
non-linear SVM. For a given linear SVM model, we have a decision boundary
given by $\mathbf{v}^{T}\mathbf{x}+b=0$. Where $\mathbf{x}\in\mathbb{R}^{d}$
(the input space). We choose to use vector $\mathbf{v}$ here as the decision
vector because we will later use $\mathbf{w}$ for the weight vector used in
the actionability problem. In the case of a non-linear SVM, the decision
boundary is given by $\mathbf{v}^{T}\phi(\mathbf{x})+b=0$, where $\phi$ is the
feature expansion used for the SVM.

Additionally, we are given an initial point $\mathbf{x_{0}}$ for which we
predict an unfavorable result, or for which we know the label is not desirable
(say with label $y_{0}$). Note, for this formulation, we require that for any
label $y$, $y\in\\{-1,+1\\}$. We are also given a weight vector $\mathbf{w}$
which symbolizes the weight vector of the loss of acting on a feature (ranging
between $0$ and $1$). This means the higher $w_{i}$ is, the harder it is to
change that feature (i.e. it is static) and the lower $w_{i}$ is, the easier
it is to change that feature (i.e. it is actionable). We assume that all
features are normalized between $-1$ and $1$ so that we can also standardize
the weight vector and the distance formulation problem.

The goal will be to find the nearest point $\mathbf{x_{n}}$ on the other side
of the SVM boundary (including the margin). In other words, for the case of a
linear SVM, the output is shown in Equation 1.

$\begin{gathered}min_{\mathbf{x_{n}}\in\mathbb{R}^{d}}dist(\mathbf{x_{0}},\mathbf{x_{n}})\\\
\mbox{such that }\mathbf{v}^{T}\mathbf{x_{n}}+b+y_{0}=0\end{gathered}$ (1)

where $dist(\mathbf{x_{0}},\mathbf{x_{n}})$ is defined in Equation 3. Equation
2 presents the case of a non-linear SVM.

$\begin{gathered}min_{\mathbf{x_{n}}\in\mathbb{R}^{d}}dist(\mathbf{x_{0}},\mathbf{x_{n}})\\\
\mbox{such that
}\mathbf{v}^{T}\phi(\mathbf{x_{n}})+b+y_{0}=0\Longleftrightarrow\sum_{i=1}^{N}\alpha_{i}y_{i}K(\mathbf{x_{i}},\mathbf{x_{n}})+b+y_{0}=0\end{gathered}$
(2)

where $dist(\mathbf{x_{0}},\mathbf{x_{n}})$ is once again defined in Equation
3, $N$ is the number of training points, $\mathbf{x_{i}}$ is a training point,
$\alpha_{i}$ is the associated dual coefficient of training point
$\mathbf{x_{i}}$, $y_{i}$ is the label for training point $\mathbf{x_{i}}$,
and $K$ is the kernel function corresponding to the feature expansion $\phi$.

The function we are aiming to minimize is the weighted Euclidean distance
between $\mathbf{x_{0}}$ and $\mathbf{x_{n}}$ in the original feature space
($\mathbb{R}^{d}$). This is shown in Equation 3.

$dist(\mathbf{x_{0}},\mathbf{x_{n}})=\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)^{T}\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)$
(3)

where $\odot$ is the Hadamard product, or element-wise multiplication.

##### Static Features

Throughout this paper, we use the weight vector $\mathbf{w}$ to account for
differing levels of actionable vs. static features. It is important to note,
by making incredibly uneven weight values where some features have weights
around $0.001$ while static features have weights at $1$, they will for all
intents and purposes be static. This is the most interesting case for this
paper. However, it should be noted that if we had a static variable such as
age that cannot be moved at all, we could easily fix that variable. To do
this, we can factor the static features into the intercept variable $b$ for
each point $\mathbf{x_{0}}$. Let’s say $x_{0_{0}}$ and $x_{0_{1}}$ are static
features. Then we can modify the original constraint as given in Equation 4.

$\begin{gathered}\mathbf{v}^{T}\mathbf{x_{n}}+b+y_{0}=0\Longleftrightarrow\mathbf{v^{\prime}}^{T}\mathbf{x_{n}^{\prime}}+v_{0}x_{0_{0}}+v_{1}x_{0_{1}}+b+y_{0}=0\\\
\mathbf{v^{\prime}}^{T}\mathbf{x_{n}^{\prime}}+b^{\prime}+y_{0}=0\mbox{, such
that }b^{\prime}=b+v_{0}x_{0_{0}}+v_{1}x_{0_{1}}\end{gathered}$ (4)

In this case, we now have
$\mathbf{v}^{\prime},\mathbf{x_{n}}^{\prime}\in\mathbb{R}^{d-2}$. We can then
simply perform the original optimization problem with the features in a lower
dimension leaving the static features as the fixed inputs of $\mathbf{x_{0}}$.

(a) Changing negative label to positive label

(b) Changing positive label to negative label

Figure 1: Examples of changing the label predicted by a linear SVM. In Figure
1(a), the weights are $w_{x}=0.2$, $w_{y}=0.7$. On the other hand, in Figure
1(b), the weights are $w_{x}=1.0$, $w_{y}=0.1$.

##  4 Solution to Linear SVM

We derive a solution for Equation 1 using a Lagrangian and then performing
gradient descent to optimize for distance. This derivation is shown in
Appendix A. The result is to choose $\mathbf{x_{1}}$ such that
$\mathbf{v}^{T}\mathbf{x_{1}}+b+y_{0}=0$ and perform iterative updates
$\mathbf{x_{j+1}}:=\mathbf{x_{j}}-\eta\nabla\mathcal{L}(\mathbf{x_{j}})$ where
$\nabla_{\mathbf{x_{j}}}\mathcal{L}(\mathbf{x_{j}},\lambda)$ is defined in
Equation 5.

$\nabla_{\mathbf{x_{j}}}\mathcal{L}(\mathbf{x_{j}},\lambda):=-2\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{j}}\right]\right)+\left(\frac{2\mathbf{v}^{T}\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{j}}\right]\right)}{\mathbf{v}^{T}\mathbf{v}}\right)\mathbf{v}$
(5)

In Figure 1 there are two examples showing the gradient descent solution for a
linear SVM. Both solutions are heavily influenced by the given weights for $x$
and $y$ respectively.

##  5 Solution to Non-Linear SVM

Just as we did for the linear SVM, we can formulate the solution to Equation 2
using a Lagrangian and then perform gradient descent to optimize for the
distance while taking into account the constraint. This derivation is shown in
Appendix B. The result is to choose $\mathbf{x_{1}}$ such that
$\sum_{i=1}^{N}\alpha_{i}y_{i}K(\mathbf{x_{i}},\mathbf{x_{1}})+b+y_{0}=0$. For
some problems, finding $\mathbf{x_{1}}$ is computationally infeasible, so an
approximation is needed. In our experiment we use the nearest support vector
on the margin. Then use
$\nabla_{\mathbf{x_{j}}}\mathcal{L}(\mathbf{x_{j}},\lambda)$ defined in
Equation 6 in the gradient descent algorithm.

$\begin{gathered}\nabla_{\mathbf{x_{j}}}\mathcal{L}(\mathbf{x_{j}},\lambda)=-2\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{j}}\right]\right)+\lambda\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})\right),\\\
\lambda=\frac{2\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})\right)^{T}\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)}{\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})\right)^{T}\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})\right)}\end{gathered}$
(6)

(a) Polynomial kernel example.

(b) RBF kernel example.

Figure 2: Examples with non-linear SVM

where $K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})$ is the derivative of the
kernel function $K(\mathbf{x_{i}},\mathbf{x_{n}})$ with respect to
$\mathbf{x_{n}}$ (this must exist to find the nearest point).

Figure 2(a), presents an example for a polynomial SVM that finds the nearest
point with the opposite label to the initial point using gradient descent.

Figure 2(b) presents an example for a RBF SVM that finds the nearest point
with the opposite label to the initial point using gradient descent.

##  6 Experiments

We evaluate our approach and present our experiments on both simulated and
medical data. We also introduce a method of implementing interpretability
through the suggested actions.

###  6.1 Simulated Data

We simulate data using the sklearn datasets package. This package creates
separable data for an SVM classifier with noise. We use three different
simulated datasets: 1) linearly separable data 2) a dataset where one group is
inside a circle and the other is outside (the circles dataset) and 3) a
dataset of two interleaving half circles (the moons dataset). To measure the
success of the action proposed by our solution, we compare it to the only
other research that’s been done in SVM actionability which is to move to the
nearest support vector that’s at least on the margin of the desired class
[22].

##### Linear SVM Results

In this section we compare the gradient descent and nearest support vector
solutions. We see the distances for 1,000 datapoints in Figure 3(a). The
average weighted normalized distance for the gradient descent solution was
$0.003$ and for the nearest support vector was $0.013$. The gradient descent
solution shows a statistically significant p-value $<0.05$ which implies that
the distance from gradient descent to the initial point is less than that of
the nearest support vector. The gradient descent found a solution in at most
$2.60$ milliseconds.

##### Non-Linear SVM Results

To compare the gradient descent and nearest support vector solutions, the
distances for 1,000 datapoints with a RBF kernel are shown in Figure 3(b) for
the moons dataset. The average weighted normalized distance for the gradient
descent solution was $0.044$ and for the nearest support vector was $0.085$.
The gradient descent solution shows a statistically significant p-value
$<0.05$ which implies that the distance from gradient descent to the initial
point is less than that of the nearest support vector. The gradient descent
found a solution in at most $916.72$ milliseconds. The distances for 1,000
datapoints with a polynomial kernel are shown in Figure 3(c) for the circles
datasets. The average weighted normalized distance for the gradient descent
solution was $0.015$ and for the nearest support vector was $0.022$. The
gradient descent solution shows a statistically significant p-value $<0.05$
which again implies that the distance from gradient descent to the initial
point is less than that of the nearest support vector. The gradient descent
found a solution in at most $64.6$ milliseconds.

(a) Linear SVM

(b) RBF SVM on moons

(c) Polynomial SVM for circles

Figure 3: Distances of the action for linear, polynomial and RBF SVM.

###  6.2 The Atherosclerosis dataset

(a) Action for RBF SVM

(b) Action for polynomial SVM

(c) Action for RBF SVM

(d) Action for polynomial SVM

Figure 4: Figures 4(a)-4(b) are of SUBSC/TRIC action. Figures 4(c)-4(d) are of
BMI action. We compare gradient descent solution and nearest support vector.

To illustrate the usefulness of our approach in preventing atherosclerosis, we
conducted experiments using the atherosclerosis dataset derived from
Stulong111https://relational.fit.cvut.cz/dataset/Atherosclerosis, a twenty
year study of the risk factors of the atherosclerosis in a population of 1,419
middle aged men. The patients have been classified into three groups: the
normal, the risk, and the pathological groups. We also dispose of a granular
integer risk score for each patient. The dataset used was pre-processed for
the PKDD Discovery Challenge 2002. Our aim is to produce actions that would
find the easiest way to take a patient at risk of atherosclerosis and get them
to a point that would have a prediction of no risk.

When creating a SVM model to describe this dataset, we used the following
features [SUBSC, TRIC, TRIGL, SYST, DIAST, BMI, WEIGHT, CHLST, ALCO_CONS,
TOBA_CONSO] as these have all shown impacts on atherosclerosis risk. There is
a description of each of these features in Appendix C. We then grouped
together the pathological risk and risk groups as positive risk or $+1$ and no
risk as negative risk or $-1$. Then, we tuned the SVM to perform best on this
prediction problem using cross validation and tested on a test set of 20% of
the data. This process was performed 10 times. The RBF SVM achieved an
f1-score of $0.68$ on $540$ negative risk examples and an f1-score of $0.91$
on $1960$ positive risk examples with an overall accuracy of $0.86$. On the
other hand, the polynomial SVM achieved an f1-score of $0.61$ on $540$
negative risk examples and an f1-score of $0.91$ on $1960$ positive risk
examples with an overall accuracy of $0.85$. This process led us to the best
parameters for the two models: an RBF SVM with hyperparameters $\gamma=1,C=10$
and a polynomial SVM with hyperparameter degree $=4$.

With the SVM models defined, the next step was to test the recommended actions
based on the solution presented in Section 5. For each point that has a label
of $+1$ and has an SVM prediction of $+1$, we found two points: one, the
nearest support vector with a label of $-1$ and two, the solution via gradient
descent of the nearest point on the margin presented in Section 5. Note, for
this example, we choose a weight vector of $[1,1,.5,.5,.5,.2,.2,.1,.05,.05]$
to correspond with the features [SUBSC, TRIC, TRIGL, SYST, DIAST, BMI, WEIGHT,
CHLST, ALCO_CONS, TOBA_CONSO]. For the RBF SVM, the algorithm took a maximum
time of $3.99s$ and for the polynomial SVM, the algorithm took a maximum time
of $1.25s$.

We compare these two solutions using three different methods: (1) measure the
weighted distance from both solutions to the point we are moving, (2) measure
the action recommended by both solutions, and (3) measure the predicted risk
of both solutions. To measure (3), we create a linear model to predict the
granular integer risk score of atherosclerosis which is measured from $1-10$.
This linear model is presented in Equation 7. Using this regression
model222Interestingly, observe the negative coefficient associated with
alcohol consumption. It has been noted in previous studies that some alcohol
consumption can be beneficial on cardiovascular health [13], we can calculate
and compare the risk associated with the old and new datapoints. This will
help assess the impact of the actions in reducing the risk of atherosclerosis
in the patient.

$Risk=0.0072\cdot SUBSC+0.0036\cdot TRIC+0.00004\cdot TRIGL+0.0088\cdot
SYST\\\ +0.0068\cdot DIAST+0.0191\cdot BMI+0.0037\cdot WEIGHT+0.0049\cdot
CHLST\\\ -0.1903\cdot ALCO\\_CONS+0.8468\cdot TOBA\\_CONS-1.0558$ (7)

Figure 5: RBF Distances Histogram

In Figures 4(a)-4(b), we see a box plot for the actions for a RBF SVM and a
polynomial SVM for two of the highly weighted features: SUBSC and TRIC. As we
would hope, the gradient descent solution requires less movement than the
nearest support vector for these features that are extremely hard to change.
On the other hand, when investigating a feature with a lower weight, such as
BMI, there is more relative movement, particularly for the gradient descent
solution, as this is a feature that a patient can more easily change. This
result is shown in Figures 4(c)-4(d). The rest of these feature action
boxplots are shown in Appendix F.

For the RBF SVM, the average weighted distance of the gradient descent
solution was $0.011$ and for the nearest support vector was $0.073$. For the
polynomial SVM, the average weighted distance of the gradient descent solution
was $0.012$ and for the nearest support vector was $0.144$. The maximum time
to run the gradient descent algorithm on either kernel was $2.265s$. For both
the RBF and polynomial SVM, our solution is statistically significantly closer
to the original point than the solution presented by the nearest support
vector. These results are presented as a histogram for the RBF SVM in Figure
5. The same plot is shown for the polynomial SVM in Appendix D. Additionally,
we see these results presented in the form of a boxplot for both the RBF and
polynomial SVM in Appendix E.

(a) Action for RBF SVM

(b) Action for Poly SVM

Figure 6: Depicting Actions for 7 Patients using both RBF and Poly SVMs.

In Figure 7, we see a graph of the predicted risk of both the original point
and the suggested point for both the gradient descent solution and the nearest
support vector for the RBF SVM. The gradient descent solution shows a higher
probability of decreasing the risk of the initial point. For the polynomial
SVM, the same conclusion is found which is that the gradient descent solution
shows a higher probability of decreasing the risk of the initial point. The
figure depicting these results is shown in Appendix G.

Figure 7: Comparing the predicted risk of the initial point against the
predicted risk of the suggested point of both algorithms for a RBF SVM.

Finally, we have included a set of figures to visualize the recommended
actions of both the gradient descent solution and the nearest support vector.
In Figure 6(a), we visualize the percent change in each feature recommended
for the solutions using a RBF SVM. In Figure 6(b), we see the same for a
polynomial SVM. Although these are just a few randomly chosen patients, we see
the desired trend that the changes required for the gradient descent solutions
are fainter colors (smaller percent change) while the changes for the nearest
support vector are darker colors (larger percent change).

####  6.2.1 Atherosclerosis Interpretability

Using the framework presented in the processing of the atherosclerosis
dataset, we present a way to delve into the interpretability of the SVM that
classifies patients as either negative or positive risk. To do this, we use
results of the SVM actionability.

|  Normalized Amplitude Mean |  Normalized Amplitude Median
---|---|---
TOBA_CONS |  0.1989 |  0.1689
CHLST |  0.0750 |  0.0604
BMI |  0.0749 |  0.0666
SYST |  0.0661 |  0.0508
TRIGL |  0.0552 |  0.0444
DIAST |  0.0525 |  0.0398
TRIC |  0.0407 |  0.0308
WEIGHT |  0.0376 |  0.0265
SUBSC |  0.0373 |  0.0269
ALCO_CONS |  0.0294 |  0.0195
Table 1: Feature interpretability

By using the same RBF SVM ($\gamma=1,C=10$) and a weight vector of all $1$s,
we can investigate the suggested actions as a proxy for feature importance.
That is, when all features have equal weight, the features that move most to
switch a classification from risk to no risk are the most important features
when decreasing atherosclerosis risk. We use this weight vector of all $1$s
and measure the amplitude of all of the recommended actions (normalized by
feature) to determine which features required the most change. In Table 1, we
present the median and mean of the normalized amplitudes of the suggestion
actions of all the features. Based on the results, changing TOBA_CONS is the
most important (by a substantial amount) in terms of decreasing the risk while
ALCO_CONS is the least important. The rest of the rankings allow one to
identify the features an at-risk patient should focus most on when attempting
to reduce risk of atherosclerosis. Polynomial SVM results are in Appendix H.

##  7 Discussion and Conclusion

The task of devising actions or actionability based on predictive models has
been relatively unexplored in modern machine learning research. We propose a
simple, yet an effective and efficient way of using SVM models for
actionability. We show an answer to the nearest point of any point that
switches the label predicted by a linear SVM to a desired result. This
provides a good solution to any application that implements a linear SVM to
divide data. For a non-linear SVM, we were able to find an algorithm via
gradient descent that predicts the nearest point with a desirable outcome.
This reduces the cost of the action. However, it is important to note that
this algorithm is sensitive to the initial guess for the nearest point. A
search of possible starting points should allow one to pick a point that will
lend itself to finding the true global optima (such as the nearest support
vector). In Section 6.2, we saw that the gradient descent solution gave an
easier action than the nearest support vector with the given weight vector for
both the RBF and polynomial SVM. Additionally, the gradient descent solution
gave a higher probability of reducing predicted risk than the nearest support
vector. Using the recommended actions, we explored the interpretability of the
SVM models for the atherosclerosis dataset, and concluded that tobacco
consumption was the most important risk factor to adjusting the
atherosclerosis risk.

We must note that the action is only as good as the SVM model itself. If the
SVM model is a good fit for the prediction problem, then the resulting action
will also be relevant. Additionally, to reduce uncertainty of a given action,
one could adjust the algorithm to go beyond the positive/negative margin to
reach a point that is more confidently classified in the desired group. In
future work, it would be interesting to explore variations of the
actionability problem. For example, it’s possible that a user has the
bandwidth to change a limited number of features. Additionally, it would not
be hard to expand the above results to a multiclass SVM classifier. We could
then provide the nearest action to the desired class. Finally, there is more
work to be done in exploring the weighting. Are there other more applicable
ways to define actionable and static features? What other metrics could we use
to replace the distance/cost metric that we aim to minimize? Paired with
visualization techniques, our SVM approach to actionability has the potential
to have an immense impact in assisting physicians in preventing diseases with
actionable risk factors. We hope that this paper will motivate the machine
learning community to conduct research on actionability so to take full
advantage of the accurate prediction models.

##  8 Broader Impact

This algorithm is an extension of the SVM algorithm, so therefore it will
impact the community in similar ways that SVM has impacted the world. A
suggested action on a predicted outcome of the SVM algorithm could have a
potentially huge beneficial impact in healthcare and the world of precision
medicine. Say a patient is told they will most likely need surgery, a doctor
could then explain to the patient the easiest changes they could make so that
the doctor would no longer recommend surgery as a solution. Thus,
actionability could be used to decide on a prevention policy by providing the
physician with a tool for investigating the effects of the various factors on
the risk of disease, for an individual patient. Additionally, this could be
used for a positive impact in the world of climate change. It’s possible that
researchers use SVMs to predict outcomes of certain environments and that
actionability would allow them to suggest how communities could achieve more
desirable outcomes.

On the other hand, SVMs can be used by companies who may be using SVM models
to predict the success of an advertisement. This algorithm could
hypothetically used in marketing and business strategies to help companies
attract more customers which may be viewed negatively in the eyes of the
consumer. Similarly, politicians may be able to use this to change opinions of
voters by feeding certain features that tend to change the minds of voters for
their own gain.

While SVMs and the actionability approach we propose can be used maliciously,
we do not believe this research would inherently put anyone at a disadvantage.
However, we recognize that there are a few faults of this system. There is
inherent bias in the weights provided by the user and therefore users could
manipulate the system to give recommendations that enforce a previous bias. In
some ways, it may be better to use this as an unweighted algorithm or require
that extensive research is done into the weights of the actions.

## References

  * [1] G. Adomavicius and A. Tuzhilin.  Discovery of actionable patterns in databases: The action hierarchy approach.  In KDD, pages 111–114, 1997.
  * [2] F. Doshi-Velez and B. Kim.  A roadmap for a rigorous science of interpretability.  CoRR, abs/1702.08608, 2017.
  * [3] E. Elenberg, A. G. Dimakis, M. Feldman, and A. Karbasi.  Streaming weak submodularity: Interpreting neural networks on the fly.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4044–4054. Curran Associates, Inc., 2017.
  * [4] Y. Elovici and D. Braha.  A decision-theoretic approach to data mining.  IEEE Transactions on Systems, Man, and Cybernetics, Part A, 33(1):42–51, 2003.
  * [5] Y. Elovici, B. Shapira, and P. B. Kantor.  Using the information structure model to compare profile-based information filtering systems.  Inf. Retr., 6(1):75–97, 2003.
  * [6] D. Gamberger and N. Lavrač.  Generating actionable knowledge by expert-guided subgroup discovery.  In PKDD’02: Proceedings of the 6th European Conference on Principles of Data Mining and Knowledge Discovery, pages 163–174. Springer-Verlag, 2002.
  * [7] Y. Jiang, K. Wang, A. Tuzhilin, and A. W.-C. Fu.  Mining patterns that respond to actions.  In ICDM, pages 669–672, 2005.
  * [8] N . Lavrač, D. Gamberger, and P. Flach.  Subgroup discovery for actionable knowledge generation: deficiencies of classification rule learning and the lesson learned.  In N. Lavrac, T. Fawcett, and H. Motoda, editors, ICML 2002 workshop on Data Mining Lessons Learned, 2002.
  * [9] N. Lavrač, D. Gamberger, and P. Flach.  Subgroup discovery for actionable knowledge generation: Shortcomings of classification rule learning and the lessons learned.  July 2002.
  * [10] T. Lei, R. Barzilay, and T. Jaakkola.  Rationalizing neural predictions.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 107–117, Austin, Texas, November 2016. Association for Computational Linguistics.
  * [11] B. Liu, W. Hsu, and Y. Ma.  Identifying non-actionable association rules.  In KDD ’01: Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, pages 329–334, New York, NY, USA, 2001. ACM Press.
  * [12] B. Liu, M. Hu, and W. Hsu.  Multi-level organization and summarization of the discovered rules.  In Knowledge Discovery and Data Mining, pages 208–217, 2000.
  * [13] N. Lucas, J. Azé, and M. Sebag.  Atherosclerosis Risk Identification and Visual Analysis.  In ECML/PKDD 2002 Discovery Challenge Workshop program, 2002.
  * [14] G. Piatetsky-Shapiro and C. Matheus.  The interestingness of deviations.  In AAAI Workshop on Knowledge Discovery in Databases, pages 25–36, Menlo Park, CA, 1994. AAAI Press.
  * [15] Z. W. Ras and S. Gupta.  Global action rules in distributed knowledge systems.  Fundam. Inf., 51(1):175–184, 2002.
  * [16] Z. W. Ras and A. A. Tzacheva.  Discovering semantic inconsistencies to improve action rules mining.  In IIS, pages 301–310, 2003.
  * [17] Z. W. Ras, A. A. Tzacheva, L.-S. Tsay, and O. Gurdal.  Mining for Interesting Action Rules.  In Proceedings of the 2005 IEEE/WIC/ACM International Conference on Intelligent Agent Technology (IAT’05), pages 187–193. IEEE, 2005\.
  * [18] M. T. Ribeiro, S. Singh, and C. Guestrin.  “Why Should I Trust You?”: Explaining the Predictions of Any Classifier.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1135–1144, New York, NY, USA, 2016. Association for Computing Machinery.
  * [19] R. Trepos, A. Salleb, M.-O. Cordier, V. Masson, and C. Gascuel.  A distance-based approach for action recommendation.  In ECML 2005, 16th European Conference on Machine Learning, LNAI 3720, pages 425–436. Springer-Verlag Berlin Heidelberg, 2005.
  * [20] R. Trepos, A. Salleb-Aouissi, M-O. Cordier, V. Masson, and C. Gascuel-Odoux.  Building actions from classification rules.  Knowledge and Information Systems, 34:267–298, 2011.
  * [21] L.-S. Tsay and Z. W. Ras.  Action rules discovery: system DEAR2, method and experiments.  Journal of Experimental & Theoretical Artificial Intelligence, 17(1-2):119–128, 2005.
  * [22] Q. Yang and H. Cheng.  Mining case bases for action recommendation.  In ICDM, pages 522–529, 2002.
  * [23] Q. Yang, J. Yin, C. X. Ling, and T. Chen.  Postprocessing decision trees to extract actionable knowledge.  In ICDM, pages 685–688, 2003.

##  Appendix A Linear SVM Solution via Gradient Descent

We can formulate the solution to Equation 1 using a Lagrangian and then
perform gradient descent to optimize for the distance while taking into
account the constraint. The Lagrangian is formulated in Equation 8. We then
take the gradient of $\mathcal{L}(\mathbf{x_{n}},\lambda)$ with respect to
$\mathbf{x_{n}}$.

$\begin{gathered}\mathcal{L}(\mathbf{x_{n}},\lambda)=\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)^{T}\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)+\lambda\left(\mathbf{v}^{T}\mathbf{x_{n}}+b+y_{0}\right)\\\
\nabla_{\mathbf{x_{n}}}\mathcal{L}(\mathbf{x_{n}},\lambda)=-2\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)+\lambda\mathbf{v}\end{gathered}$
(8)

Since $\mathcal{L}(\mathbf{x_{n}},\lambda)$ is convex, it will be optimized
when the gradient is equal to zero which allows us to solve for $\lambda$. To
do this, take the dot product of the gradient with the vector $\mathbf{v}$
which leads to the result in Equation 9.

$\begin{gathered}-2\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)+\lambda\mathbf{v}=0\\\
-2\mathbf{v}^{T}\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)+\lambda\mathbf{v}^{T}\mathbf{v}=0\\\
\lambda=\frac{2\mathbf{v}^{T}\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)}{\mathbf{v}^{T}\mathbf{v}}\end{gathered}$
(9)

We can then use this equation for $\lambda$ to find the $\lambda$ at any
$\mathbf{x_{j}}$. This gives us a value for
$\nabla_{\mathbf{x_{j}}}\mathcal{L}(\mathbf{x_{j}},\lambda)$ at any vector
$\mathbf{x_{j}}$. Algorithm 1 shows the gradient descent method for finding
$\mathbf{x_{n}}$ where
$\nabla_{\mathbf{x_{j}}}\mathcal{L}(\mathbf{x_{j}},\lambda):=-2\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{j}}\right]\right)+\left(\frac{2\mathbf{v}^{T}\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{j}}\right]\right)}{\mathbf{v}^{T}\mathbf{v}}\right)\mathbf{v}$.

Input: SVM decision vector $\mathbf{v}$, bias term $b$, initial point
$\mathbf{x_{0}}$ with label $y_{0}$, and weight vector $\mathbf{w}$ ;

set stopping tolerance $\epsilon>0$;

set $\mathbf{x_{1}}$ such that $\mathbf{v}^{T}\mathbf{x_{1}}+b+y_{0}=0$;

$\mathbf{x_{j}}=\mathbf{x_{1}}$;

while
_$\left[\mathbf{x_{0}}-\mathbf{x_{j}}\right]^{T}\left[\mathbf{x_{0}}-\mathbf{x_{j}}\right]
>\epsilon$_ do

$\mathbf{x_{j+1}}:=\mathbf{x_{j}}-\eta\nabla\mathcal{L}(\mathbf{x_{j}})$;

$\mathbf{x_{j}}=\mathbf{x_{j+1}}$;

end while

$\mathbf{x_{n}}=\mathbf{x_{j}}$;

Result: The vector $\mathbf{x_{n}}$ that minimizes the weighted distance to
$\mathbf{x_{0}}$

Algorithm 1 Gradient Descent for Solving Actionability on Linear SVM

##  Appendix B Non-Linear SVM Solution via Gradient Descent

Just as we did for the Linear SVM, we can formulate this question using a
Lagrangian and then perform gradient descent to optimize for the distance
while taking into account the constraint. The Lagrangian is formulated in
Equation 10. We then take the gradient of
$\mathcal{L}(\mathbf{x_{n}},\lambda)$ with respect to $\mathbf{x_{n}}$.

$\begin{gathered}\mathcal{L}(\mathbf{x_{n}},\lambda)=\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)^{T}\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)+\lambda\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K(\mathbf{x_{i}},\mathbf{x_{n}})+b+y_{0}\right)\\\
\nabla_{\mathbf{x_{n}}}\mathcal{L}(\mathbf{x_{n}},\lambda)=-2\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)+\lambda\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})\right)\end{gathered}$
(10)

In Equation 10, $K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})$ is the derivative
of the kernel function $K(\mathbf{x_{i}},\mathbf{x_{n}})$ with respect to
$\mathbf{x_{n}}$ (this must exist to find the nearest point). Again, like we
did in the Linear SVM, we will set the gradient equal to 0 and then solve for
$\lambda$. To do this, take the dot product of the equation with
$\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})\right)$
which leads to the result in Equation 11.

$\begin{gathered}-2\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)+\lambda\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})\right)=0\\\
-2\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})\right)^{T}\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)+\lambda\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})\right)^{T}\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})\right)=0\\\
\lambda=\frac{2\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})\right)^{T}\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{n}}\right]\right)}{\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})\right)^{T}\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})\right)}\end{gathered}$
(11)

We can then use the equation for $\lambda$ from Equation 11 when minimizing
the distance for any $\mathbf{x_{j}}$ (simply replace $\mathbf{x_{n}}$ with
the current estimate $\mathbf{x_{j}}$). This gives us a real value for
$\nabla_{\mathbf{x_{j}}}\mathcal{L}(\mathbf{x_{j}},\lambda)$ at any vector
$\mathbf{x_{j}}$. Using $\lambda$ defined in Equation 11 we define
$\nabla_{\mathbf{x_{j}}}\mathcal{L}(\mathbf{x_{j}},\lambda)$ in Equation 12.

$\nabla_{\mathbf{x_{j}}}\mathcal{L}(\mathbf{x_{j}},\lambda)=-2\left(\mathbf{w}\odot\left[\mathbf{x_{0}}-\mathbf{x_{j}}\right]\right)+\lambda\left(\sum_{i=1}^{N}\alpha_{i}y_{i}K^{\prime}(\mathbf{x_{i}},\mathbf{x_{n}})\right)$
(12)

The gradient descent algorithm for a non-linear SVM is the same as Algorithm
1, except that we must choose $\mathbf{x_{1}}$ such that
$\sum_{i=1}^{N}\alpha_{i}y_{i}K(\mathbf{x_{i}},\mathbf{x_{1}})+b+y_{0}=0$. For
some problems finding $\mathbf{x_{1}}$ is computationally infeasible, so an
approximation is needed. In our experiment we use the nearest support vector
on the margin. Then use
$\nabla_{\mathbf{x_{j}}}\mathcal{L}(\mathbf{x_{j}},\lambda)$ defined in
Equation 12 in the gradient descent algorithm.

###  B.1 Specific Kernel Derivatives

To use the gradient descent solution for a kernel, we must define
$K^{\prime}(\mathbf{x},\mathbf{y})$. The two kernels we explore in this paper
are the RBF kernel defined as
$K(\mathbf{x},\mathbf{y}):=exp(-\gamma||\mathbf{x}-\mathbf{y}||^{2})$ and the
polynomial kernel defined as
$K(\mathbf{x},\mathbf{y}):=\left(\mathbf{x}^{T}\mathbf{y}+c\right)^{D}$.

RBF Kernel: the derivative for the RBF kernel (with respect to $\mathbf{y}$)
is defined as

$K^{\prime}(\mathbf{x},\mathbf{y}):=2\gamma(\mathbf{x}-\mathbf{y})K(\mathbf{x},\mathbf{y})=2\gamma(\mathbf{x}-\mathbf{y})exp(-\gamma||\mathbf{x}-\mathbf{y}||^{2})$

Polynomial Kernel: the derivative for the polynomial kernel (with respect to
$\mathbf{y}$) is defined as

$K^{\prime}(\mathbf{x},\mathbf{y}):=D\mathbf{x}\left(\mathbf{x}^{T}\mathbf{y}+c\right)^{D-1}$

##  Appendix C Description of Atherosclerosis Risk Factors

|  Description |  Feature Range
---|---|---
SUBSC |  Skin fold subscapularis (mm) |  4 - 70
TRIC |  Skin fold triceps (mm) |  1 - 35
TRIGL |  Triglycerides in mg% |  42 - 1197
SYST |  Blood pressure systolic |  80 - 200
DIAST |  Blood pressure diastolic |  50 - 125
BMI |  Body Mass Index |  16.98 - 44.96
WEIGHT |  Weight (kg) |  52 - 133
CHLST |  Cholesterol in mg% |  134 - 510
ALCO_CONS |  Alcohol consumption |  1 - 1.67
TOBA_CONS |  Tobacco consumption |  0 - 1.25
Table 2: Description of the risk factors used for atherosclerosis SVM.

##  Appendix D Histogram of Atherosclerosis Distances

In Figure 8, we see the histogram representation of the distances of the
gradient descent solution and nearest support vector for a polynomial kernel.

Figure 8: Distances for polynomial kernel

##  Appendix E Boxplots of Atherosclerosis Distances

In Figure 9, we see the distance comparisons between the gradient descent
solution and nearest support vector.

(a) Distances for RBF kernel

(b) Distances for polynomial kernel

Figure 9: Boxplot of distances for gradient descent solution and nearest
support vector.

##  Appendix F Boxplots of Feature Changes

In Figures 10 through 16, we present the boxplots of the other feature changes
not presented in the text for a RBF and polynomial SVM.

(a) Action for RBF kernel

(b) Action for polynomial kernel

Figure 10: Boxplot of ALCO_CONS action for gradient descent solution and
nearest support vector.

(a) Action for RBF kernel

(b) Action for polynomial kernel

Figure 11: Boxplot of CHLST action for gradient descent solution and nearest
support vector.

(a) Action for RBF kernel

(b) Action for polynomial kernel

Figure 12: Boxplot of DIAST action for gradient descent solution and nearest
support vector.

(a) Action for RBF kernel

(b) Action for polynomial kernel

Figure 13: Boxplot of SYST action for gradient descent solution and nearest
support vector.

(a) Action for RBF kernel

(b) Action for polynomial kernel

Figure 14: Boxplot of TOBA_CONS action for gradient descent solution and
nearest support vector.

(a) Action for RBF kernel

(b) Action for polynomial kernel

Figure 15: Boxplot of TRIGL action for gradient descent solution and nearest
support vector.

(a) Action for RBF kernel

(b) Action for polynomial kernel

Figure 16: Boxplot of WEIGHT action for gradient descent solution and nearest
support vector.

##  Appendix G Probability of Decreased Risk for Polynomial SVM

In Figure 17 we present the predicted risk of the initial point vs. the
solution for a polynomial SVM.

Figure 17: Comparing the predicted risk of the initial point against the
predicted risk of the suggested point of both algorithms for a polynomial SVM.
In both graphs, we see the probability of decreasing the predicted risk by
moving from the initial point to the suggested point.

##  Appendix H Polynomial SVM Atherosclerosis Interpretability

|  Normalized Amplitude Mean |  Normalized Amplitude Median
---|---|---
TOBA_CONSO |  0.2059 |  0.2035
CHLST |  0.1159 |  0.1071
TRIC |  0.1056 |  0.0744
TRIGL |  0.0992 |  0.0854
WEIGHT |  0.0900 |  0.0648
SYST |  0.0834 |  0.0757
ALCO_CONS |  0.0785 |  0.0624
BMI |  0.0707 |  0.0630
DIAST |  0.0684 |  0.0576
SUBSC |  0.0654 |  0.0536
Table 3: Feature interpretability for polynomial kernel (sorted by mean)

By using the same polynomial SVM (with $degree=4$) and a weight vector of all
$1$s, we investigate the suggested actions as a proxy for feature importance.
In Table 3, we present the median and mean of the normalized amplitudes of the
suggestion actions of all the features. Based on the results, changing
TOBA_CONS is the most important (by a substantial amount) in terms of
decreasing risk of atherosclerosis while SUBSC is the least important. The
rest of the rankings are pretty similar to the rankings for the RBF SVM
presented in Table 1.