example_aggregated_shap_value.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Computing The Aggregated Shap Value\n",
    "In our paper we have grouped features, extracted from the eeg recordings based on different conditions, e.g. based on the frequency band a feature was extracted from. In this notebook we demonstrate how to compute the aggregated of shap value for a distinct grouping.\n",
    "This allows us to create rank orders based on feature group importance witch is a crucial step for our proposed ShapAgreement metric."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "abb39e641c890ab1"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "import pickle\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from config.interpolation_maps import chan_map_R12"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "2a2ee7cb11fabd80",
   "execution_count": null
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "def load_object(fname):\n",
    "    try:\n",
    "        with open(fname + \".pickle\", \"rb\") as f:\n",
    "            return pickle.load(f)\n",
    "    except Exception as ex:\n",
    "        print(\"Error during unpickling object (Possibly unsupported):\", ex)"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "6054479e9cca0870",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Loading The Training Set\n",
    "First we load the provided sample training set (`12-all` used in the paper) with the provided names of the features. The names are constructed as:\n",
    "\n",
    "`resting state` _ `frequency band` _ `extraction method` _ `channel name`\n",
    "\n",
    "`resting state`: eyes open (EO), eyes closed (EC)\n",
    "`frequency band`: the frequency band the feature was extracted from (delta, theta, alpha, beta, whole_spectrum)\n",
    "`extraction method`: the name of the extraction method used to compute the feature\n",
    "`channel name`: the name/label of the channel"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "bbec4ad0cb4bf8b6"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "data = load_object(\"data/example_training_set/training_set\")\n",
    "x = data['x']\n",
    "x_names = data['x_names']"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "cae2bdb5edc81c64",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Loading The Shap Values\n",
    "For demonstration purposes we have provided the shap values for all our models trained on set `12-all`. The *shap_values.pickle* file is a dict object containing the shap values on key `shap_values` and the fold indexes on key `fold`. We use the fold indexes to load the samples for which we computed the corresponding shap values (`x_test`)."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "3ea6bf49e6f23f82"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Corresponds to the folder name in the models folder\n",
    "MODEL = \"XGBoost\"\n",
    "\n",
    "shap_data = load_object(f\"models/{MODEL}/shap_values\")\n",
    "fold = shap_data['fold']\n",
    "shap_values = shap_data['shap_values']\n",
    "\n",
    "x_train = [x[i] for i in fold[0]]\n",
    "x_test = [x[i] for i in fold[1]]\n",
    "x_train_df = pd.DataFrame(x_train, columns=x_names)\n",
    "x_test_df = pd.DataFrame(x_test, columns=x_names)"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "7b6b7458a2e2f5fd",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Grouping Features\n",
    "Now we define our function for grouping our features. The provided function takes in a list of all feature names (`x_names`) and groups them based on the frequency band."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "eaf220861244f836"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "freq_bands = ['delta', 'theta', 'alpha', 'beta', 'whole_spec']\n",
    "\n",
    "def group_freq_bands_shap(label_arr):\n",
    "    feature_groups_fb = []\n",
    "    n_labels_fb = []\n",
    "    \n",
    "    for fb in freq_bands:\n",
    "        feature_group_idx = [i for i, name in enumerate(label_arr) if fb in name and\n",
    "                             (fb != 'whole_spec' or not any(ofb in name for ofb in freq_bands[:-1]))]\n",
    "\n",
    "        if feature_group_idx:\n",
    "            feature_groups_fb.append(feature_group_idx)\n",
    "            n_labels_fb.append(fb)\n",
    "\n",
    "    return n_labels_fb, feature_groups_fb\n",
    "\n",
    "n_labels, feature_groups = group_freq_bands_shap(x_names)"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "a765aea3c90a6e88",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Shap Value Aggrigation\n",
    "Now we compute the aggregated shap value for our feature groups by summing the shap values within a group for each sample. Then we compute the mean absolute shap value across our samples."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "d8c5f7aa2ab18915"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# Calculate aggregated SHAP values for each feature group\n",
    "grouped_shap_values = np.zeros((len(x_test), len(n_labels)))\n",
    "for i, group in enumerate(feature_groups):\n",
    "    grouped_shap_values[:, i] = np.sum(shap_values[:, group], axis=1)\n",
    "\n",
    "# Transposing the array so the dimensions are: [groups] x [shap values of samples]\n",
    "grouped_shap_values = np.array(grouped_shap_values).transpose()\n",
    "\n",
    "# Computing the mean absolute shap value\n",
    "mean_abs_shap = []\n",
    "for label_shap_values in grouped_shap_values:\n",
    "    mean_abs_shap.append(np.mean(np.abs(label_shap_values)))"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "3b18e43d743e751f",
   "execution_count": null
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Group Rank Order Based On Shap Values\n",
    "Now we sort the aggregated shap values in order to get the group rank and plot the results."
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "8b0e02f93153de2b"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "n_labels = [r'$\\mathbf{\\delta}$', r'$\\mathbf{\\theta}$', r'$\\mathbf{\\alpha}$', r'$\\mathbf{\\beta}$', r'$\\mathbf{\\omega}$']\n",
    "vals, labels = zip(* sorted(zip(mean_abs_shap, n_labels)))\n",
    "\n",
    "plt.barh(labels, vals)\n",
    "plt.title(r\"$\\mathbf{c)}$ Frequency Band Importance\", weight='bold')\n",
    "plt.xlabel('grouped SHAP value', weight='bold')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "790a51196f6c8ebc",
   "execution_count": null
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}