7d-f) Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7d) Classification\n",
    "### Data preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# conda install -c conda-forge pandas_ml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "import pandas as pandas\n",
    "import numpy as np\n",
    "import pandas_ml\n",
    "from sklearn.metrics import precision_score\n",
    "from sklearn.metrics import recall_score\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "import warnings\n",
    "#ignore the future warning and calc error if no sample from a label \n",
    "warnings.simplefilter(action='ignore')\n",
    "\n",
    "#load data and merge both tables to one, separate quality label\n",
    "redwinedata = pandas.read_csv('data/winequality-red.csv', sep =';')\n",
    "whitewinedata = pandas.read_csv('data/winequality-white.csv', sep =';')\n",
    "concat_data = redwinedata.append(whitewinedata, ignore_index=True)\n",
    "y = (concat_data[['quality']])\n",
    "y = y.values.ravel()\n",
    "X = concat_data.drop('quality', axis=1)\n",
    "# convert to nparray\n",
    "X = X.values\n",
    "# scaler = preprocessing.StandardScaler().fit(X_train)\n",
    "# using scaler to be able to use it on testset with same settings with scaler.transform(testset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting  and normalizing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler \n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)\n",
    "\n",
    "# Normalizing before or after splitting: https://goo.gl/c1GQpE\n",
    "scaler = StandardScaler().fit(X_train)\n",
    "X_train = scaler.transform(X_train)\n",
    "# before testing do scaler.transform(X_test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remarks:     \n",
    "Macro vs Micro Average in Scores:    \n",
    "macro takes the mean of each labels individual score    \n",
    "micro takes overall recall independant of labels. Take this because we are interested in the overall performance not label-wise    \n",
    "Precision vs Recall in non-binary case:\n",
    "https://datascience.stackexchange.com/questions/32032/precision-and-recall-if-not-binary/32034#32034\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Support Vector Machines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# set model, fit and predict\n",
    "from sklearn.svm import SVC\n",
    "svm = SVC(kernel='rbf')\n",
    "svm.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training-Set Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix:\n",
      "Predicted  3  4     5     6    7  8  9  __all__\n",
      "Actual                                         \n",
      "3          1  0     9    10    0  0  0       20\n",
      "4          0  2    83    60    0  0  0      145\n",
      "5          0  0   956   477    4  0  0     1437\n",
      "6          0  0   362  1475   61  0  0     1898\n",
      "7          0  0    25   505  194  0  0      724\n",
      "8          0  0     0    92   33  0  0      125\n",
      "9          0  0     0     2    1  0  0        3\n",
      "__all__    1  2  1435  2621  293  0  0     4352\n",
      "\n",
      "Classification Scores:\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          3       1.00      0.05      0.10        20\n",
      "          4       1.00      0.01      0.03       145\n",
      "          5       0.67      0.67      0.67      1437\n",
      "          6       0.56      0.78      0.65      1898\n",
      "          7       0.66      0.27      0.38       724\n",
      "          8       0.00      0.00      0.00       125\n",
      "          9       0.00      0.00      0.00         3\n",
      "\n",
      "avg / total       0.61      0.60      0.57      4352\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred_train = svm.predict(X_train)\n",
    "print(\"Confusion Matrix:\")\n",
    "print(pandas_ml.ConfusionMatrix(y_train, y_pred_train))\n",
    "print(\"\\n\"+\"Classification Scores:\")\n",
    "print(classification_report(y_train, y_pred_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test-Set Results "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix:\n",
      "Predicted  3  4    5     6    7  8  9  __all__\n",
      "Actual                                        \n",
      "3          0  0    4     6    0  0  0       10\n",
      "4          0  0   48    22    1  0  0       71\n",
      "5          0  0  456   243    2  0  0      701\n",
      "6          0  0  185   704   49  0  0      938\n",
      "7          0  0   10   271   74  0  0      355\n",
      "8          0  0    1    45   22  0  0       68\n",
      "9          0  0    0     0    2  0  0        2\n",
      "__all__    0  0  704  1291  150  0  0     2145\n",
      "\n",
      "Classification Scores:\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          3       0.00      0.00      0.00        10\n",
      "          4       0.00      0.00      0.00        71\n",
      "          5       0.65      0.65      0.65       701\n",
      "          6       0.55      0.75      0.63       938\n",
      "          7       0.49      0.21      0.29       355\n",
      "          8       0.00      0.00      0.00        68\n",
      "          9       0.00      0.00      0.00         2\n",
      "\n",
      "avg / total       0.53      0.58      0.54      2145\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred = svm.predict(scaler.transform(X_test))\n",
    "print(\"Confusion Matrix:\")\n",
    "print(pandas_ml.ConfusionMatrix(y_test, y_pred))\n",
    "print(\"\\n\"+\"Classification Scores:\")\n",
    "print(classification_report(y_test, y_pred))\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multi-layer Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MLPClassifier(activation='relu', alpha=0.0001, batch_size='auto', beta_1=0.9,\n",
       "       beta_2=0.999, early_stopping=False, epsilon=1e-08,\n",
       "       hidden_layer_sizes=(100, 100), learning_rate='adaptive',\n",
       "       learning_rate_init=0.001, max_iter=1000, momentum=0.9,\n",
       "       nesterovs_momentum=True, power_t=0.5, random_state=1, shuffle=True,\n",
       "       solver='adam', tol=0.0001, validation_fraction=0.1, verbose=False,\n",
       "       warm_start=False)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# set model, fit and predict\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "mlp = MLPClassifier(solver='adam',hidden_layer_sizes=(100, 100), random_state=1, max_iter=1000, learning_rate='adaptive')\n",
    "mlp.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training set results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix:\n",
      "Predicted   3   4     5     6    7   8  9  __all__\n",
      "Actual                                            \n",
      "3          11   1     4     4    0   0  0       20\n",
      "4           0  66    51    25    3   0  0      145\n",
      "5           0   7  1226   184   19   1  0     1437\n",
      "6           0   6   243  1534  113   2  0     1898\n",
      "7           0   1    35   173  506   9  0      724\n",
      "8           0   0     5    27   37  56  0      125\n",
      "9           0   0     0     1    2   0  0        3\n",
      "__all__    11  81  1564  1948  680  68  0     4352\n",
      "\n",
      "Classification Scores:\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          3       1.00      0.55      0.71        20\n",
      "          4       0.81      0.46      0.58       145\n",
      "          5       0.78      0.85      0.82      1437\n",
      "          6       0.79      0.81      0.80      1898\n",
      "          7       0.74      0.70      0.72       724\n",
      "          8       0.82      0.45      0.58       125\n",
      "          9       0.00      0.00      0.00         3\n",
      "\n",
      "avg / total       0.78      0.78      0.78      4352\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred_train = mlp.predict(X_train)\n",
    "print(\"Confusion Matrix:\")\n",
    "print(pandas_ml.ConfusionMatrix(y_train, y_pred_train))\n",
    "print(\"\\n\"+\"Classification Scores:\")\n",
    "print(classification_report(y_train, y_pred_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test-Set Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix:\n",
      "Predicted  3   4    5    6    7   8  9  __all__\n",
      "Actual                                         \n",
      "3          0   3    5    2    0   0  0       10\n",
      "4          2   6   47   15    1   0  0       71\n",
      "5          0  18  472  199   12   0  0      701\n",
      "6          0   5  217  594  117   5  0      938\n",
      "7          0   2   19  161  166   7  0      355\n",
      "8          0   3    2   26   30   7  0       68\n",
      "9          0   0    0    1    1   0  0        2\n",
      "__all__    2  37  762  998  327  19  0     2145\n",
      "\n",
      "Classification Scores:\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          3       0.00      0.00      0.00        10\n",
      "          4       0.16      0.08      0.11        71\n",
      "          5       0.62      0.67      0.65       701\n",
      "          6       0.60      0.63      0.61       938\n",
      "          7       0.51      0.47      0.49       355\n",
      "          8       0.37      0.10      0.16        68\n",
      "          9       0.00      0.00      0.00         2\n",
      "\n",
      "avg / total       0.56      0.58      0.57      2145\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred =  mlp.predict(scaler.transform(X_test))\n",
    "print(\"Confusion Matrix:\")\n",
    "print(pandas_ml.ConfusionMatrix(y_test, y_pred))\n",
    "print(\"\\n\"+\"Classification Scores:\")\n",
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=10, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
       "            oob_score=False, random_state=0, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "# set model, fit and predict\n",
    "rfc = RandomForestClassifier(max_depth=10, random_state=0)\n",
    "rfc.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training-Set Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix:\n",
      "Predicted   3   4     5     6    7   8  9  __all__\n",
      "Actual                                            \n",
      "3          11   0     4     5    0   0  0       20\n",
      "4           0  51    54    37    3   0  0      145\n",
      "5           0   0  1215   215    7   0  0     1437\n",
      "6           0   0   156  1718   24   0  0     1898\n",
      "7           0   0    14   201  508   1  0      724\n",
      "8           0   0     0    47   28  50  0      125\n",
      "9           0   0     0     2    1   0  0        3\n",
      "__all__    11  51  1443  2225  571  51  0     4352\n",
      "\n",
      "Classification Scores:\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          3       1.00      0.55      0.71        20\n",
      "          4       1.00      0.35      0.52       145\n",
      "          5       0.84      0.85      0.84      1437\n",
      "          6       0.77      0.91      0.83      1898\n",
      "          7       0.89      0.70      0.78       724\n",
      "          8       0.98      0.40      0.57       125\n",
      "          9       0.00      0.00      0.00         3\n",
      "\n",
      "avg / total       0.83      0.82      0.81      4352\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred_train = rfc.predict(X_train)\n",
    "print(\"Confusion Matrix:\")\n",
    "print(pandas_ml.ConfusionMatrix(y_train, y_pred_train))\n",
    "print(\"\\n\"+\"Classification Scores:\")\n",
    "print(classification_report(y_train, y_pred_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test-Set Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix:\n",
      "Predicted  3  4    5     6    7  8  9  __all__\n",
      "Actual                                        \n",
      "3          0  0    4     5    1  0  0       10\n",
      "4          0  3   49    18    1  0  0       71\n",
      "5          0  1  486   201   13  0  0      701\n",
      "6          0  1  172   689   74  2  0      938\n",
      "7          0  0   10   183  160  2  0      355\n",
      "8          0  0    2    40   21  5  0       68\n",
      "9          0  0    0     0    2  0  0        2\n",
      "__all__    0  5  723  1136  272  9  0     2145\n",
      "\n",
      "Classification Scores:\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          3       0.00      0.00      0.00        10\n",
      "          4       0.60      0.04      0.08        71\n",
      "          5       0.67      0.69      0.68       701\n",
      "          6       0.61      0.73      0.66       938\n",
      "          7       0.59      0.45      0.51       355\n",
      "          8       0.56      0.07      0.13        68\n",
      "          9       0.00      0.00      0.00         2\n",
      "\n",
      "avg / total       0.62      0.63      0.60      2145\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred = rfc.predict(scaler.transform(X_test))\n",
    "print(\"Confusion Matrix:\")\n",
    "print(pandas_ml.ConfusionMatrix(y_test, y_pred))\n",
    "print(\"\\n\"+\"Classification Scores:\")\n",
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the multilayer perceptron (mlp) and random forest have a very high fit on the training data. This fit can be increased by increasing the layer size (mlp) or maximum depth (random forrest).     \n",
    "\n",
    "This nicely shows the effect of overfitting, because  there is only a positive effect on the training- but not on the test-set-fit.\n",
    "\n",
    "Overall, the results on the test-sets were quite similar. \n",
    "Support Vector Machines (SVM) have the worst performance. I am assuming this is because the (normalized) data is very dense and it is therefor hard to make a \"distance\"-based classification.\n",
    "Furthermore random forest outperforms the multi layer peceptron. It is possible that this is caused by the relatively small sample in regard to a neural net approach like MLP."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7e) Performance >8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Only the random forrest was able to correctly predict scores of 8 (or above) in the test sets.     \n",
    "On one hand this is due to the fact that there are only very few samples with extreme values (see histogram below).    \n",
    "Splitting this data in test and training sets aggravates this issue even more.    \n",
    "Therefore one has to be very careful with conclusions about the quality of the algorithms. It seems like random forrest is most able to predict extreme values one should not draw a conclusion about the quality of the algorithm. Especially the training set predictions of 8 or above are a bad indicator because they reflect overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~timohe/18.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import plotly.plotly as py\n",
    "import plotly.graph_objs as go\n",
    "hist = y\n",
    "data = [go.Histogram(x=hist)]\n",
    "py.iplot(data, filename='basic histogram')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7f) Boosting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "VotingClassifier(estimators=[('svm', SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)), ('mlp', MLPClassifier(activation='relu', alpha=0.0001...estimators=10, n_jobs=1,\n",
       "            oob_score=False, random_state=0, verbose=0, warm_start=False))],\n",
       "         flatten_transform=True, n_jobs=1, voting='hard',\n",
       "         weights=[1, 1, 0])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import VotingClassifier\n",
    "clf1 = SVC()\n",
    "clf2 = MLPClassifier(solver='adam',hidden_layer_sizes=(100, 100), random_state=1, max_iter=1000, learning_rate='adaptive')\n",
    "clf3 = RandomForestClassifier(max_depth=10, random_state=0)\n",
    "combinedClassifier = VotingClassifier(estimators=[('svm', clf1), ('mlp', clf2), ('rfc', clf3)], weights=[1,1,0], voting='hard', flatten_transform=True)\n",
    "combinedClassifier.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training-Set Result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix:\n",
      "Predicted   3   4     5     6    7  8  9  __all__\n",
      "Actual                                           \n",
      "3          11   1     4     4    0  0  0       20\n",
      "4           0  66    58    21    0  0  0      145\n",
      "5           0   7  1259   168    3  0  0     1437\n",
      "6           0   6   439  1417   36  0  0     1898\n",
      "7           0   1    52   490  181  0  0      724\n",
      "8           0   0     5    88   32  0  0      125\n",
      "9           0   0     0     2    1  0  0        3\n",
      "__all__    11  81  1817  2190  253  0  0     4352\n",
      "\n",
      "Classification Scores:\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          3       1.00      0.55      0.71        20\n",
      "          4       0.81      0.46      0.58       145\n",
      "          5       0.69      0.88      0.77      1437\n",
      "          6       0.65      0.75      0.69      1898\n",
      "          7       0.72      0.25      0.37       724\n",
      "          8       0.00      0.00      0.00       125\n",
      "          9       0.00      0.00      0.00         3\n",
      "\n",
      "avg / total       0.66      0.67      0.64      4352\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred_train = combinedClassifier.predict(X_train)\n",
    "print(\"Confusion Matrix:\")\n",
    "print(pandas_ml.ConfusionMatrix(y_train, y_pred_train))\n",
    "print(\"\\n\"+\"Classification Scores:\")\n",
    "print(classification_report(y_train, y_pred_train))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test-Set Result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix:\n",
      "Predicted  3  4    5     6    7  8  9  __all__\n",
      "Actual                                        \n",
      "3          0  0    4     5    1  0  0       10\n",
      "4          0  3   49    18    1  0  0       71\n",
      "5          0  1  486   201   13  0  0      701\n",
      "6          0  1  172   689   74  2  0      938\n",
      "7          0  0   10   183  160  2  0      355\n",
      "8          0  0    2    40   21  5  0       68\n",
      "9          0  0    0     0    2  0  0        2\n",
      "__all__    0  5  723  1136  272  9  0     2145\n",
      "\n",
      "Classification Scores:\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          3       0.00      0.00      0.00        10\n",
      "          4       0.60      0.04      0.08        71\n",
      "          5       0.67      0.69      0.68       701\n",
      "          6       0.61      0.73      0.66       938\n",
      "          7       0.59      0.45      0.51       355\n",
      "          8       0.56      0.07      0.13        68\n",
      "          9       0.00      0.00      0.00         2\n",
      "\n",
      "avg / total       0.62      0.63      0.60      2145\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred = rfc.predict(scaler.transform(X_test))\n",
    "print(\"Confusion Matrix:\")\n",
    "print(pandas_ml.ConfusionMatrix(y_test, y_pred))\n",
    "print(\"\\n\"+\"Classification Scores:\")\n",
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "Given that the classifications are quite similar, boosting doesn't improve the result above the output from the Random Forrest. Because of the algorithms chosen, only a 'hard' and not a confidence based boosting was possible. This could also have influenced the result negatively.    \n",
    "Interestingly, if the weights are put only on SVM and MLP, the results reach the same level as the Random Forrest. It therefore can be agued that boosting works when combining those two but they are not able to outperform the approach with Random Forrest."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}