decision_tree.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#installing packages\n",
    "install.packages('themis')\n",
    "install.packages('randomForest')\n",
    "install.packages('readxl')\n",
    "install.packages('rpart')\n",
    "install.packages('vip')\n",
    "install.packages('parsnip')\n",
    "install.packages('rsample')\n",
    "install.packages('workflows')\n",
    "install.packages(\"lifecycle\")\n",
    "install.packages('yardstick')\n",
    "install.packages(\"C50\", repos=\"http://R-Forge.R-project.org\") \n",
    "install.packages(\"ranger\")\n",
    "install.packages('yardstick')\n",
    "install.packages('tune')\n",
    "install.packages(\"tibble\")\n",
    "install.packages(\"plotly\")\n",
    "install.packages(\"C5.0\")\n",
    "install.packages(\"lightgbm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#loading dataset\n",
    "credit=read.csv('credit.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(plotly)\n",
    "plot_ly(credit, x = ~Status, type = 'histogram', name = 'Status', marker = list(color = c(\"rgba(102, 194, 165,1)\",\n",
    "                                    \"rgba(141, 160, 203,1)\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(credit$Amount, credit$Assets, xlab = \"Amount\", ylab = \"Assets\",pch = 19, col = factor(credit$Status))\n",
    "legend(\"topright\",\n",
    "       legend = levels(factor(credit$Status)),\n",
    "       pch = 19,\n",
    "       col = factor(levels(factor(credit$Status))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(recipes)\n",
    "credit <- credit %>% \n",
    "  recipe(Status ~ ., data=.) %>% \n",
    "  # the next step only works with factor variables\n",
    "  step_string2factor(Status) %>% \n",
    "  themis::step_upsample(Status) %>% \n",
    "  step_factor2string(Status) %>% \n",
    "  # keep characters\n",
    "  prep(strings_as_factors=FALSE) %>% \n",
    "  juice()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_ly(credit, x = ~Status, type = 'histogram', name = 'Status', marker = list(color = c(\"rgba(102, 194, 165,1)\",\n",
    "                                    \"rgba(141, 160, 203,1)\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(credit$Amount, credit$Assets, xlab = \"Amount\", ylab = \"Assets\",pch = 19, col = factor(credit$Status))\n",
    "legend(\"topright\",\n",
    "       legend = levels(factor(credit$Status)),\n",
    "       pch = 19,\n",
    "       col = factor(levels(factor(credit$Status))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Single Trees "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   \\begin{align}\n",
    "            \\mathbf{\\hat{p}}_mk = \\frac{1}{N_m} \\sum_{x_i \\in R_m} I(y_i = k)\n",
    "       \\end{align}\n",
    "    \n",
    "where Nm is the number of items in region m and the summation counts the number of observations of class k in region m."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "\\mathbf{\\hat{f}(x)} = \\sum_{m=1}^M \\hat{c}_m I(x \\in R_m)\n",
    "\\end{align}\n",
    "\n",
    "where\n",
    "\n",
    "\\begin{align}\n",
    "\\mathbf{\\hat{c}_m} = avg(y_i\\mid x_i \\in R_m)\n",
    "\\end{align}\n",
    "\n",
    "is the average y value for the region."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### rpart "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    ". Regression, classification, Poisson, survival\n",
    "\n",
    ". Formula interface only\n",
    "\n",
    "    -Categorical inputs as character or factor\n",
    "    -Categorical outcome as character or factor\n",
    "\n",
    ". Handles missing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(rpart)\n",
    "mod_rpart <- rpart(Status ~ ., data=credit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(rpart.plot)\n",
    "rpart.plot(mod_rpart)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(vip)\n",
    "vip(mod_rpart)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ". minsplit: Minimum number of observations in a node for a split to be attempted\n",
    "\n",
    ". minbucket: Minimum number of observations in a terminal node\n",
    "\n",
    ". cp: A split must improve the fit by cp\n",
    "\n",
    ". maxdepth: The maximum number of splits for terminal nodes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "    \\mathbf{\\hat{f}} = \\frac{1}{B} \\sum_{b=1}^B \\hat{f}^\\mbox{*b}(x)\n",
    "\\end{align}\n",
    "\n",
    "where $\\hat{f}^\\mbox{*b}(x)$ is a decision tree fit on the bth bootstrapped sample of data, with columns randomly selected for evaluation at each split."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### {random forest}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ". Regression or classification\n",
    "\n",
    ". formula interface\n",
    "\n",
    "    -Categorical input data must be factors\n",
    "    -Outcome must be a factor for classification\n",
    "\n",
    ". x/y interface\n",
    "\n",
    "    -x must be a dense matrix\n",
    "    -y must be a factor for classification\n",
    "\n",
    ". Does not handle missing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "credit_imputed <- recipe(Status ~ ., data=credit) %>% \n",
    "  step_impute_knn(all_predictors()) %>% \n",
    "  prep(strings_as_factors=TRUE) %>% \n",
    "  juice()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(randomForest)\n",
    "mod_randomForest <- randomForest(Status ~ ., data=credit_imputed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot(mod_randomForest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vip(mod_randomForest)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ". ntree: Number of trees to grow\n",
    "\n",
    ". mtry: Number of candidate variables to check at each split\n",
    "\n",
    ". replace: Whether to sample with replacement\n",
    "\n",
    ". sampsize: Size of samples to draw\n",
    "\n",
    ". nodesize: Maximum size of terminal nodes\n",
    "\n",
    ". maxnodes: Maximum number of terminal nodes for each tree (complexity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### {ranger} "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ". Regression, classification, survival\n",
    "\n",
    ". formula interface\n",
    "\n",
    "    -Categorical input variables can be character or factor\n",
    "    -Output must be a factor for classification\n",
    "\n",
    ". x/y interface\n",
    "    \n",
    "    -Dense or sparse x\n",
    "    -y argument must be factor for classification\n",
    "\n",
    ". data.frame interface with name of output\n",
    ". Does not handle missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(ranger)\n",
    "mod_ranger <- ranger(Status ~ ., data=credit_imputed, importance='impurity')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# no plotting method, causes error\n",
    "plot(mod_ranger)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vip(mod_ranger)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ". num.trees: Number of trees to grow\n",
    "\n",
    ". mtry: Number of candidate variables to check at each split\n",
    "\n",
    ". max.depth: Maxim depth of any tree\n",
    "\n",
    ". replace: Whether to sample with replacement\n",
    "\n",
    ". sample.fraction: Size of samples to draw\n",
    "\n",
    ". regularization.factor: Amount of penalization on gain\n",
    "\n",
    ". regularization.usedepth: Whether to consider depth in penalization\n",
    "\n",
    ". splitrule: Type of splitting to perform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Boosted Trees "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "    \\mathbf{\\hat{y}_i}^t = \\sum_{k=1}^t f_k(x_i) = \\hat{y}_i^(t-1) + f_t(x_i)\n",
    "\\end{align}\n",
    "\n",
    "where $f_k(x)$ is a tree and $f_k(x)$ is trained on the residuals of $f_{k-1} (x)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### {gbm}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ". Regression, classification, Poisson, survival, ranking\n",
    ". formula interface (x/y interface in other function)\n",
    "\n",
    "    -Categorical inputs must be factor\n",
    "    -Binary outcome must be 0/1\n",
    "\n",
    ". Handles missing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "credit_gbm <- recipe(Status ~ ., data=credit) %>% \n",
    "  step_integer(Status, zero_based=TRUE) %>% \n",
    "  prep(strings_as_factors=TRUE) %>% \n",
    "  juice()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(gbm)\n",
    "mod_gbm <- gbm(Status ~ ., data=credit_gbm, distribution='bernoulli', n.trees=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# no plotting method, causes error\n",
    "plot(mod_gbm, i.var=c('Seniority', 'Records', 'Income'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vip(mod_gbm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ". n.trees: Number of boosting rounds\n",
    "\n",
    ". interaction.depth: Maximum depth of a tree in terms of the number of interactions\n",
    "\n",
    ". n.minobsinnode: Minimum number of observations in terminal nodes\n",
    "\n",
    ". shrinkage: Learning rate\n",
    "\n",
    ". bag.fraction: Percentage of data to sample at each round of boosting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### {C5.0}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ". Classification only\n",
    ". formula interface\n",
    "\n",
    "    -Categorical inputs as character or factor\n",
    "    -Outcome of formula must be a factor\n",
    "\n",
    ". x/y interface\n",
    "\n",
    "    -Only dense matrices for x argument\n",
    "    -y argument must be a factor\n",
    "\n",
    ". Handles missing data\n",
    "\n",
    ". Rule-based models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(\"C50\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mod_C5.0_boost <- C5.0(factor(Status) ~ ., data=credit, trials=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# no plotting method, causes error\n",
    "plot(mod_C5.0_boost, subtree=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vip(mod_C5.0_boost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ". minCases: Smallest number of samples that must be put in at least two of the splits\n",
    "\n",
    ". trials: Number of boosting rounds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mod_ranger <- ranger(Status ~ ., data=credit_imputed, importance='impurity')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# no plotting method, causes error\n",
    "plot(mod_ranger)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vip(mod_ranger)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- num.trees: Number of trees to grow \n",
    "- mtry: Number of candidate variables to check at each split\n",
    "- max.depth: Maxim depth of any tree\n",
    "- replace: Whether to sample with replacement\n",
    "- sample.fraction: Size of samples to draw\n",
    "- regularization.factor: Amount of penalization on gain\n",
    "- regularization.usedepth: Whether to consider depth in penalization\n",
    "- splitrule: Type of splitting to perform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boosted Trees\n",
    "\n",
    "\n",
    "\\begin{align}\n",
    "        \\hat{y}^t_{i} =  \\sum_{k=1}^t f_{k}(x_{i}) = \\hat{y}_{i}^{t-1}+f_{t}(x_{i})  \n",
    "\\end{align}\n",
    "  where $f_{k}(x)$  is a tree and $f_{k}(x)$ is trained on the residuals of $f_{k−1}(x)$\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### {gbm}\n",
    "- Regression, classification, Poisson, survival, ranking\n",
    "- formula interface (x/y interface in other function)\n",
    "        - Categorical inputs must be factor\n",
    "        - Binary outcome must be 0/1\n",
    "- Handles missing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "credit_gbm <- recipe(Status ~ ., data=credit) %>% \n",
    "  step_integer(Status, zero_based=TRUE) %>% \n",
    "  prep(strings_as_factors=TRUE) %>% \n",
    "  juice()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(gbm)\n",
    "mod_gbm <- gbm(Status ~ ., data=credit_gbm, distribution='bernoulli', n.trees=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# no plotting method, causes error\n",
    "plot(mod_gbm, i.var=c('Seniority', 'Records', 'Income'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vip(mod_gbm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- n.trees: Number of boosting rounds\n",
    "- interaction.depth: Maximum depth of a tree in terms of the number of interactions\n",
    "- n.minobsinnode: Minimum number of observations in terminal nodes\n",
    "- shrinkage: Learning rate\n",
    "- bag.fraction: Percentage of data to sample at each round of boosting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### {C5.0}\n",
    "- Classification only\n",
    "- formula interface\n",
    "        - Categorical inputs as character or factor\n",
    "        - Outcome of formula must be a factor\n",
    "- x/y interface\n",
    "        - Only dense matrices for x argument\n",
    "        - y argument must be a factor\n",
    "- Handles missing data\n",
    "- Rule-based models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mod_C5.0_boost <- C5.0(factor(Status) ~ ., data=credit, trials=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# no plotting method, causes error\n",
    "plot(mod_C5.0_boost, subtree=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "vip(mod_C5.0_boost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- minCases: Smallest number of samples that must be put in at least two of the splits\n",
    "- trials: Number of boosting rounds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### {xgboost}\n",
    "- Regression, classification, ranking\n",
    "- Only x/y interface via xgb.DMatrix()\n",
    "        - x can be dense or sparse matrix, or file(s) on disc\n",
    "        - Outcome must be 0/1 for binary classification\n",
    "- Handles missing values\n",
    "- Handles imbalanced data\n",
    "- GLM models: penalized\n",
    "- CPU or GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rec_xg <- recipe(Status ~ ., data=credit) %>% \n",
    "  step_integer(Status, zero_based=TRUE) %>% \n",
    "  step_dummy(all_nominal(), one_hot=TRUE) %>% \n",
    "  prep()\n",
    "\n",
    "x_xg <- juice(rec_xg, all_predictors(), composition='dgCMatrix')\n",
    "y_xg <- juice(rec_xg, all_outcomes(), composition='matrix')\n",
    "\n",
    "library(xgboost)\n",
    "credit_xg <- xgb.DMatrix(data=x_xg, label=y_xg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mod_xgboost <- xgb.train(\n",
    "  data=credit_xg, \n",
    "  objective='binary:logistic',\n",
    "  nrounds=100, \n",
    "  watchlist=list(train=credit_xg), \n",
    "  print_every_n=10\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dygraphs::dygraph(mod_xgboost$evaluation_log)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb.plot.multi.trees(mod_xgboost)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vip(mod_xgboost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- nrounds: Number of boosting rounds\n",
    "- eta: Learning rate\n",
    "- gamma: Minimum loss reduction required to make a split\n",
    "- max_depth: The most splits for any one tree\n",
    "- min_child_weight: Minimum weight needed to make a split (larger: more conservative)\n",
    "- subsample: Percent of rows to sample for each tree\n",
    "- colsample_bytree: Percent of columns to randomly sample for each tree\n",
    "- num_parallel_tree: How many trees to grow in each round\n",
    "        - Allows for boosted pseudo random forests\n",
    "- early_stopping_rounds: Number of rounds without improvement before stopping - early_stopping_rounds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### {lightgbm}\n",
    "- Regression, classification\n",
    "- Only x/y interface via lgb.Dataset()\n",
    "        - x can be dense or sparse matrix, or file(s) on disc\n",
    "        - Categorical inputs can be dummies or numeric encoding\n",
    "        - Outcome must be 0/1 for binary classification\n",
    "        - Outcome can be dense or sparse\n",
    "- Handles missing values\n",
    "- Handles imbalanced data\n",
    "- CPU or GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rec_lgb <- recipe(Status ~ ., data=credit) %>% \n",
    "  step_integer(all_nominal(), zero_based=TRUE) %>%\n",
    "  prep()\n",
    "\n",
    "credit_char <- credit %>% select(-Status) %>% \n",
    "    purrr::map_lgl(~is.character(.x)) %>% which()\n",
    "\n",
    "x_lgb <- juice(rec_lgb, all_predictors(), composition='dgCMatrix')\n",
    "y_lgb <- juice(rec_lgb, all_outcomes(), composition='matrix')\n",
    "\n",
    "library(lightgbm)\n",
    "credit_lgb <- lgb.Dataset(data=x_lgb, label=y_lgb, categorical_feature=credit_char)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mod_lightgbm <- lightgbm(data=credit_lgb, nrounds=100, obj='binary', \n",
    "                         eval_freq=10, is_unbalance=TRUE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# no plotting method, causes error\n",
    "plot(mod_lightgbm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mod_lightgbm %>% lgb.importance() %>% lgb.plot.importance()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- num_iterations: Number of boosting rounds\n",
    "- learning_rate: Learning rate\n",
    "- max_depth: The most splits for any one tree\n",
    "- bagging_fraction: Percent of rows to sample for each tree\n",
    "- feature_fraction: Percent of columns to randomly sample for each tree\n",
    "- boosting: Type of boosting to perform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Models\n",
    "- RuleFit\n",
    "- Catboost\n",
    "- BART\n",
    "- TensorFlow\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### So Many Different Interfaces\n",
    "- formula\n",
    "- matrices\n",
    "        - Dense\n",
    "        - Sparse\n",
    "- data.frame plus vector\n",
    "- Proprietary formats (xgb.DMatrix, lgb.Dataset)\n",
    "- characters vs factors vs dummy variables vs numeric encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- {rpart}: formula\n",
    "- {C5.0}: formula or dense matrices\n",
    "- {randomForest}: formula or dense matrix input with factor outcome\n",
    "- {ranger}: formula or dense/sparse matrix input with factor outcome\n",
    "- {gbm}: formula or dense/sparse matrix input with numeric outcome using gbm.fit()\n",
    "- {xgboost}: dense/sparse matrix input with numeric outcome inside xgb.DMatrix object\n",
    "- {lightgbm}: dense/sparse matrix input with numeric outcome inside lgb.Dataset object.size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- {rpart}: Categorical variables as factor or character\n",
    "- {C5.0}: Categorical variables as factor or character or dummy variables\n",
    "- {randomForest}: Categorical variables as factor or dummy variables\n",
    "- {ranger}: Categorical variables as factor or character or dummy variables\n",
    "- {gbm}: Categorical variables as factor or dummy variables\n",
    "- {xgboost}: Categorical variables as dummy variables\n",
    "- {lightgbm}: Categorical variables as dummy variables or numeric encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## {tidymodels}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(parsnip)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spec_rpart <- decision_tree(mode='classification') %>% \n",
    "  set_engine('rpart')\n",
    "\n",
    "spec_rpart"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spec_C50 <- decision_tree(mode='classification') %>% \n",
    "  set_engine('C5.0')\n",
    "\n",
    "spec_C50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spec_randomForest <- rand_forest(mode='classification') %>% \n",
    "  set_engine('randomForest')\n",
    "\n",
    "spec_randomForest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spec_ranger <- rand_forest(mode='classification') %>% \n",
    "  set_engine('ranger')\n",
    "\n",
    "spec_ranger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spec_C50_boost <- boost_tree(mode='classification') %>% \n",
    "  set_engine('C5.0')\n",
    "\n",
    "spec_C50_boost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spec_xgboost <- boost_tree(mode='classification') %>% \n",
    "  set_engine('xgboost')\n",
    "\n",
    "spec_xgboost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering with {recipes}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(rsample)\n",
    "set.seed(28676)\n",
    "data_split <- initial_split(credit, prop=.9, strata='Status')\n",
    "train <- training(data_split)\n",
    "test <- testing(data_split)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rec_rpart <- recipe(Status ~ ., data=train) %>% \n",
    "  themis::step_upsample(Status) %>% \n",
    "  step_other(all_nominal(), -Status, other='misc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rec_C50 <- recipe(Status ~ ., data=train) %>% \n",
    "  themis::step_upsample(Status) %>% \n",
    "  step_other(all_nominal(), -Status, other='misc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rec_randomForest <- recipe(Status ~ ., data=train) %>% \n",
    "  themis::step_upsample(Status) %>% \n",
    "  step_string2factor(all_nominal(), -Status) %>% \n",
    "  step_knnimpute(all_predictors()) %>% \n",
    "  step_other(all_nominal(), -Status, other='misc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rec_ranger <- recipe(Status ~ ., data=train) %>% \n",
    "  themis::step_upsample(Status) %>% \n",
    "  step_string2factor(all_nominal(), -Status) %>% \n",
    "  step_knnimpute(all_predictors()) %>% \n",
    "  step_other(all_nominal(), -Status, other='misc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rec_C50_boost <- recipe(Status ~ ., data=train) %>% \n",
    "  themis::step_upsample(Status) %>% \n",
    "  step_other(all_nominal(), -Status, other='misc')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rec_xgboost <- recipe(Status ~ ., data=train) %>% \n",
    "  themis::step_upsample(Status) %>%\n",
    "  step_other(all_nominal(), -Status, other='misc') %>% \n",
    "  step_dummy(all_nominal(), -Status, one_hot=TRUE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## {workflows} "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(workflows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "work_rpart <- workflow() %>% \n",
    "  add_recipe(rec_rpart) %>% \n",
    "  add_model(spec_rpart)\n",
    "\n",
    "work_C50 <- workflow() %>% \n",
    "  add_recipe(rec_C50) %>% \n",
    "  add_model(spec_C50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "work_randomForest <- workflow() %>% \n",
    "  add_recipe(rec_randomForest) %>% \n",
    "  add_model(spec_randomForest)\n",
    "\n",
    "work_ranger <- workflow() %>% \n",
    "  add_recipe(rec_ranger) %>% \n",
    "  add_model(spec_ranger)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "work_C50_boost <- workflow() %>% \n",
    "  add_recipe(rec_C50_boost) %>% \n",
    "  add_model(spec_C50_boost)\n",
    "\n",
    "work_xgboost <- workflow() %>% \n",
    "  add_recipe(rec_xgboost) %>% \n",
    "  add_model(spec_xgboost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit the Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# trees\n",
    "fit_rpart <- work_rpart %>% fit(data=train)\n",
    "fit_C50 <- work_C50 %>% fit(data=train)\n",
    "\n",
    "# random forests\n",
    "fit_randomForest <- work_randomForest %>% fit(data=train)\n",
    "fit_ranger <- work_ranger %>% fit(data=train)\n",
    "\n",
    "# boosted trees\n",
    "fit_C50_boost <- work_C50_boost %>% fit(data=train)\n",
    "fit_xgboost <- work_xgboost %>% fit(data=train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How did we do?\n",
    "• Accuracy\n",
    "• Speed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "models <- list(\n",
    "  'rpart'=work_rpart\n",
    "  , 'C50'=work_C50\n",
    "  , 'randomForest'=work_randomForest\n",
    "  , 'ranger'=work_ranger\n",
    "  , 'C50_boost'=work_C50_boost\n",
    "  , 'xgboost'=work_xgboost\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality_metric <- yardstick::metric_set(yardstick::roc_auc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quality <- models %>% \n",
    "  purrr::map(\n",
    "    ~tune::last_fit(.x, split=data_split, metrics=quality_metric)\n",
    "  )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_assesments <- quality %>% \n",
    "  purrr::map_dfr(tune::collect_metrics) %>% \n",
    "  mutate(Model=names(models)) %>% \n",
    "  select(Model, AUC=.estimate) %>% \n",
    "  bind_cols(quality %>% bind_rows())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(plotly)\n",
    "plot_ly(\n",
    "  data=model_assesments %>% arrange(AUC) %>% mutate(Model=factor(Model, levels=Model)), \n",
    "  x=~Model, y=~AUC) %>% \n",
    "  add_lines(marker=list(color=~AUC)) %>% add_annotations(text=~Model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(bench)\n",
    "\n",
    "# run the speed test\n",
    "model_times <- press(\n",
    "  model=models,\n",
    "  mark(fit(model[[1]], data=train), iterations=5)\n",
    ")\n",
    "\n",
    "# combine with AUC\n",
    "model_checks <- model_times %>% \n",
    "  select(Time=median, Memory=mem_alloc) %>% \n",
    "  bind_cols(model_assesments %>% select(Model, AUC))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_ly(\n",
    "  data=model_checks %>% arrange(Time) %>% mutate(Model=factor(Model, levels=Model)), \n",
    "  x=~Model, y=~Time) %>% \n",
    "  add_lines(marker=list(color=~Time)) %>% add_annotations(text=~Model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_checks %>% \n",
    "  plot_ly(x=~Time, y=~AUC) %>% \n",
    "  add_lines(marker=list(color=~AUC)) %>% \n",
    "  add_annotations(text=~Model) %>% layout(showlegend = FALSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Takeaways"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "• So many interfaces to know\n",
    "\n",
    "• Nuances for each model\n",
    "\n",
    "• {xgboost} is nearly the fastest\n",
    "\n",
    "• {xgboost} is nearly the most correct\n",
    "\n",
    "• For this data\n",
    "\n",
    "• On my computer\n",
    "\n",
    "• Everything is easier with {tidymodels}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Thank you!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.4.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}