Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4a13195f",
   "metadata": {},
   "source": [
    "# Building, Training, and Deploying a Machine Learning Model with Amazon SageMaker\n",
    "\n",
    "<p align=\"center\">\n",
    "    <a href=\"https://nbviewer.org/github/KOrfanakis/Amazon_SageMaker_Tutorial/blob/main/Analysis.ipynb\">\n",
    "        <img alt=\"Made with\" src=\"https://img.shields.io/badge/Render%20with-nbviewer-red.svg\">\n",
    "    </a>\n",
    "    <br/>\n",
    "</p>\n",
    "\n",
    "<br>\n",
    "\n",
    "**Table of Contents:**\n",
    "\n",
    "- [Libraries](#Libraries)\n",
    "- [Defining Environment Variables](#Defining-Environment-Variables)\n",
    "- [Creating an S3 Bucket](#Creating-an-S3-Bucket)\n",
    "- [Retrieving the Dataset](#Retrieving-the-Dataset)\n",
    "- [Splitting into Training/Test Sets](#Splitting-into-Training/Test-Sets)\n",
    "- [Training the Model](#Training-the-Model)\n",
    "- [Deploying the Model](#Deploying-the-Model)\n",
    "- [Evaluating Model Performance](#Evaluating-Model-Performance)\n",
    "- [Cleaning Up](#Cleaning-Up)\n",
    "    - [01. Amazon SageMaker](#01.-Amazon-SageMaker)\n",
    "    - [02. Training Artifacts and S3 Bucket](#02.-Training-Artifacts-and-S3-Bucket)\n",
    "    - [03. CloudWatch Log Groups](#03.-CloudWatch-Log-Groups)\n",
    "- [Resources](#Resources)\n",
    "- [Conclusions](#Conclusions)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dcd7941",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "# Libraries\n",
    "\n",
    "We start the notebook by importing the necessary libraries and tools. The AWS-specific tools are:\n",
    "\n",
    "- [`boto3`](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html): the official AWS Software Development Kit (SDK) for Python. `boto3` enables developers to integrate Python applications, libraries, or scripts with AWS services, such as Amazon S3 and Amazon EC2.\n",
    "\n",
    "- [`sagemaker`](https://sagemaker.readthedocs.io/en/stable/): an open-source library for training and deploying Machine Learning (ML) models on Amazon SageMaker.\n",
    "\n",
    "<br>\n",
    "\n",
    "From Sagemaker, we will separately import the `get_execution_role()` function and the `CSVSerializer`. As the name suggests, the former allows us to extract the execution role for the notebook instance, i.e. the [IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) that we created for our notebook instance. This role will later get passed to the tuning job. The latter serialises data of various formats to a CSV-formatted string for the inference endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5dcb627c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3, sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.serializers import CSVSerializer\n",
    "\n",
    "import re, os, urllib.request\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "%config InlineBackend.figure_format='retina'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31bbf222",
   "metadata": {},
   "source": [
    "Moreover, we will install and import [`watermark`](https://github.com/rasbt/watermark), an IPython magic extension that enables us to print version numbers and hardware information. The cell magic command `%%capture` is placed at the beginning of the following cell to suppress its output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b9145139",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install watermark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b3b10f0c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "re         2.2.1\n",
      "matplotlib 3.3.4\n",
      "numpy      1.19.5\n",
      "boto3      1.21.42\n",
      "pandas     1.1.5\n",
      "sagemaker  2.86.2\n",
      "watermark  2.0.2\n",
      "Fri Jun 10 2022 \n",
      "\n",
      "CPython 3.6.13\n",
      "IPython 7.16.1\n",
      "\n",
      "compiler   : GCC 9.3.0\n",
      "system     : Linux\n",
      "release    : 4.14.252-131.483.amzn1.x86_64\n",
      "machine    : x86_64\n",
      "processor  : x86_64\n",
      "CPU cores  : 2\n",
      "interpreter: 64bit\n"
     ]
    }
   ],
   "source": [
    "import watermark\n",
    "%load_ext watermark\n",
    "\n",
    "# See version of system, Python, and libraries\n",
    "%watermark -n -v -m -iv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23ad6f9f",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "# Defining Environment Variables\n",
    "\n",
    "We continue by defining the environment variables we need to prepare the data, train the ML model and deploy it.\n",
    "\n",
    "Firstly, we define the **execution role** for the notebook instance using the `get_execution_role()` function. This is the IAM role we created for this particular notebook instance; it will later be passed to the tunning job.\n",
    "\n",
    "Then, we extract the name of the **AWS region** hosting our instance. This information is necessary for building an XGBoost container (see next paragraph) and later creating an S3 bucket.\n",
    "\n",
    "Finally, we build an **XGBoost container** using `sagemaker`'s `image_uris.retrieve` function. SageMaker uses [Docker containers](https://www.docker.com/resources/what-container/) for training and deploying ML algorithms. Containers allow developers and data scientists to package software into standardised units that run consistently on any platform that supports Docker. In this project, we will use the XGBoost built-in algorithm to build an XGBoost training container. We can automatically spot the XGBoost built-in algorithm image URI using the SageMaker `image_uris.retrieve` API. After specifying the XGBoost image URI, we can use the XGBoost container to construct an estimator using the SageMaker `Estimator` API and initiate a training job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e518abb7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-11T16:13:31.265564Z",
     "start_time": "2022-06-11T16:13:31.245407Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success - the MySageMakerInstance is in the us-east-1 region.\n",
      "You will use the xxxxxxxxxxxx.xxx.xxx.us-east-1.amazonaws.com/xgboost:latest container for your SageMaker endpoint.\n"
     ]
    }
   ],
   "source": [
    "role = get_execution_role()\n",
    "prefix = 'sagemaker/DEMO-xgboost-dm'\n",
    "\n",
    "my_region = boto3.session.Session().region_name\n",
    "\n",
    "# This line automatically looks for the XGBoost image URI and builds an XGBoost container.\n",
    "xgboost_container = sagemaker.image_uris.retrieve('xgboost', my_region, 'latest')\n",
    "\n",
    "print(f'Success - the MySageMakerInstance is in the {my_region} region.')\n",
    "print(f'You will use the {xgboost_container} container for your SageMaker endpoint.')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8767c960",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "# Creating an S3 Bucket\n",
    "\n",
    "In this section, we create an **[Amazon S3](https://aws.amazon.com/s3/) bucket** to store our data. Simply put, an S3 bucket is a storage location to hold files. S3 files are referred to as objects.\n",
    "\n",
    "Using the `boto3` library with Amazon S3 allows us to create, update, and delete S3 buckets and objects from Python programs with ease. To establish a connection with S3, we need to use *[resources](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html)*, i.e. an object-oriented interface to Amazon Web Services. For this purpose, we invoke the `resource()` method of a `boto3.session` and pass in a service name, as in the following example:\n",
    "\n",
    "```s3 = boto3.resource('s3')```\n",
    "\n",
    "Once a connection with S3 is established, we can create a bucket. \n",
    "\n",
    "Every Amazon S3 Bucket must have a unique name across all AWS accounts and customers. Therefore, we need to use a name that has **not** been taken yet; otherwise, the program will yield an error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cc1f5d4f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success: S3 bucket created successfully!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/boto3/compat.py:88: PythonDeprecationWarning: Boto3 will no longer support Python 3.6 starting May 30, 2022. To continue receiving service updates, bug fixes, and security updates please upgrade to Python 3.7 or later. More information can be found here: https://aws.amazon.com/blogs/developer/python-support-policy-updates-for-aws-sdks-and-tools/\n",
      "  warnings.warn(warning, PythonDeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "bucket_name = 'unique-bucket-name'\n",
    "s3 = boto3.resource('s3')\n",
    "\n",
    "try:\n",
    "    if my_region == 'us-east-1':\n",
    "        s3.create_bucket(Bucket=bucket_name)\n",
    "    else:\n",
    "        s3.create_bucket(\n",
    "            Bucket=bucket_name,\n",
    "            CreateBucketConfiguration={'LocationConstraint': my_region})\n",
    "    print('Success: S3 bucket created successfully!')\n",
    "except Exception as e:\n",
    "    print('S3 error: ', e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7aeef990",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "# Retrieving the Dataset\n",
    "\n",
    "The next task is downloading the data to our SageMaker notebook instance and loading the data into a Pandas DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1e104180",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success: Downloaded bank_clean.csv.\n",
      "Success: Data loaded into dataframe.\n"
     ]
    }
   ],
   "source": [
    "dataset_url = 'https://d1.awsstatic.com/tmt/build-train-deploy-machine-learning-model-sagemaker/bank_clean.27f01fbbdf43271788427f3682996ae29ceca05d.csv'\n",
    "\n",
    "try:\n",
    "    urllib.request.urlretrieve(url=dataset_url, filename='bank_clean.csv')\n",
    "    print('Success: Downloaded bank_clean.csv.')\n",
    "except Exception as e:\n",
    "    print('Data load error: ', e)\n",
    "\n",
    "try:\n",
    "    model_data = pd.read_csv('./bank_clean.csv', index_col=0)\n",
    "    print('Success: Data loaded into dataframe.')\n",
    "except Exception as e:\n",
    "    print('Data load error: ', e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17dd99d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54211952",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "# Splitting into Training/Test Sets\n",
    "\n",
    "The dataset is already processed, so we can skip some typical ML pre-processing practices (such as encoding categorical features). The only modification we need to perform is randomly splitting the dataset into a training and a test set. The training data (70% of all customers) is used during the model training. The test data (remaining 30% of customers) is used to evaluate the performance of the trained model and measure how well it generalises to unseen data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2e8a36cc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Set: 28831 rows x 61 columns.\n",
      " Test Set: 12357 rows x 61 columns.\n"
     ]
    }
   ],
   "source": [
    "train_data, test_data = np.split(model_data.sample(frac=1, random_state=1729),\n",
    "                                 [int(0.7 * len(model_data))])\n",
    "\n",
    "print(f'Train Set: {train_data.shape[0]} rows x {train_data.shape[1]} columns.')\n",
    "print(f' Test Set: {test_data.shape[0]} rows x {test_data.shape[1]} columns.')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96c240ad",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "# Training the Model\n",
    "\n",
    "Once splitting is complete, we need to move the training set into the S3 bucket we created earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "78518d4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.concat([train_data['y_yes'], train_data.drop(['y_no', 'y_yes'], axis=1)], axis=1).to_csv('train.csv', index=False, header=False)\n",
    "\n",
    "boto3.Session().resource('s3').Bucket(bucket_name).Object(os.path.join(prefix, 'train/train.csv')).upload_file('train.csv')\n",
    "\n",
    "s3_input_train = sagemaker.inputs.TrainingInput(s3_data='s3://{}/{}/train'.format(bucket_name, prefix), content_type='csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5541659b",
   "metadata": {},
   "source": [
    "The following cell performs three tasks. First, it initialises a [session](https://sagemaker.readthedocs.io/en/stable/api/utility/session.html), which allows us to manage interactions with the Amazon SageMaker APIs and any other AWS services needed.\n",
    "\n",
    "Then, it initialises an `Estimator` instance. We need to specify the following parameters:\n",
    "\n",
    "- `image_uri`: Container image to use for training\n",
    "- `role`: The AWS IAM role we defined earlier\n",
    "- `instance_count`: Number of Amazon EC2 instances to use for training\n",
    "- `instance_type`: Type of EC2 instance to use for training\n",
    "- `output_path`: S3 location for saving the training result (model artifacts and output files)\n",
    "- `sagemaker_session`: Session object which manages interactions with Amazon SageMaker APIs and any other AWS services needed\n",
    "\n",
    "<br>\n",
    "\n",
    "Finally, after creating the `Estimator` instance, we also define the model’s hyperparameters.\n",
    "\n",
    "- `max_depth`: Maximum depth of a tree (Default value: 6)\n",
    "- `eta`: Step size shrinkage used in updates to prevent overfitting (Default value: 0.3)\n",
    "- `gamma`: Minimum loss reduction required to make a further partition on a leaf node of the tree (Default value: 0)\n",
    "- `min_child_weight`: Minimum sum of instance weight needed in a child (Default value: 1) \n",
    "- `subsample`: Subsample ratio of the training instance. Setting it to 0.5 means that XGBoost randomly collects half of the data instances to grow trees (Default value: 1)\n",
    "- `verbosity`: Verbosity of printing messages (Default value: 1). Valid values: 0 (silent), 1 (warning), 2 (info), 3 (debug).\n",
    "- `objective`: Specifies the learning task and the corresponding learning objective. For a full list of valid inputs, refer to [XGBoost Learning Task Parameters](https://github.com/dmlc/xgboost/blob/master/doc/parameter.rst#learning-task-parameters)\n",
    "- `num_round`: The number of rounds to run the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ac77dbcf",
   "metadata": {},
   "outputs": [],
   "source": [
    "sess = sagemaker.Session()\n",
    "\n",
    "xgb = sagemaker.estimator.Estimator(image_uri=xgboost_container,\n",
    "                                    role=role,\n",
    "                                    instance_count=1,\n",
    "                                    instance_type='ml.m4.xlarge',\n",
    "                                    output_path='s3://{}/{}/output'.format(bucket_name, prefix),\n",
    "                                    sagemaker_session=sess)\n",
    "\n",
    "xgb.set_hyperparameters(max_depth=5,\n",
    "                        eta=0.2,\n",
    "                        gamma=4,\n",
    "                        min_child_weight=6,\n",
    "                        subsample=0.8,\n",
    "                        verbosity=2,\n",
    "                        objective='binary:logistic',\n",
    "                        num_round=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19582029",
   "metadata": {},
   "source": [
    "We are now ready to start the training job. The following command trains the model using gradient optimisation on an `ml.m4.xlarge` instance. After a few minutes, we should see the training logs being generated in our Jupyter notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b7438234",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-06-10 12:18:25 Starting - Starting the training job...ProfilerReport-1654863505: InProgress\n",
      "...\n",
      "2022-06-10 12:19:09 Starting - Preparing the instances for training.........\n",
      "2022-06-10 12:20:50 Downloading - Downloading input data...\n",
      "2022-06-10 12:21:21 Training - Downloading the training image......\n",
      "2022-06-10 12:22:16 Training - Training image download completed. Training in progress..\u001b[34mArguments: train\u001b[0m\n",
      "\u001b[34m[2022-06-10:12:22:19:INFO] Running standalone xgboost training.\u001b[0m\n",
      "\u001b[34m[2022-06-10:12:22:19:INFO] Path /opt/ml/input/data/validation does not exist!\u001b[0m\n",
      "\u001b[34m[2022-06-10:12:22:19:INFO] File size need to be processed in the node: 3.38mb. Available memory size in the node: 8461.46mb\u001b[0m\n",
      "\u001b[34m[2022-06-10:12:22:19:INFO] Determined delimiter of CSV input is ','\u001b[0m\n",
      "\u001b[34m[12:22:19] S3DistributionType set as FullyReplicated\u001b[0m\n",
      "\u001b[34m[12:22:20] 28831x59 matrix with 1701029 entries loaded from /opt/ml/input/data/train?format=csv&label_column=0&delimiter=,\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[0]#011train-error:0.100482\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[1]#011train-error:0.099858\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 22 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[2]#011train-error:0.099754\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[3]#011train-error:0.099095\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[4]#011train-error:0.098991\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[5]#011train-error:0.099303\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[6]#011train-error:0.099684\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 22 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[7]#011train-error:0.09906\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[8]#011train-error:0.098852\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 36 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[9]#011train-error:0.098679\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[10]#011train-error:0.098748\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 24 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[11]#011train-error:0.098748\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 22 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[12]#011train-error:0.098748\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[13]#011train-error:0.09854\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[14]#011train-error:0.098574\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[15]#011train-error:0.098609\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[16]#011train-error:0.098817\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[17]#011train-error:0.098817\u001b[0m\n",
      "\u001b[34m[12:22:20] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[18]#011train-error:0.098679\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[19]#011train-error:0.098679\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[20]#011train-error:0.098713\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[21]#011train-error:0.098505\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[22]#011train-error:0.098401\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[23]#011train-error:0.098332\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 34 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[24]#011train-error:0.098332\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[25]#011train-error:0.09795\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[26]#011train-error:0.098262\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 24 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[27]#011train-error:0.098193\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 8 extra nodes, 24 pruned nodes, max_depth=3\u001b[0m\n",
      "\u001b[34m[28]#011train-error:0.097985\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[29]#011train-error:0.097499\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[30]#011train-error:0.097638\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 22 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[31]#011train-error:0.097395\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 28 extra nodes, 24 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[32]#011train-error:0.097222\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[33]#011train-error:0.097118\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[34]#011train-error:0.097014\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[35]#011train-error:0.09684\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[36]#011train-error:0.096667\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[37]#011train-error:0.096736\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 30 extra nodes, 22 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[38]#011train-error:0.096563\u001b[0m\n",
      "\u001b[34m[12:22:21] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 24 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[39]#011train-error:0.096355\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 36 pruned nodes, max_depth=3\u001b[0m\n",
      "\u001b[34m[40]#011train-error:0.096285\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 12 extra nodes, 38 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[34m[41]#011train-error:0.096528\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 8 extra nodes, 16 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[34m[42]#011train-error:0.096355\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 26 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[43]#011train-error:0.096459\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 36 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[44]#011train-error:0.096355\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 4 extra nodes, 34 pruned nodes, max_depth=2\u001b[0m\n",
      "\u001b[34m[45]#011train-error:0.096216\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[46]#011train-error:0.096077\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[47]#011train-error:0.0958\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[48]#011train-error:0.095904\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[49]#011train-error:0.095904\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 34 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[34m[50]#011train-error:0.095834\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[51]#011train-error:0.095765\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 8 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[52]#011train-error:0.095904\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 6 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[53]#011train-error:0.095834\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 22 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[54]#011train-error:0.095834\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 22 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[55]#011train-error:0.09573\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 30 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[56]#011train-error:0.095626\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[57]#011train-error:0.095696\u001b[0m\n",
      "\u001b[34m[12:22:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 28 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[58]#011train-error:0.095661\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 24 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[34m[59]#011train-error:0.095592\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 24 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[60]#011train-error:0.095522\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 16 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[34m[61]#011train-error:0.095383\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 8 extra nodes, 20 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[34m[62]#011train-error:0.095314\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 26 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[63]#011train-error:0.095661\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 8 extra nodes, 32 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[34m[64]#011train-error:0.095661\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 32 extra nodes, 18 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[65]#011train-error:0.095418\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 36 pruned nodes, max_depth=3\u001b[0m\n",
      "\u001b[34m[66]#011train-error:0.095314\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[67]#011train-error:0.095349\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 6 extra nodes, 28 pruned nodes, max_depth=3\u001b[0m\n",
      "\u001b[34m[68]#011train-error:0.095314\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 28 pruned nodes, max_depth=0\u001b[0m\n",
      "\u001b[34m[69]#011train-error:0.095314\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 32 pruned nodes, max_depth=3\u001b[0m\n",
      "\u001b[34m[70]#011train-error:0.095383\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[71]#011train-error:0.095453\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 12 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[72]#011train-error:0.095349\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 8 extra nodes, 24 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[34m[73]#011train-error:0.095245\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[74]#011train-error:0.095175\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 8 extra nodes, 18 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[34m[75]#011train-error:0.095071\u001b[0m\n",
      "\u001b[34m[12:22:23] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 22 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[76]#011train-error:0.095175\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 18 extra nodes, 30 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[77]#011train-error:0.095002\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 20 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[34m[78]#011train-error:0.095037\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 32 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[79]#011train-error:0.095037\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 24 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[80]#011train-error:0.095002\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[81]#011train-error:0.094794\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 24 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[82]#011train-error:0.094759\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[83]#011train-error:0.094933\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 20 extra nodes, 10 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[84]#011train-error:0.09469\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 22 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[85]#011train-error:0.094759\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 22 extra nodes, 24 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[86]#011train-error:0.094482\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 16 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[87]#011train-error:0.094447\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 20 pruned nodes, max_depth=0\u001b[0m\n",
      "\u001b[34m[88]#011train-error:0.094482\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 24 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[89]#011train-error:0.094378\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 28 pruned nodes, max_depth=0\u001b[0m\n",
      "\u001b[34m[90]#011train-error:0.094343\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 8 extra nodes, 28 pruned nodes, max_depth=4\u001b[0m\n",
      "\u001b[34m[91]#011train-error:0.094274\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[92]#011train-error:0.094239\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 16 extra nodes, 32 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[93]#011train-error:0.094169\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 14 extra nodes, 28 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[94]#011train-error:0.094169\u001b[0m\n",
      "\u001b[34m[12:22:24] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 10 extra nodes, 14 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[95]#011train-error:0.094204\u001b[0m\n",
      "\u001b[34m[12:22:25] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 28 pruned nodes, max_depth=0\u001b[0m\n",
      "\u001b[34m[96]#011train-error:0.094204\u001b[0m\n",
      "\u001b[34m[12:22:25] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 26 extra nodes, 20 pruned nodes, max_depth=5\u001b[0m\n",
      "\u001b[34m[97]#011train-error:0.093927\u001b[0m\n",
      "\u001b[34m[12:22:25] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 38 pruned nodes, max_depth=0\u001b[0m\n",
      "\u001b[34m[98]#011train-error:0.093927\u001b[0m\n",
      "\u001b[34m[12:22:25] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 32 pruned nodes, max_depth=0\u001b[0m\n",
      "\u001b[34m[99]#011train-error:0.093892\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "2022-06-10 12:22:50 Uploading - Uploading generated training model\n",
      "2022-06-10 12:22:50 Completed - Training job completed\n",
      "Training seconds: 127\n",
      "Billable seconds: 127\n"
     ]
    }
   ],
   "source": [
    "xgb.fit({'train': s3_input_train})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b82392b",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "# Deploying the Model\n",
    "\n",
    "After fitting an XGBoost Estimator, we can host the newly created model in SageMaker. For this purpose, we can call `deploy` on an XGBoost estimator to create a SageMaker endpoint. The endpoint runs a SageMaker-provided XGBoost model server and hosts the model produced by your training script, which was run when we called `fit`.\n",
    "\n",
    "This step may take a few minutes to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "43ead31a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------!"
     ]
    }
   ],
   "source": [
    "xgb_predictor = xgb.deploy(initial_instance_count=1,\n",
    "                           instance_type='ml.m4.xlarge')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1742906e",
   "metadata": {},
   "source": [
    "The next step is making predictions. We first set up a CSV serialiser, which takes numpy arrays (like our testing data) and serialises them to CSV format. \n",
    "\n",
    "Finally, we call the `predict` method that returns the result of inference against our model (stored in the `predictions_array` array)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "faa59d6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction array: 12357 rows\n"
     ]
    }
   ],
   "source": [
    "test_data_array = test_data.drop(labels=['y_no', 'y_yes'],\n",
    "                                 axis=1).values  # Load the data into an array\n",
    "\n",
    "xgb_predictor.serializer = CSVSerializer()  # Set the serialiser type\n",
    "predictions = xgb_predictor.predict(test_data_array).decode('utf-8')  # Make predictions\n",
    "\n",
    "predictions_array = np.fromstring(predictions[1:], sep=',')  # Turn the prediction into an array\n",
    "print(f'Prediction array: {predictions_array.shape[0]} rows')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e591a279",
   "metadata": {},
   "source": [
    "The output array stores prediction in the form of probabilities, i.e. values ranging from 0 to 1 describing the probability that a new customer will purchase/subscribe to the product."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b654c637",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "# Evaluating Model Performance\n",
    "\n",
    "In this section, we evaluate the performance and accuracy of the trained ML model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0bb5ffb0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Overall Classification Rate: 89.5%\n",
      "\n",
      "Predicted      No Purchase    Purchase\n",
      "Observed\n",
      "No Purchase    90% (10769)    37% (167)\n",
      "Purchase        10% (1133)     63% (288) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "cm = pd.crosstab(index=test_data['y_yes'],\n",
    "                 columns=np.round(predictions_array),\n",
    "                 rownames=['Observed'],\n",
    "                 colnames=['Predicted'])\n",
    "tn = cm.iloc[0, 0]\n",
    "fn = cm.iloc[1, 0]\n",
    "tp = cm.iloc[1, 1]\n",
    "fp = cm.iloc[0, 1]\n",
    "p = (tp + tn) / (tp + tn + fp + fn) * 100\n",
    "\n",
    "print('\\n{0:<20}{1:<4.1f}%\\n'.format('Overall Classification Rate: ', p))\n",
    "print('{0:<15}{1:<15}{2:>8}'.format('Predicted', 'No Purchase', 'Purchase'))\n",
    "print('Observed')\n",
    "print('{0:<15}{1:<2.0f}% ({2:<}){3:>6.0f}% ({4:<})'.format(\n",
    "    'No Purchase', tn / (tn + fn) * 100, tn, fp / (tp + fp) * 100, fp))\n",
    "print('{0:<16}{1:<1.0f}% ({2:<}){3:>7.0f}% ({4:<}) \\n'.format(\n",
    "    'Purchase', fn / (tn + fn) * 100, fn, tp / (tp + fp) * 100, tp))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d061c0c0",
   "metadata": {},
   "source": [
    "The model achieves 90% precision for the ‘No Purchase’ class and 63% precision for the ‘Purchase’ class."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66698c5a",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "# Cleaning Up\n",
    "\n",
    "The last step involves terminating the resources we used in this lab.\n",
    "\n",
    "<br>\n",
    "\n",
    "## 01. Amazon SageMaker\n",
    "\n",
    "### → Endpoint\n",
    "\n",
    "We can delete the endpoint in the Jupyter Notebook by running the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "80c01ec6",
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb_predictor.delete_endpoint(delete_endpoint_config=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b83d6c82",
   "metadata": {},
   "source": [
    "Alternatively, we can open the Amazon SageMaker console at https://console.aws.amazon.com/sagemaker/ and\n",
    "\n",
    "- Under **Inference**, choose **Endpoints**.\n",
    "- Choose the endpoint that we created in the example, choose **Actions**, and then choose **Delete**.\n",
    "\n",
    "Deleting the endpoint also deletes the ML compute instance or instances that support it.\n",
    "\n",
    "<br>\n",
    "\n",
    "### → Endpoint Configuration\n",
    "\n",
    "- Under **Inference**, choose **Endpoint configurations**.\n",
    "- Choose the endpoint configuration that we created in the example, choose **Actions**, and then choose **Delete**.\n",
    "\n",
    "<br>\n",
    "\n",
    "### → The Model\n",
    "\n",
    "- Under Inference, choose **Models**.\n",
    "- Choose the model that we created in the example, choose **Actions**, and then choose **Delete**.\n",
    "\n",
    "<br>\n",
    "\n",
    "### → Notebook Instance\n",
    "\n",
    "- Under **Notebook**, choose **Notebook instances**.\n",
    "- Choose the notebook instance that we created in the example, choose **Actions**, and then choose **Stop**. The notebook instance takes several minutes to stop. When the **Status** changes to **Stopped**, move on to the next step.\n",
    "- Choose **Actions**, and then choose **Delete**.\n",
    "\n",
    "<br>\n",
    "\n",
    "## 02. Training Artifacts and S3 Bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "5bbc0d4d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-06-11T16:07:42.405576Z",
     "start_time": "2022-06-11T16:07:42.391555Z"
    }
   },
   "outputs": [],
   "source": [
    "bucket_to_delete = boto3.resource('s3').Bucket(bucket_name)\n",
    "bucket_to_delete.objects.all().delete()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08439986",
   "metadata": {},
   "source": [
    "Alternatively, open the Amazon S3 console at https://console.aws.amazon.com/s3/, and then delete the bucket that we created for storing model artifacts and the training dataset. \n",
    "\n",
    "<br>\n",
    "\n",
    "## 03. CloudWatch Log Groups\n",
    "\n",
    "Open the Amazon CloudWatch console at https://console.aws.amazon.com/cloudwatch/, and then delete all of the log groups that have names starting with `/aws/sagemaker/`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a4d9efb",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "# Resources\n",
    "\n",
    "The primary resource is the ***[Build, train, and deploy a machine learning model with Amazon SageMaker](https://aws.amazon.com/getting-started/hands-on/build-train-deploy-machine-learning-model-sagemaker/?trk=el_a134p000003yWILAA2&trkCampaign=DS_SageMaker_Tutorial&sc_channel=el&sc_campaign=Data_Scientist_Hands-on_Tutorial&sc_outcome=Product_Marketing&sc_geo=mult&p=gsrc&c=lp_ds)*** tutorial provided by AWS.\n",
    "\n",
    "Additionally, the following resources helped me a lot in improving the code and writing the explanatory text:\n",
    "\n",
    "[1] [What Is Amazon SageMaker?](https://docs.aws.amazon.com/sagemaker/latest/dg/whatis.html)\n",
    "\n",
    "[2] [Get Started with Amazon SageMaker Notebook Instances](https://docs.aws.amazon.com/sagemaker/latest/dg/gs-console.html) and references thereing (specifically, the seven steps below that Topics section).\n",
    "\n",
    "[3] [Amazon SageMaker 101 Workshop](https://catalog.us-east-1.prod.workshops.aws/workshops/0c6b8a23-b837-4e0f-b2e2-4a3ffd7d645b/en-US)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52c125d9",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "# Conclusions\n",
    "\n",
    "Our mini-project came to an end. In summary, we learnt how to use Amazon SageMaker to build, train, and deploy a Machine Learning model using the XGBoost algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba1c666e",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.2"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}