t81_558_class_09_1_keras_transfer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_08_5_kaggle_project.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 9: Regularization: L1, L2 and Dropout**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 9 Material\n",
    "\n",
    "* **Part 9.1: Introduction to Keras Transfer Learning** [[Video]](https://www.youtube.com/watch?v=WLlP6S-Z8Xs&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_1_keras_transfer.ipynb)\n",
    "* Part 9.2: Popular Pretrained Neural Networks for Keras [[Video]](https://www.youtube.com/watch?v=ctVA1_46YEE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_2_popular_transfer.ipynb)\n",
    "* Part 9.3: Transfer Learning for Computer Vision and Keras [[Video]](https://www.youtube.com/watch?v=61vMUm_XBMI&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_3_transfer_cv.ipynb)\n",
    "* Part 9.4: Transfer Learning for Languages and Keras [[Video]](https://www.youtube.com/watch?v=ajmAAg9FxXA&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_4_transfer_nlp.ipynb)\n",
    "* Part 9.5: Transfer Learning for Keras Feature Engineering [[Video]](https://www.youtube.com/watch?v=Dttxsm8zpL8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_09_5_transfer_feature_eng.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "# Start CoLab\n",
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 9.1: Introduction to Keras Transfer Learning\n",
    "\n",
    "Human beings learn new skills throughout their entire lives.  However, this learning is rarely from scratch.  No matter what task a human is learning, they are most likely drawing on experiences from earlier in life to learn this new skill.  In this way, humans learn much differently than most deep learning projects. \n",
    "\n",
    "At some point, a human being learns to tell the difference between a cat and a dog.  To teach a neural network the difference you would obtain a large quantity of cat pictures and dog pictures.  The neural network would iterate over all of these pictures and train on the differences.  The human child that learned to distinguish between the two animals would probably just need to see a few examples where they were told they were looking at each type of animal.  The human child would use previous knowledge of looking at different living and non-living objects to help make this classification.  The child would already know what sub-objects, such as fur, eyes, ears, noses, tails, and teeth looked like.\n",
    "\n",
    "Transfer learning attempts to teach a neural network by similar means.  Rather than training your neural network from scratch, you begin training with preloaded set of weights. Usually you will simply remove the top-most layers of the pretrained neural network and retrain it with new topmost layers.  The layers remaining from the previous neural network will be locked so that training does not change these weights.  Only the newly added layers will be trained.  This process is summarized in the following figure.\n",
    "\n",
    "![Transfer Learning](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/transfer.png \"Transfer Learning\")\n",
    "\n",
    "It can take a great deal of compute power to train a neural network for a large image dataset.  Google, Facebook, Microsoft, and other tech companies have utilized GPU arrays to train high quality neural networks for a variety of applications.  Transferring these weights into your neural network can save you considerable effort and compute time.  It is unlikely that a pretrained model will exactly fit the application that you seek to implement.  Finding the closest pretrained model and using transfer learning is an important skill for a deep learning engineer.\n",
    "\n",
    "The [imagenet dataset]() has several high quality neural networks fit to it.  In the next parts of this module we will take a much closer look at this data set.  For many image recognition tasks, an imagenet trained neural network can be a great starting point for your own neural networks.\n",
    "\n",
    "### Transfer Learning Example\n",
    "\n",
    "Lets look at an example of where transfer learning could be used to build upon an imagenet neural network.  Microsoft released a website that accepts a picture of a dog and attempts to classify these dogs by breed. The Microsoft dog breed website is provided here: \n",
    "\n",
    "[What breed is that dog?](https://www.bing.com/visualsearch/Microsoft/WhatDog)\n",
    "\n",
    "To do this, it ie necessary to obtain pictures of dogs, labeled according to breed. Such a network could be trained entirely from scratch.  However, it would require a large quantity of breed-labeled pictures.  Transfer learning with imagenet could be very beneficial for a neural network project such as this.  A neural network pre-trained on imagenet would already contain neurons that are able to recognize many subcomponents of the various dog breeds that the neural network had previously seen on the other animal images in imagenet.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 150 samples\n",
      "Epoch 1/100\n",
      "150/150 - 1s - loss: 1.0437\n",
      "Epoch 2/100\n",
      "150/150 - 0s - loss: 0.9085\n",
      "Epoch 3/100\n",
      "150/150 - 0s - loss: 0.8192\n",
      "Epoch 4/100\n",
      "150/150 - 0s - loss: 0.7486\n",
      "Epoch 5/100\n",
      "150/150 - 0s - loss: 0.6859\n",
      "Epoch 6/100\n",
      "150/150 - 0s - loss: 0.6365\n",
      "Epoch 7/100\n",
      "150/150 - 0s - loss: 0.5906\n",
      "Epoch 8/100\n",
      "150/150 - 0s - loss: 0.5511\n",
      "Epoch 9/100\n",
      "150/150 - 0s - loss: 0.5183\n",
      "Epoch 10/100\n",
      "150/150 - 0s - loss: 0.4888\n",
      "Epoch 11/100\n",
      "150/150 - 0s - loss: 0.4636\n",
      "Epoch 12/100\n",
      "150/150 - 0s - loss: 0.4415\n",
      "Epoch 13/100\n",
      "150/150 - 0s - loss: 0.4230\n",
      "Epoch 14/100\n",
      "150/150 - 0s - loss: 0.4047\n",
      "Epoch 15/100\n",
      "150/150 - 0s - loss: 0.3886\n",
      "Epoch 16/100\n",
      "150/150 - 0s - loss: 0.3741\n",
      "Epoch 17/100\n",
      "150/150 - 0s - loss: 0.3611\n",
      "Epoch 18/100\n",
      "150/150 - 0s - loss: 0.3486\n",
      "Epoch 19/100\n",
      "150/150 - 0s - loss: 0.3357\n",
      "Epoch 20/100\n",
      "150/150 - 0s - loss: 0.3284\n",
      "Epoch 21/100\n",
      "150/150 - 0s - loss: 0.3112\n",
      "Epoch 22/100\n",
      "150/150 - 0s - loss: 0.3033\n",
      "Epoch 23/100\n",
      "150/150 - 0s - loss: 0.2909\n",
      "Epoch 24/100\n",
      "150/150 - 0s - loss: 0.2814\n",
      "Epoch 25/100\n",
      "150/150 - 0s - loss: 0.2713\n",
      "Epoch 26/100\n",
      "150/150 - 0s - loss: 0.2615\n",
      "Epoch 27/100\n",
      "150/150 - 0s - loss: 0.2531\n",
      "Epoch 28/100\n",
      "150/150 - 0s - loss: 0.2465\n",
      "Epoch 29/100\n",
      "150/150 - 0s - loss: 0.2382\n",
      "Epoch 30/100\n",
      "150/150 - 0s - loss: 0.2297\n",
      "Epoch 31/100\n",
      "150/150 - 0s - loss: 0.2248\n",
      "Epoch 32/100\n",
      "150/150 - 0s - loss: 0.2190\n",
      "Epoch 33/100\n",
      "150/150 - 0s - loss: 0.2111\n",
      "Epoch 34/100\n",
      "150/150 - 0s - loss: 0.2037\n",
      "Epoch 35/100\n",
      "150/150 - 0s - loss: 0.1987\n",
      "Epoch 36/100\n",
      "150/150 - 0s - loss: 0.1918\n",
      "Epoch 37/100\n",
      "150/150 - 0s - loss: 0.1876\n",
      "Epoch 38/100\n",
      "150/150 - 0s - loss: 0.1810\n",
      "Epoch 39/100\n",
      "150/150 - 0s - loss: 0.1779\n",
      "Epoch 40/100\n",
      "150/150 - 0s - loss: 0.1710\n",
      "Epoch 41/100\n",
      "150/150 - 0s - loss: 0.1688\n",
      "Epoch 42/100\n",
      "150/150 - 0s - loss: 0.1630\n",
      "Epoch 43/100\n",
      "150/150 - 0s - loss: 0.1596\n",
      "Epoch 44/100\n",
      "150/150 - 0s - loss: 0.1553\n",
      "Epoch 45/100\n",
      "150/150 - 0s - loss: 0.1522\n",
      "Epoch 46/100\n",
      "150/150 - 0s - loss: 0.1507\n",
      "Epoch 47/100\n",
      "150/150 - 0s - loss: 0.1452\n",
      "Epoch 48/100\n",
      "150/150 - 0s - loss: 0.1413\n",
      "Epoch 49/100\n",
      "150/150 - 0s - loss: 0.1389\n",
      "Epoch 50/100\n",
      "150/150 - 0s - loss: 0.1361\n",
      "Epoch 51/100\n",
      "150/150 - 0s - loss: 0.1367\n",
      "Epoch 52/100\n",
      "150/150 - 0s - loss: 0.1316\n",
      "Epoch 53/100\n",
      "150/150 - 0s - loss: 0.1316\n",
      "Epoch 54/100\n",
      "150/150 - 0s - loss: 0.1247\n",
      "Epoch 55/100\n",
      "150/150 - 0s - loss: 0.1233\n",
      "Epoch 56/100\n",
      "150/150 - 0s - loss: 0.1213\n",
      "Epoch 57/100\n",
      "150/150 - 0s - loss: 0.1199\n",
      "Epoch 58/100\n",
      "150/150 - 0s - loss: 0.1222\n",
      "Epoch 59/100\n",
      "150/150 - 0s - loss: 0.1143\n",
      "Epoch 60/100\n",
      "150/150 - 0s - loss: 0.1248\n",
      "Epoch 61/100\n",
      "150/150 - 0s - loss: 0.1122\n",
      "Epoch 62/100\n",
      "150/150 - 0s - loss: 0.1165\n",
      "Epoch 63/100\n",
      "150/150 - 0s - loss: 0.1228\n",
      "Epoch 64/100\n",
      "150/150 - 0s - loss: 0.1091\n",
      "Epoch 65/100\n",
      "150/150 - 0s - loss: 0.1081\n",
      "Epoch 66/100\n",
      "150/150 - 0s - loss: 0.1070\n",
      "Epoch 67/100\n",
      "150/150 - 0s - loss: 0.1034\n",
      "Epoch 68/100\n",
      "150/150 - 0s - loss: 0.1034\n",
      "Epoch 69/100\n",
      "150/150 - 0s - loss: 0.1058\n",
      "Epoch 70/100\n",
      "150/150 - 0s - loss: 0.1004\n",
      "Epoch 71/100\n",
      "150/150 - 0s - loss: 0.1021\n",
      "Epoch 72/100\n",
      "150/150 - 0s - loss: 0.1022\n",
      "Epoch 73/100\n",
      "150/150 - 0s - loss: 0.0984\n",
      "Epoch 74/100\n",
      "150/150 - 0s - loss: 0.0960\n",
      "Epoch 75/100\n",
      "150/150 - 0s - loss: 0.0954\n",
      "Epoch 76/100\n",
      "150/150 - 0s - loss: 0.0980\n",
      "Epoch 77/100\n",
      "150/150 - 0s - loss: 0.0925\n",
      "Epoch 78/100\n",
      "150/150 - 0s - loss: 0.0944\n",
      "Epoch 79/100\n",
      "150/150 - 0s - loss: 0.0911\n",
      "Epoch 80/100\n",
      "150/150 - 0s - loss: 0.0947\n",
      "Epoch 81/100\n",
      "150/150 - 0s - loss: 0.0896\n",
      "Epoch 82/100\n",
      "150/150 - 0s - loss: 0.0914\n",
      "Epoch 83/100\n",
      "150/150 - 0s - loss: 0.0898\n",
      "Epoch 84/100\n",
      "150/150 - 0s - loss: 0.0905\n",
      "Epoch 85/100\n",
      "150/150 - 0s - loss: 0.0959\n",
      "Epoch 86/100\n",
      "150/150 - 0s - loss: 0.0906\n",
      "Epoch 87/100\n",
      "150/150 - 0s - loss: 0.0898\n",
      "Epoch 88/100\n",
      "150/150 - 0s - loss: 0.0876\n",
      "Epoch 89/100\n",
      "150/150 - 0s - loss: 0.0886\n",
      "Epoch 90/100\n",
      "150/150 - 0s - loss: 0.0845\n",
      "Epoch 91/100\n",
      "150/150 - 0s - loss: 0.0861\n",
      "Epoch 92/100\n",
      "150/150 - 0s - loss: 0.0843\n",
      "Epoch 93/100\n",
      "150/150 - 0s - loss: 0.0848\n",
      "Epoch 94/100\n",
      "150/150 - 0s - loss: 0.0811\n",
      "Epoch 95/100\n",
      "150/150 - 0s - loss: 0.0836\n",
      "Epoch 96/100\n",
      "150/150 - 0s - loss: 0.0832\n",
      "Epoch 97/100\n",
      "150/150 - 0s - loss: 0.0820\n",
      "Epoch 98/100\n",
      "150/150 - 0s - loss: 0.0827\n",
      "Epoch 99/100\n",
      "150/150 - 0s - loss: 0.0794\n",
      "Epoch 100/100\n",
      "150/150 - 0s - loss: 0.0848\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1d1fb8db988>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/iris.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x = df[['sepal_l', 'sepal_w', 'petal_l', 'petal_w']].values\n",
    "dummies = pd.get_dummies(df['species']) # Classification\n",
    "species = dummies.columns\n",
    "y = dummies.values\n",
    "\n",
    "\n",
    "# Build neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "model.fit(x,y,verbose=2,epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep this example simple, we are not setting aside a validation set.  The goal of this example is to show how to create a multi-layer neural network and transfer the weights to another.  Next we evaluate the accuracy on the training set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Accuracy: 0.9866666666666667\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "pred = model.predict(x)\n",
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y,axis=1)\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(f\"Training Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Viewing the model summary is as expected, we can see the three layers previously defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 50)                250       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 25)                1275      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 3)                 78        \n",
      "=================================================================\n",
      "Total params: 1,603\n",
      "Trainable params: 1,603\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've trained a neural network on the iris dataset the knowledge of this neural network can be transferred to other neural networks.  It is possible to create a new neural network from some or all of the layers of this neural network.  Just to demonstrate the technique, we will create a new neural network that is essentially a clone of the first neural network.  This is done by transferring all of the layers from the original neural network into the new one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 50)                250       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 25)                1275      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 3)                 78        \n",
      "=================================================================\n",
      "Total params: 1,603\n",
      "Trainable params: 1,603\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model2 = Sequential()\n",
    "for layer in model.layers:\n",
    "    model2.add(layer)\n",
    "model2.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a sanity check, we would like to calculate the accuracy of the newly created model.  The in-sample accuracy should be the same as the previous model that the new model is based on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Accuracy: 0.9866666666666667\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "pred = model2.predict(x)\n",
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y,axis=1)\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(f\"Training Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The in-sample accuracy of the newly created neural network is the same as the original neural network.  We've successfully transferred all of the layers from the original neural network.\n",
    "\n",
    "For this example we are going to train a neural network to classify three new hypothetical flowers that are uncreatively named:\n",
    "\n",
    "* Fake Flower 1\n",
    "* Fake Flower 2\n",
    "* Fake Flower 3\n",
    "* Fake Flower 4\n",
    "\n",
    "We have measurements for samples of these flowers that conform to the predictors contained in the original iris dataset: sepal width, sepal length, petal width, and petal length. For transfer learning to be effective the input for the newly trained neural network most closely conform to the original neural network that we are transferring from. \n",
    "\n",
    "We will strip away the last output layer that contains the softmax activation function that performs this final classification.  A new output layer will be created that will classify the 4 new flowers.  Only the weights in this new layer will be trained.  The first two layers will be marked as non-trainable.  The hope is that the first few layers have learned to abstract the raw input data in a way that is also helpful to the new neural network.\n",
    "\n",
    "This is done by looping over the first few layers and copying them to the new neural network. We output a summary of the new neural network to verify that the previous output layer has been stripped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 50)                250       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 25)                1275      \n",
      "=================================================================\n",
      "Total params: 1,525\n",
      "Trainable params: 0\n",
      "Non-trainable params: 1,525\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model3 = Sequential()\n",
    "for i in range(2):\n",
    "    layer = model.layers[i]\n",
    "    layer.trainable = False\n",
    "    model3.add(layer)\n",
    "model3.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To complete the new neural network we add a 4-neuron classification layer and compile for classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 50)                250       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 25)                1275      \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 4)                 104       \n",
      "=================================================================\n",
      "Total params: 1,629\n",
      "Trainable params: 104\n",
      "Non-trainable params: 1,525\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model3.add(Dense(4,activation='softmax')) # Output\n",
    "\n",
    "model3.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "model3.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we generate some training data for the 4 fake flowers, and train the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1d1ff923e08>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.array([\n",
    "    [2.1,0.9,0.8,1.1], # 1\n",
    "    [2.5,1.2,0.8,1.2],\n",
    "    [1.1,3.1,1.1,1.1], # 2\n",
    "    [0.8,2.2,0.7,1.2],\n",
    "    [1.2,0.7,3.1,1.1], # 3\n",
    "    [1.0,1.1,2.4,0.9],\n",
    "    [0.1,1.1,4.1,1.2], # 4\n",
    "    [1.2,0.8,3.1,0.1],\n",
    "])\n",
    "\n",
    "y = np.array([\n",
    "    [0,0,0,1],\n",
    "    [0,0,0,1],\n",
    "    [0,0,1,0],\n",
    "    [0,0,1,0],\n",
    "    [0,1,0,0],\n",
    "    [0,1,0,0],\n",
    "    [1,0,0,0],\n",
    "    [1,0,0,0],\n",
    "])\n",
    "\n",
    "model3.fit(x,y,verbose=0,epochs=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can evaluate the in-sample accuracy for the new model, that contains transferred layers from the previous model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Accuracy: 1.0\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "pred = model3.predict(x)\n",
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y,axis=1)\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(f\"Training Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 9 Assignment\n",
    "\n",
    "You can find the first assignment here: [assignment 9](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class9.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.7 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}