t81_558_class_03_2_keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_03_2_keras.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 3: Introduction to TensorFlow**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 3 Material\n",
    "\n",
    "* Part 3.1: Deep Learning and Neural Network Introduction [[Video]](https://www.youtube.com/watch?v=zYnI4iWRmpc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_1_neural_net.ipynb)\n",
    "* **Part 3.2: Introduction to Tensorflow and Keras** [[Video]](https://www.youtube.com/watch?v=PsE73jk55cE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_2_keras.ipynb)\n",
    "* Part 3.3: Saving and Loading a Keras Neural Network [[Video]](https://www.youtube.com/watch?v=-9QfbGM1qGw&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_3_save_load.ipynb)\n",
    "* Part 3.4: Early Stopping in Keras to Prevent Overfitting [[Video]](https://www.youtube.com/watch?v=m1LNunuI2fk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_4_early_stop.ipynb)\n",
    "* Part 3.5: Extracting Weights and Manual Calculation [[Video]](https://www.youtube.com/watch?v=7PWgx16kH8s&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_03_5_weights.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: not using Google CoLab\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3.2: Introduction to Tensorflow and Keras\n",
    "\n",
    "![TensorFlow](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_tensorflow.png \"TensorFlow\")\n",
    "\n",
    "TensorFlow is an open source software library for machine learning in various kinds of perceptual and language understanding tasks. It is currently used for both research and production by different teams in many commercial Google products, such as speech recognition, Gmail, Google Photos, and search, many of which had previously used its predecessor DistBelief. TensorFlow was originally developed by the Google Brain team for Google's research and production purposes and later released under the Apache 2.0 open source license on November 9, 2015.\n",
    "\n",
    "* [TensorFlow Homepage](https://www.tensorflow.org/)\n",
    "* [TensorFlow GitHib](https://github.com/tensorflow/tensorflow)\n",
    "* [TensorFlow Google Groups Support](https://groups.google.com/forum/#!forum/tensorflow)\n",
    "* [TensorFlow Google Groups Developer Discussion](https://groups.google.com/a/tensorflow.org/forum/#!forum/discuss)\n",
    "* [TensorFlow FAQ](https://www.tensorflow.org/resources/faq)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What version of TensorFlow do you have?\n",
    "\n",
    "TensorFlow is very new and changing rapidly.  It is very important that you run the same version of it that I am using.  For this semester we will use a specific version of TensorFlow (mentioned in the last class notes)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Self Driving Car](http://imgc-cn.artprintimages.com/images/P-473-488-90/94/9475/CFB6500Z/posters/paul-noth-does-your-car-have-any-idea-why-my-car-pulled-it-over-new-yorker-cartoon.jpg)\n",
    "\n",
    "[Wrong version of TensorFlow?](https://twitter.com/reza_zadeh/status/849160032608440320)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor Flow Version: 2.0.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(\"Tensor Flow Version: {}\".format(tf.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing TensorFlow\n",
    "\n",
    "* [Google CoLab](https://colab.research.google.com/) - All platforms, use your browser (includes a GPU).\n",
    "* Windows - Supported platform.\n",
    "* Mac - Supported platform.\n",
    "* Linux - Supported platform."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why TensorFlow\n",
    "\n",
    "* Supported by Google\n",
    "* Works well on Linux/Mac\n",
    "* Excellent GPU support\n",
    "* Python is an easy to learn programming language\n",
    "* Python is [extremely popular](http://www.kdnuggets.com/2014/08/four-main-languages-analytics-data-mining-data-science.html) in the data science community\n",
    "\n",
    "### Other Deep Learning Tools\n",
    "TensorFlow is not the only only game in town.  These are some of the best supported alternatives.  Most of these are written in C++. In order of my own preference (ordered in my own estimation of approximate importance):\n",
    "\n",
    "* [TensorFlow](https://www.tensorflow.org/) Google's deep learning API.  The focus of this class, along with Keras.\n",
    "* [MXNet](https://mxnet.incubator.apache.org/) Apache foundation's deep learning API. Can be used through Keras.\n",
    "* [Theano](http://deeplearning.net/software/theano/) - Python, from the academics that created deep learning.\n",
    "* [Keras](https://keras.io/) - Also by Google, higher level framework that allows the use of TensorFlow, MXNet and Theano interchangeably.\n",
    "[Torch](http://torch.ch/) is used by Google DeepMind, the Facebook AI Research Group, IBM, Yandex and the Idiap Research Institute.  It has been used for some of the most advanced deep learning projects in the world.  However, it requires the [LUA](https://en.wikipedia.org/wiki/Lua_(programming_language)) programming language.  It is very advanced, but it is not mainstream.  I have not worked with Torch (yet!).\n",
    "* [PaddlePaddle](https://github.com/baidu/Paddle) - [Baidu](http://www.baidu.com/)'s deep learning API.\n",
    "* [Deeplearning4J](http://deeplearning4j.org/) - Java based. Supports all major platforms. GPU support in Java!\n",
    "* [Computational Network Toolkit (CNTK)](https://github.com/Microsoft/CNTK) - Microsoft.  Support for Windows/Linux, command line only.  Bindings for predictions for C#/Python. GPU support.\n",
    "* [H2O](http://www.h2o.ai/) - Java based.  Supports all major platforms.  Limited support for computer vision. No GPU support.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using TensorFlow\n",
    "\n",
    "TensorFlow is a low-level mathematics API, similar to [Numpy](http://www.numpy.org/).  However, unlike Numpy, TensorFlow is built for deep learning.  TensorFlow works by allowing you to define compute graphs with Python.  In this regard, it is similar to [Spark](http://spark.apache.org/).  TensorFlow compiles these compute graphs into highly efficient C++/[CUDA](https://en.wikipedia.org/wiki/CUDA) code.\n",
    "\n",
    "The [TensorBoard](https://www.tensorflow.org/versions/r0.10/how_tos/summaries_and_tensorboard/index.html) command line utility can be used to view these graphs.  The iris neural network's graph used in this class is shown here:\n",
    "\n",
    "![Iris Graph](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_graph_tf.png \"Iris Graph\")\n",
    "\n",
    "Expanding the DNN gives:\n",
    "\n",
    "\n",
    "![Iris DNN Graph](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_2_graph_dnn.png \"Iris DNN Graph\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using TensorFlow Directly\n",
    "\n",
    "Most of the time in the course we will communicate with TensorFlow using Keras, which allows you to specify the number of hidden layers and simply create the neural network.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAIIAlgDASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDyuiiivROUKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiilAJOAKAEoxnpUixf3j+VSABegp2Jc0RrET14p4RV7U6ighybCiiimSFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAIQCMEVG0X938qlopDTaK5BBwRSVYIDdRUTRkcjkUrGikmMooooKCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACjGelPWMnrwKlVQo4osS5JEaxH+LipAABgClopmbbYUUUUxBRRRQAUUUUAFFFFABRRRQAUUoBJwKeEA6800rickhgBPQU4R+pqSiqUUZubG7Fpdo9BTgpNLsPtVqD7EOfmM2j0FIUWnkEdaSk0NSZGYz25phBHWp6CAetS4lKb6kFFPZO4/KmVDVjRNPYKKKKBhRRRQAUUUUAMaMNyODURBU4NWKQgMMGlYpSsV6KcyFfpTaRqncKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKkWLu35UCbSGKpbpUyoF9zTsY6UU7Gbk2FFFFMkKKKKACiiigAooooAKKKKACiigDJwKACnKmeT0pypjk9afVKPczlPsAAHSiiirMwp6r3NIoyafWkI31IlLoRzS+SudhYdzkYH1qQFWGQRilYBgQRkGsq4jFvMiRsyRyHaQOle/h8LRrU7Ws1+JjFcztfU01Kuu5CGHqKQqD7UseAgA6CnEV5+Kw6hNqOwRmRFSKSpaQqDXA6fY2U+5HTWUN9aeQRSVm10ZafVEJBBwaSpiARg1GylfpWbjY2jK42iiikUFFFFABRRRQAEAjBqF028jpU1FIaditRT3TbyOlMpGydwooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAoooxnpQAU5ULfT1p6x45b8qkp2Ic+wiqF6UtFFMzCiiigAooooAKKKKACiiigAooooAKKKeqZ5NCVxNpDVUsalVQopelFaJWMpSbCiiimSFAGTRT1GOe9VGN2JuwoGBSjrRSiumEbsxkxay9Rj3Oj5fjrjoPc9q1Kq3FtHMdzKCcY5r2sFs1e1+vYinNQmmylY3jea/muyqBwsi7SPqK1gciufmh2XKxoM91QnAzWnb3QeJS3ykjoa66mHdRckneS38/M2r01pOGzL1RySLGVBB+ZtowKpy6kkblCCeM5zgVCss11dqUUIYgThj+HI7VzRwMOZxm/lfUzjSna70RqEVGRj6VIm/wAtfMxvwN23pn2oIyMV4demlJoqEiKgjIwaCMHFFchsRMmOR0ptT1Gydx+VQ49jSM+jGUUUVJoFFFFABRRRQAVC6beR0qaggEYNIaditRTnXafam0jbcKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiipFi7t+VAm0hqoW+lTKoXpS0UzNybCiiimSFFFFABRRRQAUUUUAFFFFABRRRQAUoBPSnKnc/lUgGBgVSiQ522GqgHuadRRV2M27hRRRQIKKAM9KkC4+tVGLYm7CKvc06iitkraIybuAFOoorrpU3sjKUgqNuhpxaq886xjlgK9jC0XfQhJt6FK+jDgnJB5H15qiJN4LOWCk4YL379T+P6VLc3DSKAFPzjK85yM4/pTJkRLdHAwH6JnJVu+fwx2Fb4qtBSgoPXun0vb+ux6tGLjG0iW3tfPbc6qF/ugYrWtreOFQEXGKoWVwpAXOSOuB0rUQ9K0xUfZw5YbHDiJzbs9h9Np1Ia+brx1IgxjDIplS0xhjntXDOPU3i+g2iiisyxrJu571ERg4NT0jKGFS43LjK25DRSlSp5pKg1CiiigAooooARgGGDUBBBwasUyRdwyOopMqLsQ0UUUjUKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigApyoW6dPWnLF3b8qlp2Ic+w1UC9OvrTqKKZmFFFFABRRRQAUUUUAFFFFABRRRQAUU4IT9KeEA+tNRbJckhioT7CpAoXpS0VaVjNybCiiimSFFFKFJppXASnBfWnBQKWtFDuQ5dgAx0ooorRLsQFOFAGKK6KdPUzlICcVVurjyYiwP/ANao9QEzKPLfag+9gc1ktKSxDozliBmRto7f/X7969ulRVOHPZvTp/XQ1o0Oe0mzQjviw2yYDgc81UvJ9zZVuen4f5xTZbeRpEKeWF2hQynIJHXn1ppWG3hDXeVxwVUc5zkZ+o3fgKcsfThRXNG769Leep1wowjLmQkk6WkEU7xgyZzgALnnOf1I4B/SqsGpmWaRJtxSRQirnOPTJP8Ah19KbeX9tcWyQIsiqhJHHfHXGee/0B71WsJIobgSysRtzgBc54P5H096+Uq4xvEQUZrl0v8Arfbodah7rbWpr/LFcqke7KNg5OR79h05rZjlBUc1kCOG7lM6XICOGI45BHbA/OmC6LfIHIB43Hj/AD+dfW4OtRq0HzO1tXf9O/4nFXouo1Y3TcRqyqzcscCpqw4sNe7Zc5HTcAMkHqBjgcf55rbByK5cVSi6anBbnFUp+zaQlHWnEU2vFnGzKTuRkY+lJUvWmFcdK55QtqjVS7jaKKKgoCAetRMu0+1S0UmrlRlYgopzLjkdKbWbVjVO4UUUUDCiiigCGRcHI6GmVYIyMGoCNpIpM1i76CUUUUigooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiipFi7t+VAm0hqoW6dPWpVQL06+tOopmbk2FFFFMkKKKKACiiigAooooAKKKKACinBCfpTwgH1pqLZLkkMCE/SnhAPc06irUUjNybCiiimSFFFFABQBnpShSfpUgGOlXGDZLlYaFx1p1FLitox7GbfcSilxS4rRU31J5kJiloorWMLbEN3CiikJrrpU38yGyN+RWNqG0Y5GeuK2JDgVj+ar3ZSRRIHwFH93J56jrj0r1vaeyoNtXvp/XY6sJF81+xWv8Az5IYp4nWOMkFmGRg9CemcA46Z7deAMm5uZbuQSSnLAYq7qFxGqNbwjYeFkGAQ3f7319u3WsyvhsyqL2rhFvpdX0uuny/O569KOl/uCiiivNNixZ3RtJ/MC5yMH1x/L862LW7h1B9ph8sJ0bOQOn5cKeua5+tzTN0+nsglCsHIJLc7cDP4YHTPY+lenltV+0VOT01e39evyMKySXN1LFtFJPIZCwJPGTzjGOQfWtyJdkYUdAMVk2IaOUoG3KADnHXPpWsp6V9pVhBUUqa0/y02/yPHxUm52Y+kIpaK8WpT6GMWNopSKSuWUWjVO4hUGmFSKkorNwTKUmiKipCoNMKkVk4NGikmJUTLjkdKlo61DVy4uxBRSsu00lZmydwooooAKZIuRn0p9FA07MrUU512tim1JsFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABTlQt06etOWPu35VLTsQ59hqoF6dfWnUUUzMKKKKACiiigAooooAKKKKACinBCfapAoFNRbJc0iMIT7U8IB9adRVpJGbk2FFFFMkKKKKACiilAJoASnhfWlC4pa1jC25m5dgoxRTq6KcOZmcpWDFFZl3eTQTPGxXBGVK8Ef5/wA+lC6hlFUDMpAwp4yT0r1qeDi7rm1Q3RqNKS6mizqgyzAD1JpQQayJbl7rEYVgQRu25OORnBFSafNJsIkYEDgd/wBe9axwsHLkjq7Xv0CVCUYczevY1KTIqPcKa8gUZqo4V3sc9myUtTS1Zn26RkZwvyrwSeBnOMZ/WhL7LFXI3DuMgH8DXRSo03LlT1/yN/q00rsuyNxj86yLzbuDEcZ6VYkuSUyFyuM5B7c9fyP1qhNN5hrtXs3CUeZdVudFCnKLuRaoPOt4ZIgSsa7XIPT0Pr3+lZNbkT+cotpfmj2kKOmCTnnHUZArMvLOS0cblYIxO0sACcYz396/P81oP2rqLXv5dF8n/wAA9Sm+X3WVqKKK8g1CtmwQRWnmECObja7rj5c54HftyR/KqVnp01yUdlZLdicy44GP84q88wZEiVAsacAde3rXt5PheaspTVl/Wv8Al5+RhUfMrR6bmjYZWNQSSBzjPStNWGKwoLkIBnir8VydgcqQh/i7dcc+lfdVoQaUU0r7f8A8mtSk22aIalyDVI3SquefwqvDqGPnfcY2cjd2XivPq4WMfidr6GMaM5JtI1qKjjlWRdwOR/KpK86ph3FkXtowxSYpaKx9g29h8wmBRioLvYYTuAwOearabdtKFhIUbE/P6V0Sy9ez5lvr+BpFScXJdC4y9xTalIqNhg14tSFmaxlcaRkYqEjBxU9Mde4rGSNoO2hHRRRUGoUUUUAMkXK59KhqzUDrtbFJmkH0G0UUUiwooooAKKKKACiiigAooooAKKKekeeT0oBuw1VLHiplQL9fWnAADAop2MnK4UUUUyQooooAKKKKACiiigAopQpPSpAgHuaaTZLkkMCE/SnqoX606irSSM3JsKKKKZIUUUUAFFFFABRQAT0p4UDrVKLYm0hoUn6VIBjpRRWsYpGbdwoopQKtK7sS3YBS0UV20Y66GUmUb941hPmdDxVC3VyUMgUwryjE/dPUDI/D/PFX7uEyZIbjaVx25rFuF2TfOF652gYH0r169OpOj7qVl167f16ndhOVx5b6ly7ibCiBAgTDOfulTz6/09B7VXgacHZGvzDHB46/5/Wm6vLcx7HjdfLxkP8ALlu2c9ScdcVVfV3zPsXO9Qqs3Xjufr/XvXjxzNYSUoOVr2e1/u/4NkdUYOUFpf8ArqbEd8CSPmAJO3PcUy6ufkIBGfT1qrLMlzAs4neV+h3YGPwH4f5xTHCR2vnzM43Z24GcYI7d85r24ZjRdHnlv5drb/dqYrDpNSsWPPjtbAzKZFyTwOC46Dntzjv68VHuW9tmmgEgEZw24/KBjrnA9PwGKrX8ll5KxKzPjLIyjBII43ZHr+n14zY55YlZY3KhiCceoOQfY18rWzOdGr7r038767/h8ux0xp395b+ZrSXIgkiaJvMDhd4yFTJyRkjHHse2emBTJ9QgjRI7di0LcuhHIwfoOT9T/KsiiuL+1Kyvy9bfpf77a3+Rr7FXu/6/4boaR1iTyyscMatnhiAcLjpjH4/XtTJNXupgRL5cgOcBl4HQ9O/TvmqFFcv1uvr72+ny+4fsodhzOrKAI1UjuM8/mackgQf6pCw6M2T+mcfpUdFYqck7/wCX+Rduhbi1K6hi8pHATB+XaO4xn/PpUzaruLf6NF8zZ5HIX0B/rWdRW6xddO/MyeSPY24tQtbrMM2YlyBGxGSPqeOOg5p1vdrdvIAobZHhVbpgc9RyBwPb6dDhU+OWSI5jkZD6qcV1wzaun735f0vnvt2IdGPQ29kuws7iMxuoyWA69859qk85bxjtZvKhUjzNuQxOM/Qc/hx61jS3089uIZZGcBs5J6/X1q3pUcfk3DtKPM2jYnmbecnn8MD/ACa9CWaSxVeOl1b/AD0/4bV9DN0+VXf9bG3p8mxPL3KyrjDKODxWhvrBhMtuPmBDDHyYOef6+3t9M2vtoKjB69hX1cI0q0bxle255tag3O66mgbmIMV8wZHUZ6UomVhlWBHrmsZDvdjNGBCcyEsSCcDtz1xk/TNE8rpM0sAYwuxCtnIY5yT/AJ9K5fa0IVXTnpbrdW7j+qX0TLOoS5XYSQh6kf4VJpZKq4cAMG5A7e3/AOuqEq3PmAFMlfmyBkcdalkBt5h5TuA+Cq9W98k9Dn61c6tGVblg73VtO9/+H62NXS/dchucGmsMrVawhmTfJM+5nx+GKtHrXiY6jGnPlickdHZO5FRQRg0V5R0ELDBxSVJIOM1HWbVmbxd0FFFFIYUyRcjPpT6DyMUhp2ZWopSMEikpGwUUUUAFFFFABRRRQAUUoBY4FTIgUe9AnKwiR45PWn0UVRk3cKKKKBBRRRQAUUUUAFFFKqlqAbsJ1p6x+v5U8KF6UtWo9zJz7B0oooqiAopQpPal2e9NRbFdDaKfsHrRtFVyMXMhlFSbR6UtP2YucjCk04IO9OoqlBITkwoooqyQoopQKpRb2E3YAKWiiuiELaIzbuFITxQTio3kC9TivRw9Bt6EbjZOh+lZTCN7mVZHVOmN/Tr/AJ/Wr7zKR1rJvJFZuvNetLD81Fweh14ZNSKWryTMUBSRLcnKDJ2njsPWsyt+KA6hbtAYxkYPnEZPHQZ7elY9zHBGxEMxfJyBjgL259fw/wAK+BzLDyjUdXpt8/JdvS/metTkl7nUsafepaRSeapkRuAm7oeOf0/lUE95JM8pG1FkOWVRwen+A/yar0VxvFVORU1ol+O/X57bF8kb3CiiiucsKKKKACiiigAooooAKKKKACiiigApVYo6sMZByMjNJRQBet74CSR7lpGLdPLCjnuc9u3Tr3q2imVwbY5XG9SWGQB6+nSsarVldNBJsZyIZPlcHkAeuPUV62BzGpSfs3Kye79XrcylBJXQt3eyyyyJnEZblAQRnGCf/r0ltf3FqzGOQ/MMc81Vorg+s1ebn5ncvkja1jcspZktmeZ1Jb54yArHIB6n8enUfznht7gx5TDc7treuOv1qK2gZrBYc7PLPzBv4m5OeCcDHfvitexQhG5JG4gZxkduxr7PLORUH7S7ku/yf5+X3nnYipyK8S1CpWMbic+9OPWnU09a4sXPmlc8+G9xjjmm09xxTK8yaszpjsIRkYqGp6icYY1nI1g+g2iiioNAooooAilHINR1O4yh9uagqWaxegUUUUFBRRRQAUoBY4FABY4FTqoUYFBMpWBVCjApaKKoyCiiigAooooAKKKKACigAnpUqoF5700rickhqp3P5VJRRVpWMW2woopQM/SqSuIACaeFApabNJ5MRkKlgOTjriuqjQdSXLHcylMdRSqwZQQetLitp4acNGZ86G0UuKMVn7Nj5kJRS4FLij2TDmQ2inUU/ZeYuYTFGKWirUIoXMwooorRRb0RNwpCcClprGuujS1sS2ULnUBFKYwhJAyeO3tVVnkuHy7IqEbSM5Iwc9Mj0H50/UARMj4GF7nt71BLI8VnOYCwkQHIVSAgB6evTOT7CuvFTdKLjsmtNN/Lfy8n2PQowjyqUVqQK0/lk7DtxngY49cenHWkMssVk08aK2xsPlVPB/X+f+FX+1i14JWjHlY27eCQMY6496df3qqXgiiEZ2gHD525wSOnXt1x145rx8RnMa1Fx5vh8tb200/z+7Y7fZtPbcgvLwTKiRqqqEAbA4J4J689c96pUUV8xUqzqu82dMYpKyCiiisxhRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAOjCFwJCQp7jt6Vr/ZbCAb4/MuCSdpY4VTn26n9Kxqu2NysbeW6ocjCu+SF5z0APv2713YKrShNKpG+u/X07fgZzi3qbMCi4dXdVyBj1J+tbEQCgADgViwxXED4bAUZ+btxn8unf1rWt5N8YJBB7g9Qa+9cqM6VqO3X/gnjYpSve+hZpDQDmlrw69N6nPFjSMioqlqNvvGvNqLqdMGJUcg6GpKa/3axexrHciooorM2CiiigAquwwxFWKilHzA+tJlwepHRRRSNAooooAnRdo96dRRVGG4UUUUAFFFFABRRRQAUqqWNKq7j7VKAB0pqNyJSsIqhRS0UVoZBRRRQAoGTUgGBikUYHvS1tCNjOTuKOtJIiuhVgCD1BpRTJpPLiLbS2Ow616WCg3NW3MJ6vQyLjfaz7IpWCsMMCc7Rnr+ta8R+Qc9qw53a5uAfLbaDkrg5I55/wA+tasEqmMYIxivcnD2kZRXRr8jevF8kW9y3RUYf3oEqsSAwOOtedLCPsct2SUE4pu6qt1cGFNwGeQDjrVUsJzMaTk7InjuY5JHRTyhwalrAjbOoMwcHnkqTj8+/wCVbSOCOtaywylDnitDStS9m1ZktFN3UbvauZ4fyMbjqKbu9qCSatUmFxSajY8YpScVTvJmjQbcZyM5OK7KFK2pUIuTsiG8mQAqepHSs63w0xTa53jHyPtI5zU6zx+SzTc5yJDtzgcgYP8AnqOtVDcWv2aXyGmLKck4Xp0HUepA/I9sVzZhmEFD2TjbS+tk7d7eh61GnyJopXtvb20zJFMZcccdjxnP6/561KdJI0rl3OWNNr4WtKEptwVl0/rU7oppahRRRWRQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAalkBJbySMS7sdoDnGMcnHPPGO3cfjq2d3HHGFLAY9awNOkEd6gIJWTMZwOfm4qydRhiZhbpJjGBITgnpnI6Y6/pX1OV5rToUHGpv3f3+bOStR9o+VnSi6QpkMMetRWt6z3HluDtc/uztxnuay1zdEtDIrRu2WCjPl59R7Zq7aSrHeODCVYsEGFxgY4/QZr23WpV0o07aq79ey/z8jhlh1CL6s1jUb9RUhpj9K+drqzZjTYykb7p+lLRXKbEFFFFZHQFFFFABTJBlPpT6QjIIoGtGV6KKKk2CiiigCzRRRVGAUUUUAFFFFABTlTPJ6Uqp3P5VJVKPczlPogoooqzMKKKKACnKvc0KueTT60hHqyJS6IKKKUCt4x5mZt2FpGGRS0HpXoYfR3RjIyNRtmdwyqCo64OCaqCSSI7Q4UHOCewz36fXpW44zWXeRqUPY4zXvU4OonKMmm15fLc66Fa6UJK6IjdSxxAt3xggHGPXJoguWglJlG3cAaZaTMJz8v7tVJKqTwPbn/635Upnsrtpg8gikV+WbaN3pjPP+cn286WYSpVFGr0uui6d9v+Bc6nTjqraM0Uu1fjoR1yKqXsm9ODgg5B71TkDwPnejBv4k+7nuP8+opDHMyBjjDDIy4yfw/A13RxmH5Fz6NrVfmRHDcsuaJND++aZo4SHA3ZB4B/oO/4VZhuZIwhk6MOo5HUjH6VVeRNPgS48tiz9MnOPYj0OOv19KbaTi8sfKWPfOg/vHPUc49BnA988da8qlmUYVeS+nZ9v62XkzSpTUltoaL33zBE27iCck4HAzUlrdGVTuPIOMjHP5ZrKgEscx3KuzgPuwVIz1+nTmp3/wBFcSRj903Tacgfj3PBr0IYqnVrOH2ej8zGeGio2W5sB/xpS1Zkd78yqwKluRkY49atCbI+9XQqMZax1Rxyoyi9SZmwMmqF467CG/Gpnk9Dk1QllQTHzMEY4JGRnP6961f7mDna9jWhTvIYk7PEY58+RsI3FfujGOP1/wDr1hShFkZUO5Rxn1961725u7dVBaOSFl5HKhvwzk+vHrz04xa/P82rqpU219OnS2uzPXoxsrhRRRXkmwUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFACo7RuroSGU5BHY1YFjcOu+KJ2j3YBIAPqMjPHBH51WrVsTFPaiLzER05OVI4J6579QOcdPxrrwVGNeqqU3ZMzqS5VdFyGERmJbSMYAUuxADMccnnJH6itWztnjbzZWG8jGFAAx26Co7FUEQIAGQDitAdK+1q4eGEhyw3tY8avXcnygaa33acaa33a8Grq2RDoR0UUVym5C33j9aSlbhjSVkzdbBRRRQMKKKKAK7DDEUlPlGH+tMqTZbBRRRQMs0UUVRgFFFOVCfpRa4m7DQCelSqmOT1pQABxS1ajYzlO+wUUUVRAUUAE9KeE9aai2JtIYBnpTwuOtOAxRWsYJbkOVwoopQK1jFyIbsAFLRRXVCk+hk5BSMe1BOKguJTHGWAyRXo4eg20kTq3Yc7AVk3jAttLFRjGew/D8/89XvfjcQRgj3z+oqnNI0gDlSIycbuv+TXqr2dOldysn1X6HbQoyUtUPjJNmyW5USNwVPVsg9/w6f41z5JYknqeTXR2kq2QeSSfMSnDKoJHPGTxxWJezRTTOY4wMtkvk89f/rdeeM96+Iza0knzbdO/n+mu/Q9Kk/eaS/rsaFtfQJZRi5+dsHBB3N1Ixjt1B544PrVSXVbmQLtbYy45XA6Yx/n6elUaK4J4+tKKinayS030t136GipRTuyzPf3NwhSWXcpOcbQBn/P9fWmRXUsCYibyznJZeGPTjPpxUNFc/t6vNz8zv3vqVyRta2heTVJkikTJDOSxZTjJOcn689sdKsWFxbvbG2ncq5ffvI9jxn9fxxWTRW0MZVi43d0tPk3fp+ZLpx6aHTTiG4VntnBkg+VxjB6kc9sdfoP0Z5rwv5coAYDkdcVjWd/NZMTEcA9QO/Bxn16/wD6q0zKt5arct8smNp245xj8+v+R0+oyrN1ZQfq1287v+m2csqDWj2Jnn+XqPoKjt/3s/Rml6oq8ZP17VW3VLFIscMshLIRx5gG7bn24/D3r2sfioewai9/6/roKNPlTKOpz3Ul0UuWyydAOmDyD+tUqdJI0sjSOcsxLE+pNNr82rVHUm5vqdsI8sUgooorMoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAq/YTiK3nVpxGpKgAk8568AdOOfoOvFUKtWlotyGZ5hEqsASVJ6g+n0/WujCSqRrJ0leXQidram1b3D/ACxw5bcNyFgFyPxrWsJHltEeT73esyGLMawwRyMg48wvyw9DwDj29q17aLyIFj3FiO5r7arVqzo/vlZ2XTr1PHxPJ03JD1pj9KcetNftXhVXuZw6DKKKK5jYif75ptOk+9Taye5vHYKKKKBhRRRQBFKOhqOppfu/jUNSzWOwUUUUFFmlClulOVM8npUnStFE5ZTtsNVAPc06iirsZt3CinBSaUKB71Sg2S5JDQCacFA606itFBIhybCiijFWlfYkKMUuKWtY0+5Dl2DFFFITgV10qV2Ztlae9WCULIMKRw3v6VKkodQwPBGao6lEXUN1VeqjvVeC5MI2O2R1U56ivbhh4OytutP1NlRUqalHc1i3pVeZgVI7YqB7olgqYZm4AzjNVTeB1zXTSoqMrdQhRluQSMY5Nh2lC2TuUHr15xmn3l59ijzDFmKQ4BI+Xp7r7tjn147VGixzOoyxkJHyEYDc9M544qDUr7dbxwJDLEMdHJ6eoPcHkV87nNWEXJx06bddOnn8vzPThG9kynf3Ru7pmGNgOF4xkdP6VVoor5CrUlVm5yer/r8DsjFRVkFFFFZjCiiigAooooAKntrya0LGIgbhg8f16jt+VQUVUJyhLmg7MTSaszdnhZ4Y7pFISQDg9jj+VV7svFpwQyMBI2THg4Ppz07fpUmiXSZNtOWKn7gLHA/DPr6D1o8Q4S4hiX7qpnrx1r2p4mpPDOotrWfk3pb9fmc6l76pNefyRjUUUV4Z0hRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABV+waLyXUyFZC2QoB5wOO/uaoVu6YvlmFVSHzNufMxknJPQjHrg/SvTyilUqYleztda6/IyrS5YXNbTpN4dGwHQ4IBzgfX860Kr2lstuhA6sdx+tWD0r6XG1XJ+89TwpuMptx2G0x+tPpjferxamxrHcbRRRWJoRyfe/CmU+T734Uys3ubx2CiiikMKKKKAGyfcNQVYb7p+lV6TNIbBRRRSLNCinBc08LjoK6owbPOckhgT1p4AHSlxRiuiNF9EZOYlLgDqaWqt7IFt2PUrhgPoc13YbB+0kkyOZt2RawKKghuEkAwwPGetTZFaVMFKDtYlya3FopCwFQpco8jxg/MhwacMHOS0FdvYnopu8e1IX960jhX2FdjicU0tVS8uTDCWQjIIqFb0MmScEdcGvQo4T7y1Sk1zE1w4wc1lxFWeVC21uNn1z1Hv/nBonmaWXajctgDnvT4LfZumnT5IzkPuADc+/b06VOPrQjT9h9q6/q/3/cehRp+zjdjI7UoyyPJ8ik7iuVIx6ZHrTRbwu0gjmBKjhW+Ug8cH+X1rLnvXe4aSFpEBHUsc9c59uf8APWq3mPvL723k5LZ5z6181Wzm07JuVtL3tp5ad9dfQ7I0pPVux0EMEtuJZWUBoxlTjPJ447HrWDcSCa4eQdGOe/8AUn+daEOo79PktnC8RnJc53HtjPQjjv249Dl1wZjXVWMHGV76v8rNd1+uhVNSu3JBRRRXlmwUUUUAFFFFABRRRQAUUUUAOjdo3DKSCPQkfyq3qQfzYndifMjDjPoc4/lVKppgRHASPvR5Bz1+Zh/SumnWkqE6V9HZ/c/+CS17yZDRRRXMUFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUVJFE00gUcDPLdhVRi5NRitQJ7CAyXCuYiyLk5x8uccAk8YzXSaXaiOCNipXAIC5Pr/AJ/IVV0+wXamVIjHIVjnJP8An07CtxBgV9pgsAsFR5qi99/gePjMTze5EdSGlpD1rkryvc44ISo2+8akqM9TXDU2N4biUUUVkaEcn3vwplPk+9+FMrN7m8dgooopDCiiigBG+6fpVerNVqTNIBRRRSLL2+87W9v/AN/z/wDEUnmX2f8Aj3gx/wBdj/8AE1KH9aeDXq0q0Uv6/wAjypRfb+vvK5kvv+feD/v8f/iaYZb/AHgeRDjH/PU//E1czS5rsp4mC/pf5Ge3Rf18yiZL/wD54Q/9/T/8TWTeyXReTzI4w2zgeYfQ+3NdJVO+gV4HIUbjwCfXtXbSnGtF01Jpv0/yX5mlGqoTV4owtOlvS7tsDOSdxc7T/KtQTajzmCLrx+8P+FXILRIiWA+Y9T61Y2irp8tCmqTm5W66f5MdbERnK6ijGE2pFW3QR9T1bH9KrRvcCbdGimU53jeePT+H+proSgNV47NY5pJO7nP0raNWDt77VvTX/wAlCNeKT91GS8+obk3RqDngBzyfypZJ9R8klolHuGPH6VueWPQUhQdxTjVi38b/AA/+RF9Yjp7iObeS6dlEygIcZ+Y9PypsoIn2lI1TJztkJ7nvj+lbN9bF4SEXJyKgWwUR/Moz/KqnhnW2qyWndfdt8zqp4mCje3yM+2G4kssZcfcDSEZP/fPNQ3sl8086+XlAuXAXcq5XJ5I45yfr9KszRCCYEAHaQQD3/wA4qeKVbtXhnEe1jhARluuTyOf5ZryMbg6kW488rX+L5PR6barbudHPf30v68jmqKmuLcwysisJAoyWXt9ahr46pTlSk4TVmjsTTV0FFaFvpzNZvcsoZfLYjGflIz16enr/AIHPq6tCdJRc1bmV16CjNSbt0CiiisSgooooAKKKKACiiigAooooAKM0VLKcxwcKMIenf5m61SV02IioooqRhRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUATQLbtnzpHU84AXjp6/wD1q1rNEG0AAKeRjo3Xn371h1Nb3DQOOTsJBZQeuP68mvWynMKeDrc04Jrv1RlVpucbJnaQ42jFWB0rH0++D7ULh89GXoT/ADrXU5FfX4lxqRVSDumeBVhKErMdTT1p1IeteFV2HESoz1NSVGeprkqbG8BKKKKyLI5PvfhTKfJ978KZWb3N47BRRRSGFFFFABVarNVqTNIBRRRSLNCgHFRq/Y/nUlbJnC1bceH9adUVAJFaqb6kOPYlzQwVxhhkZzzTA/rTgwNb067i7xZm4D6KbRWjrN7kcg6im0uaarCcBaKTNLW8arJcRCAaYV/OpKQjiu2jXZDVijcRgg8VlxIgklkcZKAMo/H/AD/ga0dRl8tQo4LfxdhVOCEzfPyqqfl5/X6/59K9SrF16SpJ76+i8/U78O+WHM9hiXMbuN4KFiSZGPK8k9gD1/maYDaRyyyKpdz825uNze2PxP6Vca08t1eNV3KcgHpmqpsyq4PWud5XGUrKWnfRu+35G8a0HqOSdrqCaExjDAbFUZ5HbHfgVz80flTOmc7TjOR/QkVtxbIZUfe3mgjAIwoOe5z0xVPUrOSNEnaaOXdwWUBeO3ux6+/TNfM5vhJRTS15W/W3V/f6/Lr005RUtNL/AJmbRRRXzp0hRRRQAUUUUAFFFFABRRRQAoG4gDGTxycVavyPMiQIF2RBDgY3YzzVrRLZJZ2lkcAJ2zyfXv8Ah+PtT/EKqLuJ0I2smcD69f8APpXoQpQjg5Sk/ebVvRP/AD/IwdS9VRsY9FFFeebhRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAF3T5m85YCVKMSQrLnLY49xkgV0unXYliRWdWcjPBHIrja6DTfM2QyIhEe3hQ3A5wep7kV9JkeKnJvDSa5d9b/AIHFjKUZQudHSGoredJ0yhBwcH2NSnpXZiqfK2eRG6dmJUbfeNSVG33q82psbw3EooorE0I5PvfhTKfJ978KZWb3N47BRRRSGFFFFACNwpPtVerDfdP0qvSZpAKKKKRZZpyuV+lNoqjBq5MGDdKWoKer9j+dWpdzNw7ElFAORkUVRmKGIp4YGo6KpTaE4pktFRhiKcGB9q0U0yHFodS5pKK0Ta2IaHUU2lBrop1dSJRKdzZG5lG9sRqOAOpNTRwiNAozwMZqeivUjjG1YlylZR6IiK1XmQBT6VcIqvcRs8TBeprtw1e8rNig9TDkjaWTKIxTPLAcY+tPv7a5mhEVs5MSdRz6DPO3/e7/AIVNJYMWYsepzgdP1qs4liQR7tqZ529T1/xPtWGPwVXFJz022T1S338/I9WnUi7KL2Mm8tWtLh4yG25+UkdR/jVeuhigh1FnidXDsRhsgnA7ZI449P16VjXdt9nkYK6sueBuBI9Pr9enIr4vG4J0m5w+H8n2/wAvI7IVLvlluV6KKK881CiiigAooooAKkiglm3eWhYKMsew+pp0FrLcMFjQknOODzwT/SteKL7DaqrEx3BBbK9cEjHP4f57d+DwM68lzJpem/p/VjOc7aLcdJO0NpFZoQdgG4gY59OOO9VL5zNYpnfuRuem3/PWnEFjkkk+9SoPMs5oWkfYfmKJyTjknnj/APVX0WKyx06N2ulvw/S3n8zJSilp3/4cxKKVlZGKsCrA4IIwQaSvjjpCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACr+nrGqmfzHWZW2rtIGMj8/WqFXLG6hgV0mWQq7KfkxnADev1FdWClThXjKr8JE78uh02moV8x24LtnaOn+etaNYsNw0cayxTxvAxwpbOfoemOhrWgmWeFZEPBr7HE+yqR5qOy/X7zw60JKXM+o6mP1p5601+1eHUWjHDcZRRRWBqRyfe/CmU6T71NrN7m8dgooopDCiiigBsn3DUFTS/c/GoaTNYbBRRRSKLNFFFUYBRRRQAoJHSpFYN9aiopp2JcUyeimK+eD1p9WncyasFFFFMQoJFPDA1HRVRk0JxTJaKardjTq2TT2M2rCg0tNpw5rohO+jM5IKawp1Ield1CbuZNELpn+lZN4oD5Ynb1wOp/p3/AM99hzWXeMoQ554r3aF5wavY3w7akV42e3smkhLMSc43fKDg9f04+lc8xZnJcksTkk9c1v2iP57oGCqylWYMCOf0P0+tJ/Z1nA0z3JLlnwq4IK/XBHXn6j07fI5lhHWmlB7N97Pz9ejt+B60ZqDd92VLXTY7qySRmEZ5G5eSTnuPoOg9RVOWyuIFUvGwz2we/T+v5GtSdzK4VY2QJxtJyQfT/P8AWlF1KqqNo3KMBsnP8/8AOBWksl9pCLUem662trbbX+rjU5LW/wAjGe3miz5kMiY67lIx/nI/OkEUhjEgUlCduR68f4it64RtQtUhSVS4OSB1Y8449Mnk/wCFR6fC1laG5V4xKwIALc9Rxjp7/Q15ay29Xljfl76J37W/4b8CvatLXcyhYz+U0jLtC5yCDnjOfp071bsLGB7f7VOzeXu2bcd8H8+349qu20k0k5RE4dtzBcjHqc/160+WNZGEMKqEBySvIJ5Oc9e9ejQyde2jBL3lvfVeT+XZ9e5EqrWj/AeUi05JBblt85BIzjGGPGByO4/ycRMj3EpldQGPXb0NTraOzo0jZKjGenHarYhAHINfT4HBww0feWupxTrJbO77ma8GB0xTbY+VOW3BXH3VYfK31/Q1ovEMfL+VUWhzccMVIwVwcEnPb3/wroxqjOhK/QqlU5tGY+oCX7a7TQ+Uzc7cY/H8aq1saj9pujHH5JCgAgyOMjPPXOOR69wfSsevzXG0vZ1Xa9n5deq6Xt3/AAR6NJ3igooorkNAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACrdnBFNDMZFY7SuCrYI68dMc/0+tVK19PAgtPMczEScBAwCkZ5/wDQRnpXbl9FVsRGDV0Z1XaOhbt45FIaLLqE2YcY3DGOcGtfTw4s08zO73qGxC+Uo4JwKvjpX2+LpU6MOSC6WPFr1nJ8rQHrTH6U801vu185V6ih0I6KKK5jYif75ptOf7xptZPc3WwUUUUDCiiigCOXoBUVPl+/+FMqWbR2CiiigZZoooqjAKKKKACiiigAp6v2NMooTsJq5PRUSvjg9KlrRO5i1YKKKKYgp6tng0yinF2YmrktApAcilroT6oya6DqQ8CgUyZXaIhG2sehxnFelhLSmkzGS6Gff3TRMFGAG43HtVBt0pztZlGRwfvc/hjj2/CnTiSO5CysGLHAYgHj6EVqW8CrGMDtXuJX5uZtRWmjt6v/AC+Z280aME0rszAlwYQE3IFGAAR069R3/wA9qbHA91MfMGCAMnue9bvlr6fpSJbpGWKrgtyayi6FNppf8Ht936mf1t2ehRjsQmSRnPsBVe8iEcZOK2dvtVS8t2kQbMBgwOcciuiliU/dj8jOnWbmnJmXAGtnm+eMuF27e+e4/Q9Papo4XnSNGG1VHKgdwTj9DUcaqt8VC7Ru6Fs/l61tJGAK5qUKcUqsr81/TX0OnEVnDbqZr2JU7kUEYIKHgN6frzUtnamNSWXGTwMdB+ZrQ2+1Lt+lN4mKk5LdnI68nHlZGEx0GKCpqTaaTGKhV7vcxuyFlDfWqF5EpUk1qEZ+tUb2JnQbQDk8g+ldlGqrG1GXvIz4/PeF0meQW+wlj/s9c/z+vNYMoQSN5Zyh5Ht7fhXQiEzWxSSSQKAd46hAAenP9fXrVBrC3W2laKZnYnbgxE7R19Pbr6Z+lfH5th3Od4JJat73T7f11/H2KU7Xv+BlUUrKyMVYEEdiKSvnWmnZnUFFFFIAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAACTgDJNW1vniTy40jKBshmjGSMAc+nQdKTT0L30RDFQh3lsdAOf6VK+m/MTDOrp2yPmPrx+f5V2YejXa56O9/63Ik43szWido/La2/eRuAS2D8pI6HHAPPua1LW6MvySIySYztKnp61hwrHZPtjhOVbEjH5icdcccVoWkMhu3PmbSrDICgcY4HHsa+woe2qQUK3a/8Awf6+Xc83EU4OLl+Jqmmt92nGmP0rxq2jZyQ6DKKKK5Dchb7x+tJRRWR0BRRRQAUUUE4BNAEDnLmm0UVJuFFFFAFhWDDNLVdWKnIqcEMMimZSjYWiiimSFFFFABRRRQAU5W2n2ptFANXJgQRkUtQgkdKlVg1WncxlGwtFFFUSKDg1J1qKnK2PpVwlbRkyVyQdaGOF5pKZcRmeFowQN3BJGcCvRwcoqaUnZGElcx7x1nuP3QLqMF8cjH+c/ma2IVHljHTFOSJEUAKOlP6V6dfF07NQ3fX00CpPnSilogoozRmvPdZt7kcoUhGaWinGtJMXKVorNIppJRyX9e1WQMUUVrLEyluN3k7thRRRWXtWKwUUUVcaqe4mhpFMdcipaYwruoVGmIyry3B3HOCetZ8G1ZSWeNUUZO/ODWjfs3mLGBkN1GcE+w96qzKYrSd4ciRhzwMfe4wfTqfwrfH1E6TjGN5eS23t/SPUw7airvcyb97WWdntywHX5uh6cD0x/T86dXhpVx9rEDYx1MgBIxjPHr3+tP1CwaIvMhiKgAsI84B4Bx2xk9OvPpXwdajXm5VZxt3O+M4q0UzOooorjNQooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooq3ZWhuHLYDKvJUOAx+nB9fxq6cHOXKhN2V2WNP2CCXG3zk/vYXAPBA5ye3GP586lnaB13hmG4Y4OKijfz5MG3QKc8BR745/Lp+Va1rF5cQBxnvgYr73LsM8LQcKkd+631/q55mKrWWm5H9iTy9uMj3qOztJFut5yqR8LwBuHTB79u9aQFLRiMWmrNarY4Y1Z2a7iGo36in1Gxyxr56rK5pBCUh+6fpS01/u1zvY2W5FRRRWZuFFFFABTJDhPrT6ilPIFIqKuyOiiikahRRRQAUqsVORSUUAWAQwyKWq6sVORU6sGGRTMpRsLRRRTJCiiigAooooAKOlFFAEivng0+oKcrkdeRVKXczlDsS0UgIIyKWrMxQ2PpTwQelR0VcZtEuKZLRUYY07f7Vp7RMjlY6ik3CjcPWndBZi0UZzRTEFGaKKd2gFzRmkoqlUkieVDqKbSg1pGaZLi0LSEZFLRXTSqcr1IaMy7sZJZvMV+2AD/DVdC9u+xkKgDJZD0BOOBkD0/GtojNQSwJIMMufavT/d1k1LRvrr/VjeniWkoy2MENOQdw+YDbll5A/yfrSGKd7ExwxFnc43bDwvQgH/AD1/LXa2RVwoAHtWXcxiNiQcE8HBratl9OpR5aej/S1vl+XkdlPEc7sZd3YvbbD1UoGOOcdM8jjrkfhVSt+ArBbNPPHHJB02sAxz/Tmse5FtuP2cvwcYI4I9c9fwx/hXw2NwkKLvGXy6/Ly9bHdCbbs/vIKKKK841CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooqe0tzcThcMUHzSFeoXuaunCVSShFXbE2krsijCFx5hIXqcdfpWx/oUihLW4Ubc4SUbc89ie3PANYtAODkda6MLi3h5XST1+f3kyjfW509p+6ZVbGSMjB7f0/GtiPkcVz8MqtaieVRIZOhHG3qNp6ntweta9jJuQgEkA8HGM/TnpX3NLFSxdG9tuvf8Ar17nj4qlb3i7SGlpD1rxsQ2mzmgIeBUVPc8UyuCo9bHTBaBTJD0FPqJzlqxlsaQWo2iiioNgooooAKrscsTUznCmoKTNILqFFFFIsKKKKACiiigApQSDkGkooAnVw31p1VqkSTs3507mco9iWiiimQFFFFABRRRQAUUUUAKCQcinq4PXio6KadhOKZPRUSsV+lSBgatO5k4tC0UUUyQooooAKXJHekooAcHNODA1HRVKbRLiiWiowxFPDA1qppkuLQtFFFUSKDS02lBrWE+jIkuotIRS0V3UZ6mTRBIOD9KymWN7mXzlZ1GMKCe5x/OtC6mEeVwSdpb8BWLcszzFWBU5wc+vT6V7E2vq8lzWf9fodmEg2ypqxlWVY2ZCq8AqOTgDqcc/0rNrc1gzGGOJIWNvgFGwePf6ketUG0yceaU+ZYwGBxgkHp+PP/66+HxuHq1KzcE2rL5abLul5f8AD+pSmuRXEsrL7asiqyoyfNubPT0qGa2mgdkdD8vUjkdu/wCI/Otby47a0EEbSbycsHUqefbPsP0pHEUto0bMUckliFyXyR9M+vNdn9kylQU2rNLddX2+92uL2rvdbf1qYlFat9ZWscSyxyMiHKqWXJfA+g6nj9fWssIzKzBSQvLEDp9a8WtQlSlyyNozUlcSiiisSgooooAKKKKACiiigAooooAKVVZ2CqCWJwAByTTzC6wiVlwhOAT3/wA/1q9pi2rRT+aqecigpvYgHk5/HpXRTw8pTUJaX11JlKyuQW9kZWkWVmhKYJ3Rkjn19KvRuLUGOIB49uDnOH68n061PGXmbeFxJn7wAxnOc/4+vHpzYFntAK5DKcg19fgMjUIt1NH08+39dTkniEnaRz9zbSQzuuzgc/KCQB1qOG3lnLCJC20ZOB0rfVfMLQzlDEcgksBhsfe9zg/16ikl8yGdobZvlQkgr976E9e1eVLI/wB+6cXe3T8d+nTu+5oq+livYrJNZFTAyrH93arEO2OvXrwfb29bdveyqgWNNzY/DgVHJLOJcrGqljwoUHJ9fr6VJKyPIBD8yqQdyjKOSM5OOh/P8K9zCQrYRfV5a3Xrptey+7btv0wmozV2jbifegJ/lilPWqtjdGdWRoyjp1Bq0etcWYQ5KljzVFxk0xjnnFNpSck0lePJ3Z0LRAeBmoOtSSHAx61HWcmbQWlwoooqSwooooAilPQfjUdKxyxNJUmyVkFFFFAwooooAKKKKACiiigAooooAcrlfcelTKwYcVXoBIORRclxuWaKjWQHg8GpKozasFFFFAgooooAKKKKACiiigB4cjrzTwQelQ0dKakS4Jk9FRiT1p4IPQ1aaZk4tC0UUUxBRRRQAUUUUAODetPqKlBIrSM7bkuPYkopAQaWtTMcKKaKdXVRnqjKSKN/EskR3MVx3BxWdbhZCEeKPy+gY4G5uwz/AJ/pVq9hnmlc7WWNRj72d3vgf5/lSLYyeXG+cumCqtwB07fhXtVKXtqfLZade+my/wCDsdlGap01eW/4EV7I0eHAWTf95iR8p5HGOnT1PSq8UtxuaQDeWAB3dCP8/wCTVh4XtcSNtC7txAAIXkdM1JpyMyZMe0dRjpj2qMPhqcZyhVWujT/rqauoo0uaOq2GfZXkbMshfbkLu9M1HdW2EyBz9eK2AmBTZIg6ketejCtCK5I6I41iHzXZiS2kU2nldw3Bjtk2YAxzyfyHXvTY410+2eJWSR3YMRtPA7EHAIP8u3eriW1xCriPjdzkcHOfUc4xSLaMxJk3AZ4Vmzj+n+RXkxypTq3n5631tr+d/uO11463d0Zk1nGZYYoUUMdu5XBGSOMZODnPp79MZpk+mIPL+zuzr/G56DJwOMD+Z/StVoHjQrHhVYFWx/F1x/P/ABqnIjxoU3NsJyVzxms/7AjJS5kvL5W+69tfXuaQr3as/wCv62KLaTdiNnEe/a2MLknpkH6Y/Gozp12CQYHUjPDcE/T1/CtKEMpEkjFY1UldxwGx2GeOv5VR1DUpbs7PMJjXODjbkccEZ6cV85isHQw9+Zvys9381t/wx0KU27KxTaNlUElcH0YH+VOSF5B8gBJ6LuGT9B1NR0V5acb7fj/wDXUmW0uHXcsLlcE5xxwM/wBKsHSbobjtUhW2k54+v06/lUljqr26pbyn/R8ndtX5uff2qeSFlIckMrYIZTkcjIr18HgaGKmowb+emvy6ef6mTnJXvoNh0mOPdLdSExKRtKDh/Uc4PoeO2aksrR7WSVlbJK/LzgdxyT9R09e3QyxwNKo3EkDgc9KupbM6Kj4IUbc4ySM5xnsOK+iWQ06TTjb73pp+Pp/wTkniLXuyhJOZoXW5UururAHOABnPf6U6KBLJ3UE+VIMrgDcpGMgn/OPzrSNoMZHDDkEdRVeOzllxHI7bUY9euMdj6Vricvp+0U6SXb779unl5kQxEGnfRD7CMMpYKVTjaCcnp/jmtDZxjiiCBYYwi9BU2K2qYnlsk9jz6k+eTaKZsYWLEr945Izxmnrbog+Ufj3qzRiso413sJzm92Y1/FtAJyqeoGTmpdMBkEjuBuLc49e/+RVu7RTAwbGMdziqumQSJiVs7XXI+bpXTOpde1v0aOhT5qDXU0QiqSwAyeppGOBTiajY5OK+axFZzd2RTiNoopjtgY9a4m7HQld2GMctmkoorI3WgUUUUAFMkbC49afUDtuakyoq7G0UUUjUKKKKACiiigAooooAKKKKACiiigAooooAKeshXg8imUUA1csBgwyKWq4JByDUqyA8Hg07mbjbYfRRRTICiiigAooooAKKKKACjpRRQA8SetPDA9KhopqTIcEyeiow5HXmnhgelWmmQ4tC0UUUyQooooAAcU8NmmUVUZNCauS0oNRhvWn1tGfVGTj3HdaMD0ptLmuqGIa6mbgNkiSRdrDIyDSqiqMAUuaWuhYqTVrkuL2DA9KTApaKqNcnlGFaaV9KlpCO9dlKu+4tirIgx0+orKvAAQuQMnGa2pBkGsYQGS7OSMR4Zg3IIzz9PpXfPEclByfXT0udmF1d30Kepu1vaRQq2VlXJzk4GQePxGelY9aep27Y89XZ0ODtXlUHTrnjnOOKzK+AzOTliHfayt6d/n9/fU9elblCiiivPNQrbsz51gokzIykBIwNpx0yvr0649c9BWJW1pflQWTTvGGYybd2TnHH+ev8q9HK5zhXvHt/l/SMa3wmhYkOox1rTVRisyxYtIAoYRgAKD29a1lHSvva1ZuCk1Y8XEaTsKFpQoFLRXk1K7WhgkFFBOKbmuGdXuWojqTIpKKy9qy+UHVHGHUMM5wRQMKoVQAAMADtQSB1phYnpSniJcvK2NQFZuwplFFcjdzZKwhOBk1ETk5pXbceOlNrKTubxjYKKKKRQUUUUANdtq+5qCnO25vYU2pZrFWQUUUUFBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAPWQrweRUoYMMiq9KCQcg0XJcbliimLIDweDT6ozasFFFFAgooooAKKKKACiiigAooooAeHI6808MDUNFNSZLgmT0VGHI6808MD0q00zNxaFooopkhSgkUlFAEgYGlqKnBiOtaxn3Icew+ikBBpa0IFzS02itI1GtyXHsOooorrpT1sZNEb8CsXUFBx1J+taGoTvCoARtp6sO1ZLK8jHc6tgjIf5fT/H1r241I06Lcu3a/odeFpu6mRag32a2SEwZRsCRgefUjOPUcdeh68isiWKSFtsilWxnBroJZzFLHiIbQA64bJ5x3/Dp7UyVIr+MG5by5DzvA64OB9Byc/QGvlsdgZ1kqvNpsu3muu7v5fej0ac3Hdepz9FaF1pqWsSym4yjZ24UZPTpz6n9D1qCygjuZvLdmBIOMD2JJP09O9eI8NUVRU2tXtquuxupprmGW1u91MI0xnqfp9Oprbs7WWzdhcTh0YY2A5JGFGPyPb0HpRElnZSmLy33KrAyDgkn8+MdKZ5LcFizRryAfT8+OnrX0OW5VNx9rHddddfJHPUqc2j0RPZzmIldhwPmOPQ//rrbiYMgYdCMisaIB78tLknrhlIIyeh49+v/ANattQAoA6V7dao3QTm9f+B17s8zFKPMrC0hNKabXiVJ/eZRQUUUwv6VztpbmqVxxIFNLE9KbRWTm2WopBRRRUFBUbvngUO+eBTKiUjSMerCiiipNAooooAKZI21eOpp54Gars25iaTKirsSiiikahRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAU9ZCvB5FMooBq5YDBhkUtVwSDkGpVkB4PBp3M3G2w+iiimQFFFFABRRRQAUUUUAFFFFABRRRQA9Xx15p4IIyKhoBI6VSkQ4J7E9FMV+x/On1SdzNprcKKKKYgp4f1plFNSa2E0mS0VGCRTwQa2jJMhxsLTqbRWsJ8pm1cVlDDBFU7y1EkR2rz7DmrtFerh8VKGlzNNxd0Y8Vi333UBiOg7VVu4BGcKOTzgV0BWqtxbrJzzn1FetRrRlDkil5XN4Yl895GPcWz3VnFDnY2eA3Qc7ecd857HjPpVG0sZBcszPt8obwwzhu/Xr3H/wBatCaB4NrrtDJ0Kjrz1pbiTfAqeZlgBuYNw/PT14xnn3rwMTlajWhOej9fOy+7fT0Z6UKl17uzGsFe6V0bdvOcYPTvzgVsRwrtGRmqNlbDhjn1wa1kFe9Lmw9Pkvc87E1E2kuhAbKJ2RiuNhyMVaopDXjYrEyn8XQwV5biGiimMe1eVKVtWbJX0Bmz9KbRRWDdzVKwUUUhIAyaQC1Ez54HShmLfSm1DkaxjbVhRRRUlhRRRQAUUU122r70AtRkjZ+UfjUdFFSbJWQUUUUDCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAHrIV4PIqUEEZFV6UEg5FFyXG5YopiyBuDwafVGbVgooooEFFFFABRRRQAUUUUAFFFFABTlYr9KbRQDVyZWDUtQVIr9jVqXcylDsPoooqiAo6UUUAPVs/WnVFUinI961hK+jM5RtqOBpabSiumnLoZSXUWo26VJUEsqpkEjPXFexhG5NWMra6GffSKoIJ5OeKoBMLskJUZ3BsZA7f5+lT3Ewe6G0huMbW6Hn/ACfwq9BaZi/e/Mx+9716VZOcuVuyj163/wCAehGaowTfUq2tyYm2SjHvnitaGRZFDKwIPcVny6WHclSoXH3cf4U2FZbO6CFQ5cHG3j36dqid6qcZWfn/AMAxqRp1FeD17GvTaELGNS67WIGVznB9KOlfNV371jKCEY4FR0pOTmkrhlK7OiKsFFFMZ8cDrUt2KSuOZgtREknmk60Vm3c1jGwUUUUigooooAKKKKAEJwMmoGbcc0533HjpTKlmsY2CiiigoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAp6yFeDyKZRQDVywCCMilquCQcg1KsgPB4NO5m42H0UUUyAooooAKKKKACiiigAooooAKKKKAHKxX6VICCMioaUEg5FNSsTKNyaimqwb606tDJqwUoODSUUCJQcigdaYp7U+uiEr6mMlbQdWVqTRh0ViNx7FcnH8/5Vqk4FZN7IklxGAcqh3OR2Fe/gablCS6CofxLkNla+bIyTBmUDIJXYD9AOnb8q2woApsYAUYpxNZYmuqcVShsiak3Vldi1FJCJGjbO0o2en6U/NISBXAsZOGsQUBxNRs2fpSFiaSuCpUcmbRjYKKCcDJqJn3cDpWTdjVRuKz9hTKKKzbuapWCiiigYUUUUAFFFFABUUj54HSh3zwOlR0maRj1YUUUUiwooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAHK5X3HpUysG6VXoouS4plmioll/vfnUgIIyDTM2mhaKKKYgooooAKKKKACiiigAooooAKer9j+dMooTsJpMnoqEMV6VIHB+taKVzJxaHU9Wz9aZRVxlYhq4s6PKm1XUA9QRyfx7U5IIo02KihfQCkDmjf7V6CzCagoLSxk6b2HIqxoFXOB6mgsBTCxNJXJVryqS5nuUoWHFiabRRWLbe5olYKQsFHNNZ+w/OoycnJqHIuML7isxY0lFFQahRRRQAUUUUAFFFBIAyaAConfPA6Ujvu4HSmUrmkY9WFFFFIsKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAClBK9DSUUASrKP4uKkqtSqxXoadyHDsWKKjWUHrxUmc9KCGmgooopiCiiigAooooAKKKKACiiigBwcjvThIO/FR0U02iXFMm3Ke9LUFFPmJ9mT9KQso71DRRzD9mSGT0FMLE9aSik22UopBRRRSGFFFFABRRRQAUUZx1qJpey/nSGk2PZwvXr6VEzFj7elNopGiikFFFFBQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFKCV6GiigCRZf7w/KnghuhoopkSirXFooopmYUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUxpAOnNFFJlxSZEzFjzSUUUjQKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA//9k=\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import libraries for simulation\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "# Imports for visualization\n",
    "import PIL.Image\n",
    "from io import BytesIO\n",
    "from IPython.display import Image, display\n",
    "\n",
    "def DisplayFractal(a, fmt='jpeg'):\n",
    "  \"\"\"Display an array of iteration counts as a\n",
    "     colorful picture of a fractal.\"\"\"\n",
    "  a_cyclic = (6.28*a/20.0).reshape(list(a.shape)+[1])\n",
    "  img = np.concatenate([10+20*np.cos(a_cyclic),\n",
    "                        30+50*np.sin(a_cyclic),\n",
    "                        155-80*np.cos(a_cyclic)], 2)\n",
    "  img[a==a.max()] = 0\n",
    "  a = img\n",
    "  a = np.uint8(np.clip(a, 0, 255))\n",
    "  f = BytesIO()\n",
    "  PIL.Image.fromarray(a).save(f, fmt)\n",
    "  display(Image(data=f.getvalue()))\n",
    "\n",
    "# Use NumPy to create a 2D array of complex numbers\n",
    "\n",
    "Y, X = np.mgrid[-1.3:1.3:0.005, -2:1:0.005]\n",
    "Z = X+1j*Y\n",
    "\n",
    "xs = tf.constant(Z.astype(np.complex64))\n",
    "zs = tf.Variable(xs)\n",
    "ns = tf.Variable(tf.zeros_like(xs, tf.float32))\n",
    "\n",
    "\n",
    "\n",
    "# Operation to update the zs and the iteration count.\n",
    "#\n",
    "# Note: We keep computing zs after they diverge! This\n",
    "#       is very wasteful! There are better, if a little\n",
    "#       less simple, ways to do this.\n",
    "#\n",
    "for i in range(200):\n",
    "    # Compute the new values of z: z^2 + x\n",
    "    zs_ = zs*zs + xs\n",
    "\n",
    "    # Have we diverged with this new value?\n",
    "    not_diverged = tf.abs(zs_) < 4\n",
    "\n",
    "    zs.assign(zs_),\n",
    "    ns.assign_add(tf.cast(not_diverged, tf.float32))\n",
    "    \n",
    "DisplayFractal(ns.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([[12.]], shape=(1, 1), dtype=float32)\n",
      "12.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "# Create a Constant op that produces a 1x2 matrix.  The op is\n",
    "# added as a node to the default graph.\n",
    "#\n",
    "# The value returned by the constructor represents the output\n",
    "# of the Constant op.\n",
    "matrix1 = tf.constant([[3., 3.]])\n",
    "\n",
    "# Create another Constant that produces a 2x1 matrix.\n",
    "matrix2 = tf.constant([[2.],[2.]])\n",
    "\n",
    "# Create a Matmul op that takes 'matrix1' and 'matrix2' as inputs.\n",
    "# The returned value, 'product', represents the result of the matrix\n",
    "# multiplication.\n",
    "product = tf.matmul(matrix1, matrix2)\n",
    "\n",
    "print(product)\n",
    "print(float(product))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([-2. -1.], shape=(2,), dtype=float32)\n",
      "[-2. -1.]\n"
     ]
    }
   ],
   "source": [
    "# Enter an interactive TensorFlow Session.\n",
    "import tensorflow as tf\n",
    "\n",
    "x = tf.Variable([1.0, 2.0])\n",
    "a = tf.constant([3.0, 3.0])\n",
    "\n",
    "# Add an op to subtract 'a' from 'x'.  Run it and print the result\n",
    "sub = tf.subtract(x, a)\n",
    "print(sub)\n",
    "print(sub.numpy())\n",
    "# ==> [-2. -1.]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2,) dtype=float32, numpy=array([4., 6.], dtype=float32)>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.assign([4.0, 6.0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([1. 3.], shape=(2,), dtype=float32)\n",
      "[1. 3.]\n"
     ]
    }
   ],
   "source": [
    "sub = tf.subtract(x, a)\n",
    "print(sub)\n",
    "print(sub.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction to Keras\n",
    "\n",
    "[Keras](https://keras.io/) is a layer on top of Tensorflow that makes it much easier to create neural networks.  Rather than define the graphs, like you see above, you define the individual layers of the network with a much more high level API.  Unless you are performing research into entirely new structures of deep neural networks it is unlikely that you need to program TensorFlow directly.  \n",
    "\n",
    "**For this class, we will use usually use TensorFlow through Keras, rather than direct TensorFlow**\n",
    "\n",
    "Keras is a separate install from TensorFlow.  To install Keras, use **pip install keras** after **pip install tensorflow**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple TensorFlow Regression: MPG\n",
    "\n",
    "This example shows how to encode the MPG dataset for regression.  This is slightly more complex than Iris, because:\n",
    "\n",
    "* Input has both numeric and categorical\n",
    "* Input has missing values\n",
    "\n",
    "This example uses functions defined above in this notepad, the \"helpful functions\". These functions allow you to build the feature vector for a neural network. Consider the following:\n",
    "\n",
    "* Predictors/Inputs \n",
    "    * Fill any missing inputs with the median for that column.  Use **missing_median**.\n",
    "    * Encode textual/categorical values with **encode_text_dummy**.\n",
    "    * Encode numeric values with **encode_numeric_zscore**.\n",
    "* Output\n",
    "    * Discard rows with missing outputs.\n",
    "    * Encode textual/categorical values with **encode_text_index**.\n",
    "    * Do not encode output numeric values.\n",
    "* Produce final feature vectors (x) and expected output (y) with **to_xy**.\n",
    "\n",
    "To encode categorical values that are part of the feature vector, use the functions from above.  If the categorical value is the target (as was the case with Iris, use the same technique as Iris). The iris technique allows you to decode back to Iris text strings from the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 398 samples\n",
      "Epoch 1/100\n",
      "398/398 - 1s - loss: 451.0939\n",
      "Epoch 2/100\n",
      "398/398 - 0s - loss: 169.9040\n",
      "Epoch 3/100\n",
      "398/398 - 0s - loss: 147.6547\n",
      "Epoch 4/100\n",
      "398/398 - 0s - loss: 130.4363\n",
      "Epoch 5/100\n",
      "398/398 - 0s - loss: 120.0428\n",
      "Epoch 6/100\n",
      "398/398 - 0s - loss: 112.3975\n",
      "Epoch 7/100\n",
      "398/398 - 0s - loss: 110.2450\n",
      "Epoch 8/100\n",
      "398/398 - 0s - loss: 110.8281\n",
      "Epoch 9/100\n",
      "398/398 - 0s - loss: 100.5697\n",
      "Epoch 10/100\n",
      "398/398 - 0s - loss: 93.0300\n",
      "Epoch 11/100\n",
      "398/398 - 0s - loss: 91.2283\n",
      "Epoch 12/100\n",
      "398/398 - 0s - loss: 82.8229\n",
      "Epoch 13/100\n",
      "398/398 - 0s - loss: 80.6588\n",
      "Epoch 14/100\n",
      "398/398 - 0s - loss: 75.4649\n",
      "Epoch 15/100\n",
      "398/398 - 0s - loss: 71.0691\n",
      "Epoch 16/100\n",
      "398/398 - 0s - loss: 68.4286\n",
      "Epoch 17/100\n",
      "398/398 - 0s - loss: 66.8159\n",
      "Epoch 18/100\n",
      "398/398 - 0s - loss: 62.2833\n",
      "Epoch 19/100\n",
      "398/398 - 0s - loss: 58.0148\n",
      "Epoch 20/100\n",
      "398/398 - 0s - loss: 55.8333\n",
      "Epoch 21/100\n",
      "398/398 - 0s - loss: 53.6490\n",
      "Epoch 22/100\n",
      "398/398 - 0s - loss: 51.5476\n",
      "Epoch 23/100\n",
      "398/398 - 0s - loss: 49.9140\n",
      "Epoch 24/100\n",
      "398/398 - 0s - loss: 48.5369\n",
      "Epoch 25/100\n",
      "398/398 - 0s - loss: 46.3741\n",
      "Epoch 26/100\n",
      "398/398 - 0s - loss: 45.7409\n",
      "Epoch 27/100\n",
      "398/398 - 0s - loss: 43.3046\n",
      "Epoch 28/100\n",
      "398/398 - 0s - loss: 43.6720\n",
      "Epoch 29/100\n",
      "398/398 - 0s - loss: 44.1135\n",
      "Epoch 30/100\n",
      "398/398 - 0s - loss: 46.1302\n",
      "Epoch 31/100\n",
      "398/398 - 0s - loss: 44.3032\n",
      "Epoch 32/100\n",
      "398/398 - 0s - loss: 43.1229\n",
      "Epoch 33/100\n",
      "398/398 - 0s - loss: 42.0082\n",
      "Epoch 34/100\n",
      "398/398 - 0s - loss: 38.3037\n",
      "Epoch 35/100\n",
      "398/398 - 0s - loss: 37.7679\n",
      "Epoch 36/100\n",
      "398/398 - 0s - loss: 38.7171\n",
      "Epoch 37/100\n",
      "398/398 - 0s - loss: 37.7919\n",
      "Epoch 38/100\n",
      "398/398 - 0s - loss: 37.9572\n",
      "Epoch 39/100\n",
      "398/398 - 0s - loss: 36.6300\n",
      "Epoch 40/100\n",
      "398/398 - 0s - loss: 37.8653\n",
      "Epoch 41/100\n",
      "398/398 - 0s - loss: 38.2734\n",
      "Epoch 42/100\n",
      "398/398 - 0s - loss: 36.4496\n",
      "Epoch 43/100\n",
      "398/398 - 0s - loss: 35.8224\n",
      "Epoch 44/100\n",
      "398/398 - 0s - loss: 35.5522\n",
      "Epoch 45/100\n",
      "398/398 - 0s - loss: 33.1960\n",
      "Epoch 46/100\n",
      "398/398 - 0s - loss: 33.5158\n",
      "Epoch 47/100\n",
      "398/398 - 0s - loss: 32.7692\n",
      "Epoch 48/100\n",
      "398/398 - 0s - loss: 33.3188\n",
      "Epoch 49/100\n",
      "398/398 - 0s - loss: 31.2705\n",
      "Epoch 50/100\n",
      "398/398 - 0s - loss: 33.7431\n",
      "Epoch 51/100\n",
      "398/398 - 0s - loss: 31.8444\n",
      "Epoch 52/100\n",
      "398/398 - 0s - loss: 32.5999\n",
      "Epoch 53/100\n",
      "398/398 - 0s - loss: 36.1910\n",
      "Epoch 54/100\n",
      "398/398 - 0s - loss: 32.6835\n",
      "Epoch 55/100\n",
      "398/398 - 0s - loss: 32.8383\n",
      "Epoch 56/100\n",
      "398/398 - 0s - loss: 29.6513\n",
      "Epoch 57/100\n",
      "398/398 - 0s - loss: 31.1441\n",
      "Epoch 58/100\n",
      "398/398 - 0s - loss: 31.9602\n",
      "Epoch 59/100\n",
      "398/398 - 0s - loss: 27.9736\n",
      "Epoch 60/100\n",
      "398/398 - 0s - loss: 28.8627\n",
      "Epoch 61/100\n",
      "398/398 - 0s - loss: 28.0135\n",
      "Epoch 62/100\n",
      "398/398 - 0s - loss: 27.7121\n",
      "Epoch 63/100\n",
      "398/398 - 0s - loss: 27.8517\n",
      "Epoch 64/100\n",
      "398/398 - 0s - loss: 28.6249\n",
      "Epoch 65/100\n",
      "398/398 - 0s - loss: 27.5334\n",
      "Epoch 66/100\n",
      "398/398 - 0s - loss: 27.4154\n",
      "Epoch 67/100\n",
      "398/398 - 0s - loss: 30.4497\n",
      "Epoch 68/100\n",
      "398/398 - 0s - loss: 28.9736\n",
      "Epoch 69/100\n",
      "398/398 - 0s - loss: 28.5237\n",
      "Epoch 70/100\n",
      "398/398 - 0s - loss: 25.5618\n",
      "Epoch 71/100\n",
      "398/398 - 0s - loss: 25.2815\n",
      "Epoch 72/100\n",
      "398/398 - 0s - loss: 25.2269\n",
      "Epoch 73/100\n",
      "398/398 - 0s - loss: 25.2716\n",
      "Epoch 74/100\n",
      "398/398 - 0s - loss: 28.5679\n",
      "Epoch 75/100\n",
      "398/398 - 0s - loss: 23.2467\n",
      "Epoch 76/100\n",
      "398/398 - 0s - loss: 23.3773\n",
      "Epoch 77/100\n",
      "398/398 - 0s - loss: 25.1101\n",
      "Epoch 78/100\n",
      "398/398 - 0s - loss: 24.7725\n",
      "Epoch 79/100\n",
      "398/398 - 0s - loss: 25.3017\n",
      "Epoch 80/100\n",
      "398/398 - 0s - loss: 26.4926\n",
      "Epoch 81/100\n",
      "398/398 - 0s - loss: 25.3778\n",
      "Epoch 82/100\n",
      "398/398 - 0s - loss: 23.9042\n",
      "Epoch 83/100\n",
      "398/398 - 0s - loss: 25.5613\n",
      "Epoch 84/100\n",
      "398/398 - 0s - loss: 21.7860\n",
      "Epoch 85/100\n",
      "398/398 - 0s - loss: 20.8380\n",
      "Epoch 86/100\n",
      "398/398 - 0s - loss: 20.9516\n",
      "Epoch 87/100\n",
      "398/398 - 0s - loss: 20.9954\n",
      "Epoch 88/100\n",
      "398/398 - 0s - loss: 20.1044\n",
      "Epoch 89/100\n",
      "398/398 - 0s - loss: 20.3450\n",
      "Epoch 90/100\n",
      "398/398 - 0s - loss: 20.8967\n",
      "Epoch 91/100\n",
      "398/398 - 0s - loss: 19.6073\n",
      "Epoch 92/100\n",
      "398/398 - 0s - loss: 19.3661\n",
      "Epoch 93/100\n",
      "398/398 - 0s - loss: 19.2692\n",
      "Epoch 94/100\n",
      "398/398 - 0s - loss: 19.0664\n",
      "Epoch 95/100\n",
      "398/398 - 0s - loss: 19.3474\n",
      "Epoch 96/100\n",
      "398/398 - 0s - loss: 17.9823\n",
      "Epoch 97/100\n",
      "398/398 - 0s - loss: 19.6501\n",
      "Epoch 98/100\n",
      "398/398 - 0s - loss: 20.6769\n",
      "Epoch 99/100\n",
      "398/398 - 0s - loss: 20.0518\n",
      "Epoch 100/100\n",
      "398/398 - 0s - loss: 17.7057\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1a4748cfd0>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "import pandas as pd\n",
    "import io\n",
    "import os\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/auto-mpg.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "cars = df['name']\n",
    "\n",
    "# Handle missing value\n",
    "df['horsepower'] = df['horsepower'].fillna(df['horsepower'].median())\n",
    "\n",
    "# Pandas to Numpy\n",
    "x = df[['cylinders', 'displacement', 'horsepower', 'weight',\n",
    "       'acceleration', 'year', 'origin']].values\n",
    "y = df['mpg'].values # regression\n",
    "\n",
    "# Build the neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(25, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(10, activation='relu')) # Hidden 2\n",
    "model.add(Dense(1)) # Output\n",
    "model.compile(loss='mean_squared_error', optimizer='adam')\n",
    "model.fit(x,y,verbose=2,epochs=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction to Neural Network Hyperparameters\n",
    "\n",
    "If you look at the above code you will see that the neural network is made up of 4 layers.  The first layer is the input layer.  This is specified by **input_dim** and it is set to be the number of inputs that the dataset has.  One input neuron is needed for ever input (including dummy variables).  However, there are also several hidden layers, with 25 and 10 neurons each. You might be wondering how these numbers were chosen?  This is one of the most common questions about neural networks.  Unfortunately, there is not a good answer.  These are hyperparameters.  They are settings that can affect neural network performance, yet there is not a clearly defined means of setting them.\n",
    "\n",
    "In general, more hidden neurons mean more capability to fit to complex problems.  However, too many neurons can lead to overfitting and lengthy training times.  Too few can lead to underfitting the problem and will sacrifice accuracy.  Also, how many layers you have is another hyperparameter.  In general, more layers allow the neural network to be able to perform more of its own feature engineering and data preprocessing.  But this also comes at the expense of training times and risk of overfitting.  In general, you will see that neuron counts start out larger near the input layer and tend to shrink towards the output layer in a sort of triangular fashion. \n",
    "\n",
    "There are techniques that use machine learning to optimize these values.  These will be discussed in [Module 8.3](t81_558_class_08_3_keras_hyperparameters.ipynb).\n",
    "\n",
    "### Controlling the Amount of Output\n",
    "\n",
    "One line is produced for each training epoch.  You can eliminate this output by setting the verbose setting of the fit command:\n",
    "\n",
    "* **verbose=0** - No progress output (use with Juputer if you do not want output)\n",
    "* **verbose=1** - Display progress bar, does not work well with Jupyter\n",
    "* **verbose=2** - Summary progress output (use with Jupyter if you want to know the loss at each epoch)\n",
    "\n",
    "### Regression Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will perform actual predictions.  These predictions are assigned to the **pred** variable. These are all MPG predictions from the neural network.  Notice that this is a 2D array?  You can always see the dimensions of what is returned by printing out **pred.shape**.  Neural networks can return multiple values, so the result is always an array.  Here the neural network only returns 1 value per prediction (there are 398 cars, so 398 predictions).  However, a 2D array is needed because the neural network has the potential of returning more than one value.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: (398, 1)\n",
      "[[14.390662 ]\n",
      " [14.292335 ]\n",
      " [15.508216 ]\n",
      " [16.497688 ]\n",
      " [15.40821  ]\n",
      " [10.367469 ]\n",
      " [10.617591 ]\n",
      " [11.007239 ]\n",
      " [10.962989 ]\n",
      " [13.115668 ]\n",
      " [12.98722  ]\n",
      " [14.003058 ]\n",
      " [ 9.598975 ]\n",
      " [15.1698675]\n",
      " [26.876045 ]\n",
      " [20.194618 ]\n",
      " [20.495552 ]\n",
      " [20.095375 ]\n",
      " [27.834633 ]\n",
      " [26.000862 ]\n",
      " [25.57331  ]\n",
      " [26.17234  ]\n",
      " [27.52909  ]\n",
      " [27.585793 ]\n",
      " [20.193062 ]\n",
      " [15.613624 ]\n",
      " [18.24308  ]\n",
      " [18.17234  ]\n",
      " [17.376686 ]\n",
      " [28.225716 ]\n",
      " [25.100746 ]\n",
      " [27.524055 ]\n",
      " [29.123909 ]\n",
      " [19.245857 ]\n",
      " [18.026863 ]\n",
      " [16.54264  ]\n",
      " [15.59806  ]\n",
      " [17.680061 ]\n",
      " [13.16001  ]\n",
      " [10.317359 ]\n",
      " [12.540809 ]\n",
      " [14.216895 ]\n",
      " [10.156287 ]\n",
      " [ 9.100135 ]\n",
      " [ 8.291297 ]\n",
      " [17.714119 ]\n",
      " [23.788612 ]\n",
      " [16.592476 ]\n",
      " [15.914496 ]\n",
      " [25.61103  ]\n",
      " [26.926033 ]\n",
      " [28.513176 ]\n",
      " [27.573952 ]\n",
      " [29.711205 ]\n",
      " [30.305428 ]\n",
      " [27.30288  ]\n",
      " [28.079308 ]\n",
      " [28.05149  ]\n",
      " [27.742378 ]\n",
      " [26.739845 ]\n",
      " [25.821877 ]\n",
      " [26.472328 ]\n",
      " [13.347022 ]\n",
      " [11.054542 ]\n",
      " [14.583289 ]\n",
      " [12.912635 ]\n",
      " [16.431435 ]\n",
      " [11.285865 ]\n",
      " [12.066199 ]\n",
      " [12.639686 ]\n",
      " [12.322913 ]\n",
      " [30.031136 ]\n",
      " [15.935676 ]\n",
      " [13.697974 ]\n",
      " [14.611304 ]\n",
      " [14.8634405]\n",
      " [26.477333 ]\n",
      " [25.399239 ]\n",
      " [25.221992 ]\n",
      " [27.21122  ]\n",
      " [25.843575 ]\n",
      " [28.952583 ]\n",
      " [26.918922 ]\n",
      " [27.203835 ]\n",
      " [29.102364 ]\n",
      " [15.338355 ]\n",
      " [16.822395 ]\n",
      " [13.116492 ]\n",
      " [15.243567 ]\n",
      " [15.903724 ]\n",
      " [ 9.914771 ]\n",
      " [ 9.052772 ]\n",
      " [12.997779 ]\n",
      " [14.849708 ]\n",
      " [11.3408575]\n",
      " [10.679542 ]\n",
      " [15.263495 ]\n",
      " [20.003883 ]\n",
      " [17.97422  ]\n",
      " [19.637794 ]\n",
      " [17.458748 ]\n",
      " [21.242928 ]\n",
      " [26.910866 ]\n",
      " [ 7.769325 ]\n",
      " [ 9.220313 ]\n",
      " [12.608497 ]\n",
      " [14.41587  ]\n",
      " [19.932472 ]\n",
      " [29.420082 ]\n",
      " [24.690804 ]\n",
      " [28.515236 ]\n",
      " [30.46894  ]\n",
      " [26.904915 ]\n",
      " [25.647896 ]\n",
      " [28.564247 ]\n",
      " [12.821326 ]\n",
      " [16.01139  ]\n",
      " [28.760612 ]\n",
      " [26.604958 ]\n",
      " [26.464119 ]\n",
      " [27.268318 ]\n",
      " [16.79795  ]\n",
      " [27.456825 ]\n",
      " [26.390175 ]\n",
      " [16.74601  ]\n",
      " [21.10978  ]\n",
      " [21.680061 ]\n",
      " [20.167274 ]\n",
      " [17.987526 ]\n",
      " [30.051933 ]\n",
      " [26.032997 ]\n",
      " [31.07795  ]\n",
      " [24.407356 ]\n",
      " [16.589485 ]\n",
      " [17.66285  ]\n",
      " [18.849281 ]\n",
      " [15.483314 ]\n",
      " [11.996192 ]\n",
      " [14.354163 ]\n",
      " [14.312903 ]\n",
      " [16.157997 ]\n",
      " [28.694984 ]\n",
      " [28.995705 ]\n",
      " [27.676369 ]\n",
      " [29.38993  ]\n",
      " [29.146523 ]\n",
      " [28.03025  ]\n",
      " [28.610054 ]\n",
      " [26.426247 ]\n",
      " [27.852516 ]\n",
      " [28.58744  ]\n",
      " [28.977242 ]\n",
      " [19.381264 ]\n",
      " [18.225685 ]\n",
      " [16.457283 ]\n",
      " [17.024788 ]\n",
      " [10.812293 ]\n",
      " [12.674354 ]\n",
      " [14.812049 ]\n",
      " [12.096229 ]\n",
      " [19.289711 ]\n",
      " [17.338509 ]\n",
      " [17.941689 ]\n",
      " [18.331184 ]\n",
      " [20.8433   ]\n",
      " [18.334877 ]\n",
      " [17.593422 ]\n",
      " [28.83979  ]\n",
      " [25.180061 ]\n",
      " [20.517586 ]\n",
      " [25.192207 ]\n",
      " [26.410805 ]\n",
      " [28.492348 ]\n",
      " [28.51487  ]\n",
      " [23.165047 ]\n",
      " [28.816154 ]\n",
      " [18.922127 ]\n",
      " [27.377357 ]\n",
      " [26.0615   ]\n",
      " [26.412086 ]\n",
      " [28.535774 ]\n",
      " [28.763237 ]\n",
      " [28.265022 ]\n",
      " [28.372261 ]\n",
      " [26.143166 ]\n",
      " [28.924904 ]\n",
      " [29.127876 ]\n",
      " [15.669471 ]\n",
      " [15.877601 ]\n",
      " [14.980567 ]\n",
      " [14.0821905]\n",
      " [20.181404 ]\n",
      " [18.656532 ]\n",
      " [21.076881 ]\n",
      " [19.826576 ]\n",
      " [28.908287 ]\n",
      " [28.441383 ]\n",
      " [29.246468 ]\n",
      " [29.13485  ]\n",
      " [19.318    ]\n",
      " [16.917122 ]\n",
      " [18.500496 ]\n",
      " [18.498482 ]\n",
      " [28.900734 ]\n",
      " [30.23861  ]\n",
      " [29.359413 ]\n",
      " [24.195076 ]\n",
      " [26.232704 ]\n",
      " [16.702065 ]\n",
      " [26.42759  ]\n",
      " [26.367378 ]\n",
      " [23.831978 ]\n",
      " [15.885169 ]\n",
      " [13.884131 ]\n",
      " [16.45475  ]\n",
      " [17.439735 ]\n",
      " [29.850334 ]\n",
      " [28.420479 ]\n",
      " [30.436989 ]\n",
      " [28.85804  ]\n",
      " [30.732063 ]\n",
      " [17.444374 ]\n",
      " [17.668922 ]\n",
      " [16.133705 ]\n",
      " [15.491187 ]\n",
      " [19.323494 ]\n",
      " [20.355537 ]\n",
      " [19.775124 ]\n",
      " [18.786873 ]\n",
      " [13.781714 ]\n",
      " [15.956885 ]\n",
      " [14.5248785]\n",
      " [14.171668 ]\n",
      " [29.98098  ]\n",
      " [25.262062 ]\n",
      " [29.756966 ]\n",
      " [25.878029 ]\n",
      " [28.447441 ]\n",
      " [29.872719 ]\n",
      " [29.59832  ]\n",
      " [29.117287 ]\n",
      " [26.540901 ]\n",
      " [28.97541  ]\n",
      " [31.316658 ]\n",
      " [29.418388 ]\n",
      " [29.376106 ]\n",
      " [30.354042 ]\n",
      " [31.082588 ]\n",
      " [30.306435 ]\n",
      " [19.559181 ]\n",
      " [17.270607 ]\n",
      " [18.521492 ]\n",
      " [20.85093  ]\n",
      " [22.72837  ]\n",
      " [21.998024 ]\n",
      " [26.215157 ]\n",
      " [20.696877 ]\n",
      " [20.138008 ]\n",
      " [20.672981 ]\n",
      " [21.844185 ]\n",
      " [21.25129  ]\n",
      " [20.215706 ]\n",
      " [19.401955 ]\n",
      " [25.146767 ]\n",
      " [19.355385 ]\n",
      " [16.2844   ]\n",
      " [28.842964 ]\n",
      " [28.081642 ]\n",
      " [30.0084   ]\n",
      " [28.424568 ]\n",
      " [28.34031  ]\n",
      " [26.952766 ]\n",
      " [25.347542 ]\n",
      " [29.717567 ]\n",
      " [28.136238 ]\n",
      " [26.853615 ]\n",
      " [29.749977 ]\n",
      " [27.121956 ]\n",
      " [30.141045 ]\n",
      " [29.587135 ]\n",
      " [22.267921 ]\n",
      " [22.780098 ]\n",
      " [26.443825 ]\n",
      " [20.551918 ]\n",
      " [22.048561 ]\n",
      " [17.632637 ]\n",
      " [17.685158 ]\n",
      " [14.949897 ]\n",
      " [17.3256   ]\n",
      " [15.524329 ]\n",
      " [15.197059 ]\n",
      " [19.99311  ]\n",
      " [15.483161 ]\n",
      " [30.560417 ]\n",
      " [30.62223  ]\n",
      " [30.607002 ]\n",
      " [27.063484 ]\n",
      " [22.005226 ]\n",
      " [14.884772 ]\n",
      " [25.784462 ]\n",
      " [19.372078 ]\n",
      " [28.226784 ]\n",
      " [28.716347 ]\n",
      " [31.31916  ]\n",
      " [29.76966  ]\n",
      " [26.435982 ]\n",
      " [26.758186 ]\n",
      " [26.741219 ]\n",
      " [26.24656  ]\n",
      " [30.32508  ]\n",
      " [31.142403 ]\n",
      " [29.711342 ]\n",
      " [31.109825 ]\n",
      " [26.899818 ]\n",
      " [27.054298 ]\n",
      " [26.582863 ]\n",
      " [21.068672 ]\n",
      " [30.629616 ]\n",
      " [28.21479  ]\n",
      " [28.63459  ]\n",
      " [29.999428 ]\n",
      " [29.727425 ]\n",
      " [31.117378 ]\n",
      " [27.112465 ]\n",
      " [31.42762  ]\n",
      " [29.925697 ]\n",
      " [29.531319 ]\n",
      " [26.795753 ]\n",
      " [24.777077 ]\n",
      " [31.023735 ]\n",
      " [34.034187 ]\n",
      " [30.58207  ]\n",
      " [30.685081 ]\n",
      " [28.579536 ]\n",
      " [32.939583 ]\n",
      " [28.96952  ]\n",
      " [26.67295  ]\n",
      " [29.799568 ]\n",
      " [28.107033 ]\n",
      " [26.891792 ]\n",
      " [26.953987 ]\n",
      " [26.957344 ]\n",
      " [27.88932  ]\n",
      " [32.219963 ]\n",
      " [31.20597  ]\n",
      " [32.107307 ]\n",
      " [31.18557  ]\n",
      " [32.28205  ]\n",
      " [31.154182 ]\n",
      " [31.506844 ]\n",
      " [28.693977 ]\n",
      " [30.04612  ]\n",
      " [29.873726 ]\n",
      " [29.910194 ]\n",
      " [31.767738 ]\n",
      " [30.190987 ]\n",
      " [30.164406 ]\n",
      " [30.469093 ]\n",
      " [28.804207 ]\n",
      " [26.35276  ]\n",
      " [25.942757 ]\n",
      " [27.864174 ]\n",
      " [29.580574 ]\n",
      " [22.165413 ]\n",
      " [14.814124 ]\n",
      " [23.384712 ]\n",
      " [20.35633  ]\n",
      " [30.531624 ]\n",
      " [30.22599  ]\n",
      " [30.878395 ]\n",
      " [29.703468 ]\n",
      " [28.446877 ]\n",
      " [27.796455 ]\n",
      " [27.863045 ]\n",
      " [27.643166 ]\n",
      " [31.176216 ]\n",
      " [32.2026   ]\n",
      " [32.25811  ]\n",
      " [29.328804 ]\n",
      " [30.830086 ]\n",
      " [31.145699 ]\n",
      " [30.61744  ]\n",
      " [30.476173 ]\n",
      " [31.679512 ]\n",
      " [31.816414 ]\n",
      " [31.81997  ]\n",
      " [26.95234  ]\n",
      " [20.166267 ]\n",
      " [27.474617 ]\n",
      " [24.279305 ]\n",
      " [28.78315  ]\n",
      " [28.347115 ]\n",
      " [26.984077 ]\n",
      " [27.427467 ]\n",
      " [31.084755 ]\n",
      " [28.308998 ]\n",
      " [29.052742 ]\n",
      " [29.22419  ]]\n"
     ]
    }
   ],
   "source": [
    "pred = model.predict(x)\n",
    "print(\"Shape: {}\".format(pred.shape))\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would like to see how good these predictions are.  We know what the correct MPG is for each car, so we can measure how close the neural network was."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final score (RMSE): 4.148288375456488\n"
     ]
    }
   ],
   "source": [
    "# Measure RMSE error.  RMSE is common for regression.\n",
    "score = np.sqrt(metrics.mean_squared_error(pred,y))\n",
    "print(f\"Final score (RMSE): {score}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that, on average the predictions were within +/- 5.89 values of the correct value.  This is not very good, but we will soon see how to improve it.\n",
    "\n",
    "We can also print out the first 10 cars, with predictions and actual MPG."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Car name: chevrolet chevelle malibu, MPG: 18.0, predicted MPG: [14.390662]\n",
      "2. Car name: buick skylark 320, MPG: 15.0, predicted MPG: [14.292335]\n",
      "3. Car name: plymouth satellite, MPG: 18.0, predicted MPG: [15.508216]\n",
      "4. Car name: amc rebel sst, MPG: 16.0, predicted MPG: [16.497688]\n",
      "5. Car name: ford torino, MPG: 17.0, predicted MPG: [15.40821]\n",
      "6. Car name: ford galaxie 500, MPG: 15.0, predicted MPG: [10.367469]\n",
      "7. Car name: chevrolet impala, MPG: 14.0, predicted MPG: [10.617591]\n",
      "8. Car name: plymouth fury iii, MPG: 14.0, predicted MPG: [11.007239]\n",
      "9. Car name: pontiac catalina, MPG: 14.0, predicted MPG: [10.962989]\n",
      "10. Car name: amc ambassador dpl, MPG: 15.0, predicted MPG: [13.115668]\n"
     ]
    }
   ],
   "source": [
    "# Sample predictions\n",
    "for i in range(10):\n",
    "    print(f\"{i+1}. Car name: {cars[i]}, MPG: {y[i]}, predicted MPG: {pred[i]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple TensorFlow Classification: Iris\n",
    "\n",
    "This is a very simple example of how to perform the Iris classification using TensorFlow.  The iris.csv file is used, rather than using the built-in files that many of the Google examples require.  \n",
    "\n",
    "**Make sure that you always run previous code blocks.  If you run the code block below, without the codeblock above, you will get errors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 150 samples\n",
      "Epoch 1/100\n",
      "150/150 - 1s - loss: 0.9697\n",
      "Epoch 2/100\n",
      "150/150 - 0s - loss: 0.8672\n",
      "Epoch 3/100\n",
      "150/150 - 0s - loss: 0.7989\n",
      "Epoch 4/100\n",
      "150/150 - 0s - loss: 0.7582\n",
      "Epoch 5/100\n",
      "150/150 - 0s - loss: 0.7161\n",
      "Epoch 6/100\n",
      "150/150 - 0s - loss: 0.6793\n",
      "Epoch 7/100\n",
      "150/150 - 0s - loss: 0.6448\n",
      "Epoch 8/100\n",
      "150/150 - 0s - loss: 0.6115\n",
      "Epoch 9/100\n",
      "150/150 - 0s - loss: 0.5819\n",
      "Epoch 10/100\n",
      "150/150 - 0s - loss: 0.5554\n",
      "Epoch 11/100\n",
      "150/150 - 0s - loss: 0.5295\n",
      "Epoch 12/100\n",
      "150/150 - 0s - loss: 0.5044\n",
      "Epoch 13/100\n",
      "150/150 - 0s - loss: 0.4821\n",
      "Epoch 14/100\n",
      "150/150 - 0s - loss: 0.4636\n",
      "Epoch 15/100\n",
      "150/150 - 0s - loss: 0.4450\n",
      "Epoch 16/100\n",
      "150/150 - 0s - loss: 0.4282\n",
      "Epoch 17/100\n",
      "150/150 - 0s - loss: 0.4123\n",
      "Epoch 18/100\n",
      "150/150 - 0s - loss: 0.3994\n",
      "Epoch 19/100\n",
      "150/150 - 0s - loss: 0.3854\n",
      "Epoch 20/100\n",
      "150/150 - 0s - loss: 0.3724\n",
      "Epoch 21/100\n",
      "150/150 - 0s - loss: 0.3624\n",
      "Epoch 22/100\n",
      "150/150 - 0s - loss: 0.3483\n",
      "Epoch 23/100\n",
      "150/150 - 0s - loss: 0.3387\n",
      "Epoch 24/100\n",
      "150/150 - 0s - loss: 0.3254\n",
      "Epoch 25/100\n",
      "150/150 - 0s - loss: 0.3168\n",
      "Epoch 26/100\n",
      "150/150 - 0s - loss: 0.3055\n",
      "Epoch 27/100\n",
      "150/150 - 0s - loss: 0.2964\n",
      "Epoch 28/100\n",
      "150/150 - 0s - loss: 0.2840\n",
      "Epoch 29/100\n",
      "150/150 - 0s - loss: 0.2766\n",
      "Epoch 30/100\n",
      "150/150 - 0s - loss: 0.2670\n",
      "Epoch 31/100\n",
      "150/150 - 0s - loss: 0.2581\n",
      "Epoch 32/100\n",
      "150/150 - 0s - loss: 0.2506\n",
      "Epoch 33/100\n",
      "150/150 - 0s - loss: 0.2407\n",
      "Epoch 34/100\n",
      "150/150 - 0s - loss: 0.2331\n",
      "Epoch 35/100\n",
      "150/150 - 0s - loss: 0.2251\n",
      "Epoch 36/100\n",
      "150/150 - 0s - loss: 0.2195\n",
      "Epoch 37/100\n",
      "150/150 - 0s - loss: 0.2112\n",
      "Epoch 38/100\n",
      "150/150 - 0s - loss: 0.2082\n",
      "Epoch 39/100\n",
      "150/150 - 0s - loss: 0.1982\n",
      "Epoch 40/100\n",
      "150/150 - 0s - loss: 0.1941\n",
      "Epoch 41/100\n",
      "150/150 - 0s - loss: 0.1862\n",
      "Epoch 42/100\n",
      "150/150 - 0s - loss: 0.1804\n",
      "Epoch 43/100\n",
      "150/150 - 0s - loss: 0.1745\n",
      "Epoch 44/100\n",
      "150/150 - 0s - loss: 0.1704\n",
      "Epoch 45/100\n",
      "150/150 - 0s - loss: 0.1648\n",
      "Epoch 46/100\n",
      "150/150 - 0s - loss: 0.1605\n",
      "Epoch 47/100\n",
      "150/150 - 0s - loss: 0.1556\n",
      "Epoch 48/100\n",
      "150/150 - 0s - loss: 0.1514\n",
      "Epoch 49/100\n",
      "150/150 - 0s - loss: 0.1511\n",
      "Epoch 50/100\n",
      "150/150 - 0s - loss: 0.1429\n",
      "Epoch 51/100\n",
      "150/150 - 0s - loss: 0.1416\n",
      "Epoch 52/100\n",
      "150/150 - 0s - loss: 0.1408\n",
      "Epoch 53/100\n",
      "150/150 - 0s - loss: 0.1339\n",
      "Epoch 54/100\n",
      "150/150 - 0s - loss: 0.1309\n",
      "Epoch 55/100\n",
      "150/150 - 0s - loss: 0.1282\n",
      "Epoch 56/100\n",
      "150/150 - 0s - loss: 0.1250\n",
      "Epoch 57/100\n",
      "150/150 - 0s - loss: 0.1224\n",
      "Epoch 58/100\n",
      "150/150 - 0s - loss: 0.1201\n",
      "Epoch 59/100\n",
      "150/150 - 0s - loss: 0.1183\n",
      "Epoch 60/100\n",
      "150/150 - 0s - loss: 0.1162\n",
      "Epoch 61/100\n",
      "150/150 - 0s - loss: 0.1158\n",
      "Epoch 62/100\n",
      "150/150 - 0s - loss: 0.1127\n",
      "Epoch 63/100\n",
      "150/150 - 0s - loss: 0.1100\n",
      "Epoch 64/100\n",
      "150/150 - 0s - loss: 0.1078\n",
      "Epoch 65/100\n",
      "150/150 - 0s - loss: 0.1070\n",
      "Epoch 66/100\n",
      "150/150 - 0s - loss: 0.1040\n",
      "Epoch 67/100\n",
      "150/150 - 0s - loss: 0.1044\n",
      "Epoch 68/100\n",
      "150/150 - 0s - loss: 0.1021\n",
      "Epoch 69/100\n",
      "150/150 - 0s - loss: 0.1002\n",
      "Epoch 70/100\n",
      "150/150 - 0s - loss: 0.1025\n",
      "Epoch 71/100\n",
      "150/150 - 0s - loss: 0.0964\n",
      "Epoch 72/100\n",
      "150/150 - 0s - loss: 0.0979\n",
      "Epoch 73/100\n",
      "150/150 - 0s - loss: 0.0946\n",
      "Epoch 74/100\n",
      "150/150 - 0s - loss: 0.0990\n",
      "Epoch 75/100\n",
      "150/150 - 0s - loss: 0.0933\n",
      "Epoch 76/100\n",
      "150/150 - 0s - loss: 0.0919\n",
      "Epoch 77/100\n",
      "150/150 - 0s - loss: 0.0918\n",
      "Epoch 78/100\n",
      "150/150 - 0s - loss: 0.0901\n",
      "Epoch 79/100\n",
      "150/150 - 0s - loss: 0.0909\n",
      "Epoch 80/100\n",
      "150/150 - 0s - loss: 0.0868\n",
      "Epoch 81/100\n",
      "150/150 - 0s - loss: 0.0897\n",
      "Epoch 82/100\n",
      "150/150 - 0s - loss: 0.0899\n",
      "Epoch 83/100\n",
      "150/150 - 0s - loss: 0.0865\n",
      "Epoch 84/100\n",
      "150/150 - 0s - loss: 0.0855\n",
      "Epoch 85/100\n",
      "150/150 - 0s - loss: 0.0856\n",
      "Epoch 86/100\n",
      "150/150 - 0s - loss: 0.0841\n",
      "Epoch 87/100\n",
      "150/150 - 0s - loss: 0.0844\n",
      "Epoch 88/100\n",
      "150/150 - 0s - loss: 0.0841\n",
      "Epoch 89/100\n",
      "150/150 - 0s - loss: 0.0832\n",
      "Epoch 90/100\n",
      "150/150 - 0s - loss: 0.0812\n",
      "Epoch 91/100\n",
      "150/150 - 0s - loss: 0.0805\n",
      "Epoch 92/100\n",
      "150/150 - 0s - loss: 0.0811\n",
      "Epoch 93/100\n",
      "150/150 - 0s - loss: 0.0798\n",
      "Epoch 94/100\n",
      "150/150 - 0s - loss: 0.0780\n",
      "Epoch 95/100\n",
      "150/150 - 0s - loss: 0.0788\n",
      "Epoch 96/100\n",
      "150/150 - 0s - loss: 0.0789\n",
      "Epoch 97/100\n",
      "150/150 - 0s - loss: 0.0780\n",
      "Epoch 98/100\n",
      "150/150 - 0s - loss: 0.0769\n",
      "Epoch 99/100\n",
      "150/150 - 0s - loss: 0.0777\n",
      "Epoch 100/100\n",
      "150/150 - 0s - loss: 0.0753\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1a49208e90>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import io\n",
    "import requests\n",
    "import numpy as np\n",
    "from sklearn import metrics\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation\n",
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://data.heatonresearch.com/data/t81-558/iris.csv\", \n",
    "    na_values=['NA', '?'])\n",
    "\n",
    "# Convert to numpy - Classification\n",
    "x = df[['sepal_l', 'sepal_w', 'petal_l', 'petal_w']].values\n",
    "dummies = pd.get_dummies(df['species']) # Classification\n",
    "species = dummies.columns\n",
    "y = dummies.values\n",
    "\n",
    "\n",
    "# Build neural network\n",
    "model = Sequential()\n",
    "model.add(Dense(50, input_dim=x.shape[1], activation='relu')) # Hidden 1\n",
    "model.add(Dense(25, activation='relu')) # Hidden 2\n",
    "model.add(Dense(y.shape[1],activation='softmax')) # Output\n",
    "\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "model.fit(x,y,verbose=2,epochs=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Iris-setosa', 'Iris-versicolor', 'Iris-virginica'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "# Print out number of species found:\n",
    "\n",
    "print(species)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have a neural network trained, we would like to be able to use it. The following code makes use of our neural network. Exactly like before, we will generate predictions.  Notice that 3 values come back for each of the 150 iris flowers.  There were 3 types of iris (Iris-setosa, Iris-versicolor, and Iris-virginica).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape: {pred.shape}\n",
      "[[9.97803628e-01 2.19636527e-03 3.15674958e-10]\n",
      " [9.95171249e-01 4.82870825e-03 2.72401635e-09]\n",
      " [9.96381223e-01 3.61878541e-03 1.68265046e-09]\n",
      " [9.93011296e-01 6.98864833e-03 6.66026612e-09]\n",
      " [9.97854292e-01 2.14564498e-03 3.05058062e-10]\n",
      " [9.96514142e-01 3.48593434e-03 3.64354269e-10]\n",
      " [9.94918764e-01 5.08119352e-03 2.94717273e-09]\n",
      " [9.96609688e-01 3.39028845e-03 8.45293335e-10]\n",
      " [9.91480827e-01 8.51911586e-03 1.48466883e-08]\n",
      " [9.95911241e-01 4.08877851e-03 1.70896475e-09]\n",
      " [9.98355806e-01 1.64413138e-03 1.07293466e-10]\n",
      " [9.94890571e-01 5.10939537e-03 2.18607576e-09]\n",
      " [9.95968103e-01 4.03186679e-03 2.09606044e-09]\n",
      " [9.96716440e-01 3.28350184e-03 2.63777267e-09]\n",
      " [9.99593318e-01 4.06734936e-04 4.22363360e-12]\n",
      " [9.99084949e-01 9.15081764e-04 1.44664307e-11]\n",
      " [9.98612523e-01 1.38745771e-03 7.40323497e-11]\n",
      " [9.97162879e-01 2.83708563e-03 5.30569477e-10]\n",
      " [9.97710228e-01 2.28978368e-03 1.27924019e-10]\n",
      " [9.97498810e-01 2.50116806e-03 3.05275305e-10]\n",
      " [9.96230543e-01 3.76941520e-03 6.26486918e-10]\n",
      " [9.96339083e-01 3.66093870e-03 7.10231984e-10]\n",
      " [9.98727620e-01 1.27238454e-03 1.96546210e-10]\n",
      " [9.86842096e-01 1.31579172e-02 1.05611289e-08]\n",
      " [9.89762723e-01 1.02373241e-02 7.26916927e-09]\n",
      " [9.92892146e-01 7.10790418e-03 4.68419747e-09]\n",
      " [9.92512107e-01 7.48796295e-03 3.79694276e-09]\n",
      " [9.97496784e-01 2.50323932e-03 3.52478574e-10]\n",
      " [9.97751772e-01 2.24828231e-03 3.26659672e-10]\n",
      " [9.92874980e-01 7.12508801e-03 5.46778134e-09]\n",
      " [9.92565393e-01 7.43460143e-03 5.80015858e-09]\n",
      " [9.95839119e-01 4.16086288e-03 8.48154214e-10]\n",
      " [9.99038815e-01 9.61192825e-04 3.17677447e-11]\n",
      " [9.99333084e-01 6.66855427e-04 1.13767962e-11]\n",
      " [9.94625866e-01 5.37417643e-03 2.93806690e-09]\n",
      " [9.97806847e-01 2.19317852e-03 4.93771801e-10]\n",
      " [9.98782337e-01 1.21771288e-03 7.04715383e-11]\n",
      " [9.98176575e-01 1.82348175e-03 2.40179376e-10]\n",
      " [9.93959188e-01 6.04081573e-03 7.30572802e-09]\n",
      " [9.96911705e-01 3.08828196e-03 6.38769371e-10]\n",
      " [9.97510552e-01 2.48938031e-03 4.75189388e-10]\n",
      " [9.81543362e-01 1.84565857e-02 1.00936496e-07]\n",
      " [9.95213628e-01 4.78634983e-03 3.89821864e-09]\n",
      " [9.88360763e-01 1.16393128e-02 8.31775093e-09]\n",
      " [9.91700828e-01 8.29921570e-03 2.60114619e-09]\n",
      " [9.92981195e-01 7.01878686e-03 6.25745766e-09]\n",
      " [9.97577250e-01 2.42277421e-03 2.68389089e-10]\n",
      " [9.95031714e-01 4.96828277e-03 3.28865868e-09]\n",
      " [9.98194873e-01 1.80516730e-03 1.42002937e-10]\n",
      " [9.96954799e-01 3.04525578e-03 7.83447029e-10]\n",
      " [1.82549166e-03 9.95544255e-01 2.63030897e-03]\n",
      " [1.84984249e-03 9.86768067e-01 1.13820983e-02]\n",
      " [9.23068495e-04 9.76991236e-01 2.20857300e-02]\n",
      " [2.42693839e-03 8.87552381e-01 1.10020638e-01]\n",
      " [1.12187490e-03 9.45179105e-01 5.36990762e-02]\n",
      " [1.56196090e-03 8.96508574e-01 1.01929463e-01]\n",
      " [1.11923600e-03 9.47363496e-01 5.15173227e-02]\n",
      " [2.09620576e-02 9.74658966e-01 4.37898468e-03]\n",
      " [1.81024289e-03 9.90554512e-01 7.63521530e-03]\n",
      " [3.22331488e-03 9.13634598e-01 8.31421316e-02]\n",
      " [7.76900817e-03 9.69992220e-01 2.22388376e-02]\n",
      " [2.42087129e-03 9.70757306e-01 2.68217977e-02]\n",
      " [5.24342526e-03 9.88323033e-01 6.43354375e-03]\n",
      " [1.07058382e-03 8.97402585e-01 1.01526864e-01]\n",
      " [1.24871526e-02 9.84983265e-01 2.52962671e-03]\n",
      " [2.93755718e-03 9.94492769e-01 2.56971107e-03]\n",
      " [1.12529483e-03 7.78505087e-01 2.20369667e-01]\n",
      " [6.87700463e-03 9.89707708e-01 3.41534102e-03]\n",
      " [4.86535806e-04 5.61541438e-01 4.37972069e-01]\n",
      " [5.52871497e-03 9.86493409e-01 7.97789544e-03]\n",
      " [2.50020268e-04 3.85135680e-01 6.14614308e-01]\n",
      " [5.00886049e-03 9.91089940e-01 3.90122505e-03]\n",
      " [2.49598495e-04 4.43942517e-01 5.55807948e-01]\n",
      " [1.49478263e-03 9.56907392e-01 4.15978432e-02]\n",
      " [3.11296852e-03 9.92891669e-01 3.99538176e-03]\n",
      " [2.40660179e-03 9.93225873e-01 4.36747866e-03]\n",
      " [1.01478794e-03 9.70552087e-01 2.84330714e-02]\n",
      " [3.76931042e-04 7.34224737e-01 2.65398324e-01]\n",
      " [1.24921056e-03 8.93484414e-01 1.05266415e-01]\n",
      " [3.54671627e-02 9.63734925e-01 7.97901943e-04]\n",
      " [5.95440622e-03 9.84434068e-01 9.61148646e-03]\n",
      " [1.10406773e-02 9.84760225e-01 4.19905083e-03]\n",
      " [6.18012343e-03 9.89255667e-01 4.56420705e-03]\n",
      " [4.52015520e-05 1.16904646e-01 8.83050263e-01]\n",
      " [9.29347007e-04 6.55200958e-01 3.43869656e-01]\n",
      " [1.64717284e-03 9.61875856e-01 3.64769772e-02]\n",
      " [1.27065333e-03 9.82478738e-01 1.62505321e-02]\n",
      " [1.46887649e-03 9.36127245e-01 6.24039248e-02]\n",
      " [3.89028923e-03 9.84593451e-01 1.15162851e-02]\n",
      " [2.98712379e-03 9.40230846e-01 5.67820184e-02]\n",
      " [1.80958409e-03 8.74775410e-01 1.23414956e-01]\n",
      " [1.44749333e-03 9.51385796e-01 4.71667424e-02]\n",
      " [4.28229570e-03 9.86986816e-01 8.73086415e-03]\n",
      " [1.98634453e-02 9.75505650e-01 4.63094003e-03]\n",
      " [2.50488147e-03 9.45574999e-01 5.19201420e-02]\n",
      " [3.95674910e-03 9.89075541e-01 6.96771359e-03]\n",
      " [3.06344125e-03 9.77781355e-01 1.91552117e-02]\n",
      " [2.85654888e-03 9.90567505e-01 6.57600677e-03]\n",
      " [5.39160706e-02 9.44798529e-01 1.28546311e-03]\n",
      " [3.39037925e-03 9.79389966e-01 1.72196385e-02]\n",
      " [5.29445146e-08 9.06231464e-04 9.99093771e-01]\n",
      " [2.32936782e-06 9.85938031e-03 9.90138292e-01]\n",
      " [6.63906349e-07 1.03143808e-02 9.89684939e-01]\n",
      " [2.58954492e-06 1.76425297e-02 9.82354879e-01]\n",
      " [1.66428975e-07 2.25956272e-03 9.97740269e-01]\n",
      " [7.07251075e-08 2.88804225e-03 9.97111797e-01]\n",
      " [1.26343821e-05 1.82228275e-02 9.81764615e-01]\n",
      " [7.84905751e-07 1.50228478e-02 9.84976351e-01]\n",
      " [7.27235772e-07 7.96301384e-03 9.92036283e-01]\n",
      " [1.30224876e-07 3.43394466e-03 9.96565998e-01]\n",
      " [6.39816499e-05 2.09598973e-01 7.90337086e-01]\n",
      " [4.34307549e-06 2.42407769e-02 9.75754917e-01]\n",
      " [3.10230530e-06 2.57371925e-02 9.74259734e-01]\n",
      " [1.25951874e-06 5.10992017e-03 9.94888842e-01]\n",
      " [3.89117531e-07 2.23651831e-03 9.97763157e-01]\n",
      " [1.41941848e-06 1.05200885e-02 9.89478528e-01]\n",
      " [1.33263502e-05 7.04364404e-02 9.29550231e-01]\n",
      " [4.00797489e-07 1.45100309e-02 9.85489666e-01]\n",
      " [1.51071209e-08 8.95211648e-04 9.99104798e-01]\n",
      " [2.39721812e-05 6.30142018e-02 9.36961710e-01]\n",
      " [5.66593940e-07 7.71682011e-03 9.92282569e-01]\n",
      " [2.93732637e-06 9.96197853e-03 9.90035117e-01]\n",
      " [7.80519400e-08 3.32737924e-03 9.96672511e-01]\n",
      " [7.39061070e-05 1.84248507e-01 8.15677583e-01]\n",
      " [2.33903643e-06 2.19693296e-02 9.78028357e-01]\n",
      " [1.19917095e-05 1.16814099e-01 8.83173883e-01]\n",
      " [1.37719486e-04 2.74012357e-01 7.25849807e-01]\n",
      " [1.28096566e-04 2.62956500e-01 7.36915410e-01]\n",
      " [3.31264431e-07 3.36591969e-03 9.96633708e-01]\n",
      " [6.50201473e-05 3.77714604e-01 6.22220337e-01]\n",
      " [1.06393748e-06 1.86348744e-02 9.81364071e-01]\n",
      " [1.84566888e-05 2.91336507e-01 7.08645046e-01]\n",
      " [2.36530880e-07 2.53736763e-03 9.97462392e-01]\n",
      " [2.41047877e-04 4.80097324e-01 5.19661665e-01]\n",
      " [1.05941481e-05 4.53686416e-02 9.54620779e-01]\n",
      " [2.50052437e-07 6.99782604e-03 9.93001878e-01]\n",
      " [2.63475187e-07 2.92268069e-03 9.97077107e-01]\n",
      " [1.48397130e-05 7.37483799e-02 9.26236808e-01]\n",
      " [1.67222635e-04 2.94138402e-01 7.05694318e-01]\n",
      " [1.16453866e-05 7.56622702e-02 9.24326062e-01]\n",
      " [2.02740367e-07 2.79367156e-03 9.97206032e-01]\n",
      " [1.47882984e-05 8.08725506e-02 9.19112682e-01]\n",
      " [2.32936782e-06 9.85938031e-03 9.90138292e-01]\n",
      " [1.46886265e-07 2.61785462e-03 9.97381985e-01]\n",
      " [1.33889657e-07 2.12873449e-03 9.97871161e-01]\n",
      " [2.89800687e-06 2.04284284e-02 9.79568720e-01]\n",
      " [8.71564771e-06 3.46175954e-02 9.65373695e-01]\n",
      " [1.69295527e-05 7.53162876e-02 9.24666762e-01]\n",
      " [1.00699617e-06 7.68297072e-03 9.92316008e-01]\n",
      " [2.43493032e-05 7.06994981e-02 9.29276109e-01]]\n"
     ]
    }
   ],
   "source": [
    "pred = model.predict(x)\n",
    "print(\"Shape: {pred.shape}\")\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you would like to turn of scientific notation, the following line can be used:\n",
    "np.set_printoptions(suppress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]\n",
      " [1 0 0]]\n"
     ]
    }
   ],
   "source": [
    "# The to_xy function represented the input in the same way.  Each row has only 1.0 value because each row is only one type\n",
    "# of iris.  This is the training data, we KNOW what type of iris it is.  This is called one-hot encoding.  Only one value\n",
    "# is 1.0 (hot)\n",
    "print(y[0:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictions: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 2 1\n",
      " 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2]\n",
      "Expected: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      " 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
      " 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2\n",
      " 2 2]\n"
     ]
    }
   ],
   "source": [
    "# Usually the column (pred) with the highest prediction is considered to be the prediction of the neural network.  It is easy\n",
    "# to convert the predictions to the expected iris species.  The argmax function finds the index of the maximum prediction\n",
    "# for each row.\n",
    "predict_classes = np.argmax(pred,axis=1)\n",
    "expected_classes = np.argmax(y,axis=1)\n",
    "print(f\"Predictions: {predict_classes}\")\n",
    "print(f\"Expected: {expected_classes}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',\n",
      "       'Iris-setosa', 'Iris-setosa', 'Iris-setosa', 'Iris-setosa',\n",
      "       'Iris-setosa'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "# Of course it is very easy to turn these indexes back into iris species.  We just use the species list that we created earlier.\n",
    "print(species[predict_classes[1:10]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.98\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "# Accuracy might be a more easily understood error metric.  It is essentially a test score.  For all of the iris predictions,\n",
    "# what percent were correct?  The downside is it does not consider how confident the neural network was in each prediction.\n",
    "correct = accuracy_score(expected_classes,predict_classes)\n",
    "print(f\"Accuracy: {correct}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below performs two ad hoc predictions.  The first prediction is simply a single iris flower.  The second predicts two iris flowers.  Notice that the argmax in the second prediction requires **axis=1**?  Since we have a 2D array now, we must specify which axis to take the argmax over.  The value **axis=1** specifies we want the max column index for each row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.00024024 0.17223348 0.8275264 ]]\n",
      "Predict that [[5. 3. 4. 2.]] is: Iris-virginica\n"
     ]
    }
   ],
   "source": [
    "# ad hoc prediction\n",
    "sample_flower = np.array( [[5.0,3.0,4.0,2.0]], dtype=float)\n",
    "pred = model.predict(sample_flower)\n",
    "print(pred)\n",
    "pred = np.argmax(pred)\n",
    "print(f\"Predict that {sample_flower} is: {species[pred]}\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.00024024 0.17223357 0.8275262 ]\n",
      " [0.9863484  0.01365156 0.00000001]]\n",
      "Predict that these two flowers [[5.  3.  4.  2. ]\n",
      " [5.2 3.5 1.5 0.8]] are: Index(['Iris-virginica', 'Iris-setosa'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "# predict two sample flowers\n",
    "sample_flower = np.array( [[5.0,3.0,4.0,2.0],[5.2,3.5,1.5,0.8]], dtype=float)\n",
    "pred = model.predict(sample_flower)\n",
    "print(pred)\n",
    "pred = np.argmax(pred,axis=1)\n",
    "print(f\"Predict that these two flowers {sample_flower} are: {species[pred]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (tensorflow)",
   "language": "python",
   "name": "rga"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}