notebooks/4_Convolutional_Neural_Network-based_Image_Classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# A convolutional neural network-based classifier for the MNIST handwritten digit dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's start by importing some packages we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (5.0, 4.0)\n",
    "plt.rcParams['image.cmap'] = 'Greys'\n",
    "\n",
    "import numpy as np\n",
    "np.set_printoptions(suppress=True)\n",
    "np.set_printoptions(precision=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Getting a feel for the data\n",
    "\n",
    "MNIST is a dataset that contains 70,000 labelled images of handwritten digits that look like the following.\n",
    "\n",
    "![MNIST Data Sample](images/mnist-sample.png \"MNIST Data Sample\")\n",
    "\n",
    "We're going to train a linear classifier on a part of this data set, and test it against another portion of the data set to see how well we did.\n",
    "\n",
    "The TensorFlow tutorial comes with a handy loader for this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The loader even handily splits the data set into three parts:\n",
    "\n",
    "* A training set (55000 examples) used to train the model\n",
    "* A validation set (5000 examples) used to optimise hyperparameters (not shown today)\n",
    "* A test set (10000 examples) used to gauge the accuracy of the trained model\n",
    "\n",
    "The images are greyscale and each 28 pixels wide by 28 pixels tall, and this is stored as an array of length 784.\n",
    "\n",
    "The labels are a *one hot* vector of length 10, meaning it is a vector of all zeros except at the location that corresponds to the label it's referring to. E.g. An image with a label `3` will be represented as `(0, 0, 0, 1, 0, 0, 0, 0, 0, 0)`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(55000, 784)\n",
      "(55000, 10)\n"
     ]
    }
   ],
   "source": [
    "print mnist.train.images.shape\n",
    "print mnist.train.labels.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 784)\n",
      "(10000, 10)\n"
     ]
    }
   ],
   "source": [
    "print mnist.test.images.shape\n",
    "print mnist.test.labels.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We can get a better sense for one of these examples by visualising the image and looking at the label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.  0.  0.  1.  0.  0.  0.  0.  0.  0.]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f8c3fe3e590>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADoJJREFUeJzt3W+IXfWdx/HPN7Z9MGkVNZNxsKMTS1gJoqlck4XKkLW2\n2LEY+0BtHoQsaiYPKjZYpKIPNogkItvGUaQwtUPHtSZdaMUY4m7d4B8KS/AqE43V3Yk6pQmTzARL\nasyDVPPtgzmWqc75nev9d27m+37BMPee7zlzv1z95Nx7fuecn7m7AMSzqOwGAJSD8ANBEX4gKMIP\nBEX4gaAIPxAU4QeCIvxAUIQfCOoL7XyxJUuWeH9/fztfEghlcnJSx44ds1rWbSj8ZnadpGFJZ0l6\n3N0fTK3f39+varXayEsCSKhUKjWvW/fHfjM7S9Jjkr4jaYWkdWa2ot6/B6C9GvnOv0rSQXd/191P\nSdopaW1z2gLQao2E/0JJf5rz/FC27B+Y2ZCZVc2sOjMz08DLAWimlh/td/cRd6+4e6W7u7vVLweg\nRo2E/7CkvjnPv5otA3AGaCT8r0habmbLzOxLkr4vaVdz2gLQanUP9bn7R2Z2h6T/1uxQ36i7v9m0\nzgC0VEPj/O6+R9KeJvUCoI04vRcIivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4g\nKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+\nICjCDwRF+IGgGpql18wmJX0g6WNJH7l7pRlNoX2OHz+erI+NjSXrmzdvTtbNLLfm7sltr7zyymT9\nscceS9ZXr16drEfXUPgz/+Lux5rwdwC0ER/7gaAaDb9L+p2ZvWpmQ81oCEB7NPqx/2p3P2xmSyU9\nb2Zvu/vLc1fI/lEYkqSLLrqowZcD0CwN7fnd/XD2e1rS05JWzbPOiLtX3L3S3d3dyMsBaKK6w29m\ni83sK588lvRtSQea1RiA1mrkY3+PpKezoZwvSHrK3f+rKV0BaLm6w+/u70q6oom9oE4nT57MrQ0P\nDye3ffTRR5P16enpZD01jl9LPWV8fDxZX79+fd3bd3V11dXTQsJQHxAU4QeCIvxAUIQfCIrwA0ER\nfiCoZlzVhxZ7/PHHk/WhofzLKoqG2oouqy3aftmyZcl6I6d0Hzp0KFmfmJhI1gcGBnJr1Wq1rp4W\nEvb8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4/xngKeeeipZT43FN3JJrVR8++yXXnopWW/k0tmi\ncfxLL700WS+6JDg69vxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBTj/B2g6PbYRdeep66pL7qevre3\nN1nfvn17sr5169Zk/e67786tnXPOOcltly9fnqyfPn06WV+0KH/ftmfPnuS2g4ODyfpCwJ4fCIrw\nA0ERfiAowg8ERfiBoAg/EBThB4IqHOc3s1FJ35U07e6XZcvOk/RrSf2SJiXd7O5/bl2bC9vSpUuT\n9XfeeSdZX7x4cW6t0amoi8bDt23blqxv2rQpt1Y0zr9v375kPTWOL6XvZbBmzZrkthHUsuf/paTr\nPrXsHkl73X25pL3ZcwBnkMLwu/vLkt7/1OK1ksayx2OSbmxyXwBarN7v/D3uPpU9PiKpp0n9AGiT\nhg/4+exkb7kTvpnZkJlVzaw6MzPT6MsBaJJ6w3/UzHolKfude2WKu4+4e8XdK93d3XW+HIBmqzf8\nuyRtyB5vkPRMc9oB0C6F4TezHZL+V9I/mdkhM7tN0oOSvmVmE5KuzZ4DOIMUjvO7+7qc0jeb3Aty\nlPl16fzzz0/Wr7jiimT97LPPzq3t3Lkzue1dd92VrM8ebsrX05N/HLrR8x8WAs7wA4Ii/EBQhB8I\nivADQRF+ICjCDwTFrbsXgNRU1kXTXBcN5aVuCy5J+/fvT9ZXrFiRWzty5Ehy26LpxS+44IJkveiS\n4OjY8wNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIzzLwBjY2O5taJbaxddFls01l60fWosv5FLciXp\n/vvvT9b7+vqS9ejY8wNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIzzL3BF4/Rlbn/DDTckt33kkUeS\ndcbxG8OeHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCKhznN7NRSd+VNO3ul2XLtkjaKGkmW+1ed9/T\nqiaRtmHDhtzae++9l9x2amoqWa9Wq8n6iRMnkvWUhx56KFlnHL+1atnz/1LSdfMs3+7uK7Mfgg+c\nYQrD7+4vS3q/Db0AaKNGvvPfYWavm9momZ3btI4AtEW94f+ZpK9JWilpStJP8lY0syEzq5pZdWZm\nJm81AG1WV/jd/ai7f+zupyX9XNKqxLoj7l5x90p3d3e9fQJosrrCb2a9c55+T9KB5rQDoF1qGerb\nIWmNpCVmdkjSv0laY2YrJbmkSUmbWtgjgBawonunN1OlUvGicWN0lqLjNPfdd1+yPjo6mlsbGBhI\nbrt79+5kvaurK1mPqFKpqFqt1nQTBs7wA4Ii/EBQhB8IivADQRF+ICjCDwTFrbtrdPLkydzaQh5y\nKjorc2RkJFn/8MMPc2s7duxIbvvss88m67fcckuyjjT2/EBQhB8IivADQRF+ICjCDwRF+IGgCD8Q\nFOP8mYmJiWR906b8WxZcfvnlyW0ffvjhunpaCLZs2ZJb27lzZ3LbAwfS94hhnL8x7PmBoAg/EBTh\nB4Ii/EBQhB8IivADQRF+IKgw4/yp6/Gl4jHjiy++OLcWeRz/1KlTyfq6detya+28bTw+iz0/EBTh\nB4Ii/EBQhB8IivADQRF+ICjCDwRVOM5vZn2SnpDUI8kljbj7sJmdJ+nXkvolTUq62d3/3LpWG/Pi\niy8m6/v370/Wr7/++iZ2c+aYnp5O1gcHB5P18fHx3JpZeibpovskoDG17Pk/kvQjd18h6Z8l/cDM\nVki6R9Jed18uaW/2HMAZojD87j7l7q9ljz+Q9JakCyWtlTSWrTYm6cZWNQmg+T7Xd34z65f0dUn7\nJPW4+1RWOqLZrwUAzhA1h9/MvizpN5I2u/tf5tZ89iTteU/UNrMhM6uaWXVmZqahZgE0T03hN7Mv\najb4v3L332aLj5pZb1bvlTTvkSF3H3H3irtXiiZ9BNA+heG32UOyv5D0lrv/dE5pl6QN2eMNkp5p\nfnsAWqWWS3q/IWm9pDfM7JNxm3slPSjpP83sNkl/lHRza1psjkqlkqyfPn06WX/uuedya9dee21y\n20suuSRZ7+vrS9aLHD9+PLeWGmqTpCeffDJZHx0dTdaLLstNDec98MADyW1vuummZB2NKQy/u/9e\nUt5/wW82tx0A7cIZfkBQhB8IivADQRF+ICjCDwRF+IGgwty6e+nSpcn6xo0bk/XUePc111yT3Lbo\n0tWBgYFkvcjbb7+dWyu6JLeRcfpaDA8P59ZuvfXWhv42GsOeHwiK8ANBEX4gKMIPBEX4gaAIPxAU\n4QeCCjPOX6Romu2DBw/m1l544YXktosWpf+NLbqteNFYe2qsvmjbrq6uZP2qq65K1rdt25asr169\nOllHedjzA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQjPNnisa7d+/enVsrGususnXr1mT99ttvT9aL\n7lWQcueddybrzLK0cLHnB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgrIb7tvdJekJSjySXNOLuw2a2\nRdJGSTPZqve6+57U36pUKl6tVhtuGsD8KpWKqtVqTZMt1HKSz0eSfuTur5nZVyS9ambPZ7Xt7v7v\n9TYKoDyF4Xf3KUlT2eMPzOwtSRe2ujEArfW5vvObWb+kr0valy26w8xeN7NRMzs3Z5shM6uaWXVm\nZma+VQCUoObwm9mXJf1G0mZ3/4ukn0n6mqSVmv1k8JP5tnP3EXevuHuF88SBzlFT+M3si5oN/q/c\n/beS5O5H3f1jdz8t6eeSVrWuTQDNVhh+m7396y8kveXuP52zvHfOat+TdKD57QFolVqO9n9D0npJ\nb5jZeLbsXknrzGylZof/JiVtakmHAFqilqP9v5c037hhckwfQGfjDD8gKMIPBEX4gaAIPxAU4QeC\nIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhbfubuqLmc1I+uOcRUskHWtbA59Pp/bW\nqX1J9FavZvZ2sbvXdL+8tob/My9uVnX3SmkNJHRqb53al0Rv9SqrNz72A0ERfiCossM/UvLrp3Rq\nb53al0Rv9Sqlt1K/8wMoT9l7fgAlKSX8Znadmf2fmR00s3vK6CGPmU2a2RtmNm5mpU4pnE2DNm1m\nB+YsO8/Mnjeziez3vNOkldTbFjM7nL1342Y2WFJvfWb2gpn9wczeNLMfZstLfe8SfZXyvrX9Y7+Z\nnSXp/yV9S9IhSa9IWufuf2hrIznMbFJSxd1LHxM2swFJJyQ94e6XZcsekvS+uz+Y/cN5rrv/uEN6\n2yLpRNkzN2cTyvTOnVla0o2S/lUlvneJvm5WCe9bGXv+VZIOuvu77n5K0k5Ja0voo+O5+8uS3v/U\n4rWSxrLHY5r9n6ftcnrrCO4+5e6vZY8/kPTJzNKlvneJvkpRRvgvlPSnOc8PqbOm/HZJvzOzV81s\nqOxm5tGTTZsuSUck9ZTZzDwKZ25up0/NLN0x7109M143Gwf8Putqd79S0nck/SD7eNuRfPY7WycN\n19Q0c3O7zDOz9N+V+d7VO+N1s5UR/sOS+uY8/2q2rCO4++Hs97Skp9V5sw8f/WSS1Oz3dMn9/F0n\nzdw838zS6oD3rpNmvC4j/K9IWm5my8zsS5K+L2lXCX18hpktzg7EyMwWS/q2Om/24V2SNmSPN0h6\npsRe/kGnzNycN7O0Sn7vOm7Ga3dv+4+kQc0e8X9H0n1l9JDT1yWS9mc/b5bdm6Qdmv0Y+FfNHhu5\nTdL5kvZKmpD0P5LO66De/kPSG5Je12zQekvq7WrNfqR/XdJ49jNY9nuX6KuU940z/ICgOOAHBEX4\ngaAIPxAU4QeCIvxAUIQfCIrwA0ERfiCovwGyCoCFRwOAggAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8caa29cf90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "example_image = mnist.train.images[1]\n",
    "example_image_reshaped = example_image.reshape((28, 28)) # Can't render a line of 784 numbers\n",
    "example_label = mnist.train.labels[1]\n",
    "\n",
    "print example_label\n",
    "plt.imshow(example_image_reshaped)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Setting up a score function, loss function and optimisation algorithm\n",
    "\n",
    "Now that we have a better sense of the dataset we're working with, let's move onto the machine learning bits.\n",
    "\n",
    "First, we setup some placeholders to hold batches of this training data for when we learn our model. The reason why we work in batches is that it's easier on memory than holding the entire set. And it's this notion of working with (random) batches of input rather than the entire set that moves us from the realm of *Gradient Descent* that we saw earlier, to *Stochastic Gradient Descent* that we have here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, [None, 784])\n",
    "y_ = tf.placeholder(tf.float32, [None, 10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In abstract notation, the models we've seen so far are:\n",
    "\n",
    "1. `Linear: Input -> FC -> Softmax -> Loss`\n",
    "2. `Neural Network: Input -> FC -> ReLU -> FC -> Softmax -> Loss`\n",
    "\n",
    "In this notation, the model we're going to work out in this example is:\n",
    "\n",
    "`Input -> CONV (-> ReLU -> Pool) -> FC -> ReLU -> FC -> Softmax -> Loss`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Define some helper functions to ease the definition of the model\n",
    "def weight_variable(shape):\n",
    "    return tf.Variable(tf.truncated_normal(shape, stddev=0.1))\n",
    "\n",
    "def bias_variable(shape):\n",
    "    return tf.Variable(tf.constant(0.1, shape=shape))\n",
    "\n",
    "def conv(x, W):\n",
    "    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')\n",
    "\n",
    "def pool(x):\n",
    "    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')\n",
    "\n",
    "\n",
    "# A score function (model) involving some layers\n",
    "\n",
    "# Reshape the input to look like a volume (Input)\n",
    "x_image = tf.reshape(x, [-1, 28, 28, 1])\n",
    "\n",
    "# A convolutional layer (CONV -> RELU -> POOL)\n",
    "W_conv = weight_variable([5, 5, 1, 32])\n",
    "b_conv = bias_variable([32])\n",
    "h_conv = tf.nn.relu(conv(x_image, W_conv) + b_conv)\n",
    "h_pool = pool(h_conv)\n",
    "\n",
    "# A densely connected layer (FC)\n",
    "W_fc1 = weight_variable([14*14*32, 1024])\n",
    "b_fc1 = bias_variable([1024])\n",
    "h_pool_flat = tf.reshape(h_pool, [-1, 14*14*32])\n",
    "h_fc1 = tf.nn.relu(tf.matmul(h_pool_flat, W_fc1) + b_fc1)\n",
    "\n",
    "# Another densely connected layer (for \"readout\") (FC)\n",
    "W_fc2 = weight_variable([1024, 10])\n",
    "b_fc2 = bias_variable([10])\n",
    "y = tf.matmul(h_fc1, W_fc2) + b_fc2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We define our loss function to measure how poorly this model performs on images with known labels. We use the a specific form called the [cross entropy loss](https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y, labels=y_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Using the magic of blackbox optimisation algorithms provided by TensorFlow, we can define a single step of the [Adam optimiser](https://arxiv.org/abs/1412.6980) (to improve our parameters for our score function and reduce the loss) in one line of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Training the model\n",
    "\n",
    "The way TensorFlow works, we haven't really executed any of the code above in the classic sense. All we've done is defined what's called the computational graph.\n",
    "\n",
    "Now we go ahead and initialise a session to actually train the model and evaluate its performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "sess = tf.Session()\n",
    "sess.run(init)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We train the model iteratively for 20000 steps, loading a batch of example images each time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0, training accuracy 0.12\n",
      "step 100, training accuracy 0.86\n",
      "step 200, training accuracy 0.84\n",
      "step 300, training accuracy 0.86\n",
      "step 400, training accuracy 0.98\n",
      "step 500, training accuracy 0.82\n",
      "step 600, training accuracy 0.98\n",
      "step 700, training accuracy 0.94\n",
      "step 800, training accuracy 0.86\n",
      "step 900, training accuracy 0.96\n",
      "step 1000, training accuracy 0.92\n",
      "step 1100, training accuracy 0.94\n",
      "step 1200, training accuracy 0.92\n",
      "step 1300, training accuracy 0.96\n",
      "step 1400, training accuracy 0.94\n",
      "step 1500, training accuracy 0.94\n",
      "step 1600, training accuracy 0.88\n",
      "step 1700, training accuracy 0.98\n",
      "step 1800, training accuracy 1\n",
      "step 1900, training accuracy 1\n",
      "step 2000, training accuracy 0.96\n",
      "step 2100, training accuracy 0.94\n",
      "step 2200, training accuracy 0.98\n",
      "step 2300, training accuracy 0.98\n",
      "step 2400, training accuracy 1\n",
      "step 2500, training accuracy 1\n",
      "step 2600, training accuracy 0.98\n",
      "step 2700, training accuracy 0.96\n",
      "step 2800, training accuracy 1\n",
      "step 2900, training accuracy 0.96\n",
      "step 3000, training accuracy 1\n",
      "step 3100, training accuracy 1\n",
      "step 3200, training accuracy 0.92\n",
      "step 3300, training accuracy 1\n",
      "step 3400, training accuracy 0.92\n",
      "step 3500, training accuracy 1\n",
      "step 3600, training accuracy 1\n",
      "step 3700, training accuracy 1\n",
      "step 3800, training accuracy 1\n",
      "step 3900, training accuracy 0.98\n",
      "step 4000, training accuracy 0.98\n",
      "step 4100, training accuracy 0.98\n",
      "step 4200, training accuracy 1\n",
      "step 4300, training accuracy 0.98\n",
      "step 4400, training accuracy 1\n",
      "step 4500, training accuracy 0.98\n",
      "step 4600, training accuracy 1\n",
      "step 4700, training accuracy 1\n",
      "step 4800, training accuracy 1\n",
      "step 4900, training accuracy 1\n",
      "step 5000, training accuracy 0.96\n",
      "step 5100, training accuracy 0.94\n",
      "step 5200, training accuracy 0.98\n",
      "step 5300, training accuracy 0.94\n",
      "step 5400, training accuracy 1\n",
      "step 5500, training accuracy 1\n",
      "step 5600, training accuracy 0.98\n",
      "step 5700, training accuracy 1\n",
      "step 5800, training accuracy 1\n",
      "step 5900, training accuracy 1\n",
      "step 6000, training accuracy 1\n",
      "step 6100, training accuracy 1\n",
      "step 6200, training accuracy 1\n",
      "step 6300, training accuracy 0.98\n",
      "step 6400, training accuracy 1\n",
      "step 6500, training accuracy 0.96\n",
      "step 6600, training accuracy 0.98\n",
      "step 6700, training accuracy 1\n",
      "step 6800, training accuracy 1\n",
      "step 6900, training accuracy 1\n",
      "step 7000, training accuracy 0.98\n",
      "step 7100, training accuracy 1\n",
      "step 7200, training accuracy 1\n",
      "step 7300, training accuracy 1\n",
      "step 7400, training accuracy 1\n",
      "step 7500, training accuracy 1\n",
      "step 7600, training accuracy 1\n",
      "step 7700, training accuracy 1\n",
      "step 7800, training accuracy 1\n",
      "step 7900, training accuracy 1\n",
      "step 8000, training accuracy 0.98\n",
      "step 8100, training accuracy 1\n",
      "step 8200, training accuracy 1\n",
      "step 8300, training accuracy 1\n",
      "step 8400, training accuracy 0.98\n",
      "step 8500, training accuracy 0.98\n",
      "step 8600, training accuracy 0.98\n",
      "step 8700, training accuracy 0.92\n",
      "step 8800, training accuracy 1\n",
      "step 8900, training accuracy 0.98\n",
      "step 9000, training accuracy 1\n",
      "step 9100, training accuracy 1\n",
      "step 9200, training accuracy 1\n",
      "step 9300, training accuracy 1\n",
      "step 9400, training accuracy 1\n",
      "step 9500, training accuracy 0.96\n",
      "step 9600, training accuracy 1\n",
      "step 9700, training accuracy 1\n",
      "step 9800, training accuracy 1\n",
      "step 9900, training accuracy 1\n",
      "step 10000, training accuracy 0.98\n",
      "step 10100, training accuracy 1\n",
      "step 10200, training accuracy 1\n",
      "step 10300, training accuracy 0.98\n",
      "step 10400, training accuracy 1\n",
      "step 10500, training accuracy 1\n",
      "step 10600, training accuracy 0.98\n",
      "step 10700, training accuracy 1\n",
      "step 10800, training accuracy 1\n",
      "step 10900, training accuracy 0.96\n",
      "step 11000, training accuracy 0.98\n",
      "step 11100, training accuracy 0.98\n",
      "step 11200, training accuracy 1\n",
      "step 11300, training accuracy 1\n",
      "step 11400, training accuracy 1\n",
      "step 11500, training accuracy 1\n",
      "step 11600, training accuracy 1\n",
      "step 11700, training accuracy 0.98\n",
      "step 11800, training accuracy 1\n",
      "step 11900, training accuracy 1\n",
      "step 12000, training accuracy 0.98\n",
      "step 12100, training accuracy 1\n",
      "step 12200, training accuracy 1\n",
      "step 12300, training accuracy 1\n",
      "step 12400, training accuracy 1\n",
      "step 12500, training accuracy 1\n",
      "step 12600, training accuracy 1\n",
      "step 12700, training accuracy 1\n",
      "step 12800, training accuracy 1\n",
      "step 12900, training accuracy 1\n",
      "step 13000, training accuracy 1\n",
      "step 13100, training accuracy 1\n",
      "step 13200, training accuracy 1\n",
      "step 13300, training accuracy 1\n",
      "step 13400, training accuracy 1\n",
      "step 13500, training accuracy 1\n",
      "step 13600, training accuracy 1\n",
      "step 13700, training accuracy 1\n",
      "step 13800, training accuracy 1\n",
      "step 13900, training accuracy 1\n",
      "step 14000, training accuracy 1\n",
      "step 14100, training accuracy 1\n",
      "step 14200, training accuracy 1\n",
      "step 14300, training accuracy 1\n",
      "step 14400, training accuracy 1\n",
      "step 14500, training accuracy 1\n",
      "step 14600, training accuracy 1\n",
      "step 14700, training accuracy 1\n",
      "step 14800, training accuracy 1\n",
      "step 14900, training accuracy 1\n",
      "step 15000, training accuracy 1\n",
      "step 15100, training accuracy 1\n",
      "step 15200, training accuracy 1\n",
      "step 15300, training accuracy 1\n",
      "step 15400, training accuracy 1\n",
      "step 15500, training accuracy 1\n",
      "step 15600, training accuracy 1\n",
      "step 15700, training accuracy 1\n",
      "step 15800, training accuracy 1\n",
      "step 15900, training accuracy 1\n",
      "step 16000, training accuracy 1\n",
      "step 16100, training accuracy 1\n",
      "step 16200, training accuracy 1\n",
      "step 16300, training accuracy 1\n",
      "step 16400, training accuracy 1\n",
      "step 16500, training accuracy 1\n",
      "step 16600, training accuracy 1\n",
      "step 16700, training accuracy 1\n",
      "step 16800, training accuracy 1\n",
      "step 16900, training accuracy 1\n",
      "step 17000, training accuracy 1\n",
      "step 17100, training accuracy 1\n",
      "step 17200, training accuracy 1\n",
      "step 17300, training accuracy 1\n",
      "step 17400, training accuracy 1\n",
      "step 17500, training accuracy 1\n",
      "step 17600, training accuracy 1\n",
      "step 17700, training accuracy 1\n",
      "step 17800, training accuracy 1\n",
      "step 17900, training accuracy 1\n",
      "step 18000, training accuracy 1\n",
      "step 18100, training accuracy 1\n",
      "step 18200, training accuracy 1\n",
      "step 18300, training accuracy 1\n",
      "step 18400, training accuracy 1\n",
      "step 18500, training accuracy 1\n",
      "step 18600, training accuracy 1\n",
      "step 18700, training accuracy 1\n",
      "step 18800, training accuracy 1\n",
      "step 18900, training accuracy 1\n",
      "step 19000, training accuracy 1\n",
      "step 19100, training accuracy 1\n",
      "step 19200, training accuracy 1\n",
      "step 19300, training accuracy 1\n",
      "step 19400, training accuracy 1\n",
      "step 19500, training accuracy 1\n",
      "step 19600, training accuracy 1\n",
      "step 19700, training accuracy 1\n",
      "step 19800, training accuracy 1\n",
      "step 19900, training accuracy 1\n"
     ]
    }
   ],
   "source": [
    "for i in range(20000):\n",
    "    batch = mnist.train.next_batch(50)\n",
    "    if i%100 == 0:\n",
    "        train_accuracy = accuracy.eval(session=sess, feed_dict={\n",
    "            x:batch[0], y_: batch[1]})\n",
    "        print(\"step %d, training accuracy %g\"%(i, train_accuracy))\n",
    "    train_step.run(session=sess, feed_dict={x: batch[0], y_: batch[1]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Verifying the results\n",
    "\n",
    "At this point, our model is trained. And we can deterime in the *accuracy* by passing in all the test images and labels, figuring out our own labels, and averaging out the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.989\n"
     ]
    }
   ],
   "source": [
    "print(sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extension: Reproduce the Deep MNIST for Experts tutorial\n",
    "\n",
    "The model we used in this exercise,\n",
    "\n",
    "`Input -> CONV (-> ReLU -> Pool) -> FC -> ReLU -> FC -> Softmax -> Loss`\n",
    "\n",
    "is a simplified version of the model used in the the [Deep MNIST for Experts](https://www.tensorflow.org/get_started/mnist/pros) tutorial on the TensorFlow website:\n",
    "\n",
    "`Input -> CONV (-> RELU -> Pool) -> CONV (-> RELU -> Pool) -> FC -> ReLU -> Dropout -> FC -> Softmax -> Loss`\n",
    "\n",
    "You're now in a greate place to extend this code to their mode, further upping the classification accuracy!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}