01 NN_backprop.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vs8PmwWNii0Y"
      },
      "source": [
        "### Your very own neural network\n",
        "\n",
        "In this notebook, we're going to build a neural network using naught but pure numpy and steel nerves. It's going to be fun, I promise!\n",
        "\n",
        "![img](https://github.com/yandexdataschool/Practical_DL/blob/fall19/week01_backprop/frank.png?raw=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "collapsed": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BKBb8RWJii0a",
        "outputId": "c3199773-d79b-4875-c744-2bd4575e6fc7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2022-09-22 14:31:02--  https://raw.githubusercontent.com/yandexdataschool/Practical_DL/35c067adcc1ab364c8803830cdb34d0d50eea37e/week01_backprop/util.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 3782 (3.7K) [text/plain]\n",
            "Saving to: ‘util.py’\n",
            "\n",
            "\rutil.py               0%[                    ]       0  --.-KB/s               \rutil.py             100%[===================>]   3.69K  --.-KB/s    in 0s      \n",
            "\n",
            "2022-09-22 14:31:02 (40.7 MB/s) - ‘util.py’ saved [3782/3782]\n",
            "\n",
            "--2022-09-22 14:31:02--  https://raw.githubusercontent.com/yandexdataschool/Practical_DL/35c067adcc1ab364c8803830cdb34d0d50eea37e/week01_backprop/mnist.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2697 (2.6K) [text/plain]\n",
            "Saving to: ‘mnist.py’\n",
            "\n",
            "mnist.py            100%[===================>]   2.63K  --.-KB/s    in 0s      \n",
            "\n",
            "2022-09-22 14:31:02 (33.0 MB/s) - ‘mnist.py’ saved [2697/2697]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# if running in colab, uncomment this:\n",
        "!wget https://raw.githubusercontent.com/yandexdataschool/Practical_DL/35c067adcc1ab364c8803830cdb34d0d50eea37e/week01_backprop/util.py -O util.py\n",
        "!wget https://raw.githubusercontent.com/yandexdataschool/Practical_DL/35c067adcc1ab364c8803830cdb34d0d50eea37e/week01_backprop/mnist.py -O mnist.py\n",
        "#from __future__ import print_function\n",
        "import numpy as np\n",
        "np.random.seed(42)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JhLyRGKOii0c"
      },
      "source": [
        "Here goes our main class: a layer that can .forward() and .backward()."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "collapsed": true,
        "id": "HCpUeJ7Xii0d"
      },
      "outputs": [],
      "source": [
        "class Layer:\n",
        "    \"\"\"\n",
        "    A building block. Each layer is capable of performing two things:\n",
        "    \n",
        "    - Process input to get output:           output = layer.forward(input)\n",
        "    \n",
        "    - Propagate gradients through itself:    grad_input = layer.backward(input, grad_output)\n",
        "    \n",
        "    Some layers also have learnable parameters which they update during layer.backward.\n",
        "    \"\"\"\n",
        "    def __init__(self):\n",
        "        \"\"\"Here you can initialize layer parameters (if any) and auxiliary stuff.\"\"\"\n",
        "        # A dummy layer does nothing\n",
        "        pass\n",
        "    \n",
        "    def forward(self, input):\n",
        "        \"\"\"\n",
        "        Takes input data of shape [batch, input_units], returns output data [batch, output_units]\n",
        "        \"\"\"\n",
        "        # A dummy layer just returns whatever it gets as input.\n",
        "        return input\n",
        "\n",
        "    def backward(self, input, grad_output):\n",
        "        \"\"\"\n",
        "        Performs a backpropagation step through the layer, with respect to the given input.\n",
        "        \n",
        "        To compute loss gradients w.r.t input, you need to apply chain rule (backprop):\n",
        "        \n",
        "        d loss / d x  = (d loss / d layer) * (d layer / d x)\n",
        "        \n",
        "        Luckily, you already receive d loss / d layer as input, so you only need to multiply it by d layer / d x.\n",
        "        \n",
        "        If your layer has parameters (e.g. dense layer), you also need to update them here using d loss / d layer\n",
        "        \"\"\"\n",
        "        # The gradient of a dummy layer is precisely grad_output, but we'll write it more explicitly\n",
        "        num_units = input.shape[1]\n",
        "        \n",
        "        d_layer_d_input = np.eye(num_units)\n",
        "        \n",
        "        return np.dot(grad_output, d_layer_d_input) # chain rule"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZDpNo_lqii0e"
      },
      "source": [
        "### The road ahead\n",
        "\n",
        "We're going to build a neural network that classifies MNIST digits. To do so, we'll need a few building blocks:\n",
        "- Dense layer - a fully-connected layer, $f(X)=X \\cdot W + \\vec{b}$\n",
        "- ReLU layer (or any other nonlinearity you want)\n",
        "- Loss function - crossentropy\n",
        "- Backprop algorithm - a stochastic gradient descent with backpropageted gradients\n",
        "\n",
        "Let's approach them one at a time.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Br7hZThRii0f"
      },
      "source": [
        "### Nonlinearity layer\n",
        "\n",
        "This is the simplest layer you can get: it simply applies a nonlinearity to each element of your network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "collapsed": true,
        "id": "99O1u11vii0g"
      },
      "outputs": [],
      "source": [
        "class ReLU(Layer):\n",
        "    def __init__(self):\n",
        "        \"\"\"ReLU layer simply applies elementwise rectified linear unit to all inputs\"\"\"\n",
        "        pass\n",
        "    \n",
        "    def forward(self, input):\n",
        "        \"\"\"Apply elementwise ReLU to [batch, input_units] matrix\"\"\"\n",
        "        # <your code. Try np.maximum>\n",
        "        return np.maximum(input,0)\n",
        "    \n",
        "    def backward(self, input, grad_output):\n",
        "        \"\"\"Compute gradient of loss w.r.t. ReLU input\"\"\"\n",
        "        relu_grad = input > 0\n",
        "        return grad_output*relu_grad        "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "collapsed": true,
        "id": "bZlW2096ii0h"
      },
      "outputs": [],
      "source": [
        "# some tests\n",
        "from util import eval_numerical_gradient\n",
        "x = np.linspace(-1,1,10*32).reshape([10,32])\n",
        "l = ReLU()\n",
        "grads = l.backward(x,np.ones([10,32])/(32*10))\n",
        "numeric_grads = eval_numerical_gradient(lambda x: l.forward(x).mean(), x=x)\n",
        "assert np.allclose(grads, numeric_grads, rtol=1e-3, atol=0),\\\n",
        "    \"gradient returned by your layer does not match the numerically computed gradient\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5QwB2BMlii0i"
      },
      "source": [
        "#### Instant primer: lambda functions\n",
        "\n",
        "In python, you can define functions in one line using the `lambda` syntax: `lambda param1, param2: expression`\n",
        "\n",
        "For example: `f = lambda x, y: x+y` is equivalent to a normal function:\n",
        "\n",
        "```\n",
        "def f(x,y):\n",
        "    return x+y\n",
        "```\n",
        "For more information, click [here](http://www.secnetix.de/olli/Python/lambda_functions.hawk).    "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WNFr_QR_ii0j"
      },
      "source": [
        "### Dense layer\n",
        "\n",
        "Now let's build something more complicated. Unlike nonlinearity, a dense layer actually has something to learn.\n",
        "\n",
        "A dense layer applies affine transformation. In a vectorized form, it can be described as:\n",
        "$$f(X)= X \\cdot W + \\vec b $$\n",
        "\n",
        "Where \n",
        "* X is an object-feature matrix of shape [batch_size, num_features],\n",
        "* W is a weight matrix [num_features, num_outputs] \n",
        "* and b is a vector of num_outputs biases.\n",
        "\n",
        "Both W and b are initialized during layer creation and updated each time backward is called."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "collapsed": true,
        "id": "trh5h-3Vii0j"
      },
      "outputs": [],
      "source": [
        "class Dense(Layer):\n",
        "    def __init__(self, input_units, output_units, learning_rate=0.1):\n",
        "        \"\"\"\n",
        "        A dense layer is a layer which performs a learned affine transformation:\n",
        "        f(x) = <x*W> + b\n",
        "        \"\"\"\n",
        "        self.learning_rate = learning_rate\n",
        "        \n",
        "        # initialize weights with small random numbers. We use normal initialization, \n",
        "        # but surely there is something better. Try this once you got it working: http://bit.ly/2vTlmaJ\n",
        "        self.weights = np.random.randn(input_units, output_units)*0.01\n",
        "        self.biases = np.zeros(output_units)\n",
        "        \n",
        "    def forward(self,input):\n",
        "        \"\"\"\n",
        "        Perform an affine transformation:\n",
        "        f(x) = <x*W> + b\n",
        "        \n",
        "        input shape: [batch, input_units]\n",
        "        output shape: [batch, output units]\n",
        "        \"\"\"\n",
        "        return np.dot(input, self.weights) + self.biases\n",
        "    \n",
        "    def backward(self,input,grad_output):\n",
        "        \n",
        "        # compute d f / d x = d f / d dense * d dense / d x\n",
        "        # where d dense/ d x = weights transposed\n",
        "        grad_input = grad_output @ np.transpose(self.weights)\n",
        "\n",
        "        # compute gradient w.r.t. weights and biases\n",
        "        grad_weights = np.transpose(input) @ grad_output\n",
        "        grad_biases = np.sum(grad_output, axis=0)\n",
        "        \n",
        "        assert grad_weights.shape == self.weights.shape and grad_biases.shape == self.biases.shape\n",
        "        # Here we perform a stochastic gradient descent step. \n",
        "        # Later on, you can try replacing that with something better.\n",
        "        self.weights = self.weights - self.learning_rate * grad_weights\n",
        "        self.biases = self.biases - self.learning_rate * grad_biases\n",
        "        \n",
        "        return grad_input"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-qtlNt66ii0k"
      },
      "source": [
        "### Testing the dense layer\n",
        "\n",
        "Here we have a few tests to make sure your dense layer works properly. You can just run them, get 3 \"well done\"s and forget they ever existed.\n",
        "\n",
        "... or not get 3 \"well done\"s and go fix stuff. If that is the case, here are some tips for you:\n",
        "* Make sure you compute gradients for b as __sum of gradients over batch__, not mean over gradients. Grad_output is already divided by batch size.\n",
        "* If you're debugging, try saving gradients in class fields, like \"self.grad_w = grad_w\" or print first 3-5 weights. This helps debugging.\n",
        "* If nothing else helps, try ignoring tests and proceed to network training. If it trains alright, you may be off by something that does not affect network training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mj8EVUzLii0l",
        "outputId": "f641d19b-383d-419a-cc27-4549ede1b076"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Well done!\n"
          ]
        }
      ],
      "source": [
        "l = Dense(128, 150)\n",
        "\n",
        "assert -0.05 < l.weights.mean() < 0.05 and 1e-3 < l.weights.std() < 1e-1,\\\n",
        "    \"The initial weights must have zero mean and small variance. \"\\\n",
        "    \"If you know what you're doing, remove this assertion.\"\n",
        "assert -0.05 < l.biases.mean() < 0.05, \"Biases must be zero mean. Ignore if you have a reason to do otherwise.\"\n",
        "\n",
        "# To test the outputs, we explicitly set weights with fixed values. DO NOT DO THAT IN ACTUAL NETWORK!\n",
        "l = Dense(3,4)\n",
        "\n",
        "x = np.linspace(-1,1,2*3).reshape([2,3])\n",
        "l.weights = np.linspace(-1,1,3*4).reshape([3,4])\n",
        "l.biases = np.linspace(-1,1,4)\n",
        "\n",
        "assert np.allclose(l.forward(x),np.array([[ 0.07272727,  0.41212121,  0.75151515,  1.09090909],\n",
        "                                          [-0.90909091,  0.08484848,  1.07878788,  2.07272727]]))\n",
        "print(\"Well done!\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EwUADELQii0l",
        "outputId": "f895f94d-ed96-40da-dcc8-cd8e61bf8755"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Well done!\n"
          ]
        }
      ],
      "source": [
        "# To test the grads, we use gradients obtained via finite differences\n",
        "\n",
        "from util import eval_numerical_gradient\n",
        "\n",
        "x = np.linspace(-1,1,10*32).reshape([10,32])\n",
        "l = Dense(32,64,learning_rate=0)\n",
        "\n",
        "numeric_grads = eval_numerical_gradient(lambda x: l.forward(x).sum(),x)\n",
        "grads = l.backward(x,np.ones([10,64]))\n",
        "\n",
        "assert np.allclose(grads,numeric_grads,rtol=1e-3,atol=0), \"input gradient does not match numeric grad\"\n",
        "print(\"Well done!\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YIeD-dZqii0m",
        "outputId": "be731f8c-f1e2-48b1-ee33-8f4ab2676eeb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Well done!\n"
          ]
        }
      ],
      "source": [
        "#test gradients w.r.t. params\n",
        "def compute_out_given_wb(w,b):\n",
        "    l = Dense(32,64,learning_rate=1)\n",
        "    l.weights = np.array(w)\n",
        "    l.biases = np.array(b)\n",
        "    x = np.linspace(-1,1,10*32).reshape([10,32])\n",
        "    return l.forward(x)\n",
        "    \n",
        "def compute_grad_by_params(w,b):\n",
        "    l = Dense(32,64,learning_rate=1)\n",
        "    l.weights = np.array(w)\n",
        "    l.biases = np.array(b)\n",
        "    x = np.linspace(-1,1,10*32).reshape([10,32])\n",
        "    l.backward(x,np.ones([10,64]) / 10.)\n",
        "    return w - l.weights, b - l.biases\n",
        "    \n",
        "w,b = np.random.randn(32,64), np.linspace(-1,1,64)\n",
        "\n",
        "numeric_dw = eval_numerical_gradient(lambda w: compute_out_given_wb(w,b).mean(0).sum(),w )\n",
        "numeric_db = eval_numerical_gradient(lambda b: compute_out_given_wb(w,b).mean(0).sum(),b )\n",
        "grad_w,grad_b = compute_grad_by_params(w,b)\n",
        "\n",
        "assert np.allclose(numeric_dw,grad_w,rtol=1e-3,atol=0), \"weight gradient does not match numeric weight gradient\"\n",
        "assert np.allclose(numeric_db,grad_b,rtol=1e-3,atol=0), \"weight gradient does not match numeric weight gradient\"\n",
        "print(\"Well done!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GLNHi7gdii0n"
      },
      "source": [
        "### The loss function\n",
        "\n",
        "Since we want to predict probabilities, it would be logical for us to define softmax nonlinearity on top of our network and compute loss given predicted probabilities. However, there is a better way to do so.\n",
        "\n",
        "If you write down the expression for crossentropy as a function of softmax logits (a), you'll see:\n",
        "\n",
        "$$ loss = - log \\space {e^{a_{correct}} \\over {\\underset i \\sum e^{a_i} } } $$\n",
        "\n",
        "If you take a closer look, ya'll see that it can be rewritten as:\n",
        "\n",
        "$$ loss = - a_{correct} + log {\\underset i \\sum e^{a_i} } $$\n",
        "\n",
        "It's called Log-softmax and it's better than naive log(softmax(a)) in all aspects:\n",
        "* Better numerical stability\n",
        "* Easier to get derivative right\n",
        "* Marginally faster to compute\n",
        "\n",
        "So why not just use log-softmax throughout our computation and never actually bother to estimate probabilities.\n",
        "\n",
        "Here you are! We've defined the both loss functions for you so that you could focus on neural network part."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "collapsed": true,
        "id": "3ON5KZr_ii0n"
      },
      "outputs": [],
      "source": [
        "def softmax_crossentropy_with_logits(logits,reference_answers):\n",
        "    \"\"\"Compute crossentropy from logits[batch,n_classes] and ids of correct answers\"\"\"\n",
        "    logits_for_answers = logits[np.arange(len(logits)),reference_answers]\n",
        "    \n",
        "    xentropy = - logits_for_answers + np.log(np.sum(np.exp(logits),axis=-1))\n",
        "    \n",
        "    return xentropy\n",
        "\n",
        "def grad_softmax_crossentropy_with_logits(logits,reference_answers):\n",
        "    \"\"\"Compute crossentropy gradient from logits[batch,n_classes] and ids of correct answers\"\"\"\n",
        "    ones_for_answers = np.zeros_like(logits)\n",
        "    ones_for_answers[np.arange(len(logits)),reference_answers] = 1\n",
        "    \n",
        "    softmax = np.exp(logits) / np.exp(logits).sum(axis=-1,keepdims=True)\n",
        "    \n",
        "    return (- ones_for_answers + softmax) / logits.shape[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "collapsed": true,
        "id": "l71RKwZtii0o"
      },
      "outputs": [],
      "source": [
        "logits = np.linspace(-1,1,500).reshape([50,10])\n",
        "answers = np.arange(50)%10\n",
        "\n",
        "softmax_crossentropy_with_logits(logits,answers)\n",
        "grads = grad_softmax_crossentropy_with_logits(logits,answers)\n",
        "numeric_grads = eval_numerical_gradient(lambda l: softmax_crossentropy_with_logits(l,answers).mean(),logits)\n",
        "\n",
        "assert np.allclose(numeric_grads,grads,rtol=1e-3,atol=0), \"The reference implementation has just failed. Someone has just changed the rules of math.\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bK6xraXZii0p"
      },
      "source": [
        "### Full network\n",
        "\n",
        "Now let's combine what we've just built into a working neural network. As we announced, we're gonna use this monster to classify handwritten digits, so let's get them loaded."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 459
        },
        "id": "NFeG0eUEii0p",
        "outputId": "a2980060-ad9c-420d-e41c-3dfbb0a781eb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading train-images-idx3-ubyte.gz\n",
            "Downloading train-labels-idx1-ubyte.gz\n",
            "Downloading t10k-images-idx3-ubyte.gz\n",
            "Downloading t10k-labels-idx1-ubyte.gz\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x432 with 4 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW0AAAF1CAYAAAAjhLvUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de7RVdbn/8c8DQt5CRQsJRMyBNMihmGhkpBRYRjrETIuhokOPOIbS0Ybx0/xhaqVRXsp7chS56FHrEGGmqQdRcmgc0VARRM2fEITgDQG1DHh+f6zJOFu+38Vee13mmt+1368x1thrPWtenrl5fJx7Xr7T3F0AgDR0aXYCAIDK0bQBICE0bQBICE0bABJC0waAhNC0ASAhNO2cmdmjZvZvec8LNBq1nQ+adpXM7DUzG9nsPMoxs9PMbJOZbWjzGt7svFB8Ra9tSTKz75nZ62a2zsymmNnHmp1TXmjare1Jd9+5zevRZicE1MrMvibpQkkjJO0t6dOSLmtqUjmiadeZme1mZveZ2Rtm9k72vu9Wk+1rZv+T7SXMNrOebeYfamZPmNlaM3uWvWMURYFq+1RJt7n7C+7+jqQfSzqtymUlh6Zdf10k3a7SHkA/SR9IumGracZKOl1Sb0kbJV0nSWbWR9IfJP1EUk9J35c008w+sfVKzKxfVvz9tpHLQWb2ppm9ZGYXm9l2tW0aOrmi1PZnJT3b5vOzknqZ2e5VbldSaNp15u5vuftMd3/f3ddLulzSEVtNNsPdF7n7e5IulnSimXWVdLKk+939fnff7O4PS1ogaVRkPcvdfVd3X14mlXmS9pf0SUnHSxojaUJdNhKdUoFqe2dJ77b5vOX9x2vYvGTQtOvMzHY0s1vMbJmZrVOpee6aFe4Wf2vzfpmkbpL2UGkP5oRsL2Otma2VNEylvZYOcfdX3f3/Zf+BPC/pR5K+Ve12AUWpbUkbJPVo83nL+/VVLCs5NO36O1/SQEmfd/cekg7P4tZmmr3avO8n6V+S3lSp4GdkexlbXju5+6Q65OVb5QB0VFFq+wVJB7b5fKCk1e7+VhXLSg5NuzbdzGz7Nq/tVPoT7QNJa7OTMJdE5jvZzAaZ2Y4q7QH/l7tvknSHpGPM7Gtm1jVb5vDIyZ52mdnXzaxX9v4zKv2pOrvK7UTnU9jaljRd0hnZenaVNFHS1Go2MkU07drcr1IRb3ldKumXknZQae/iz5L+GJlvhkpF9rqk7SX9uyS5+98kHSvpIklvqLR3MkGRf6fsZM2GbZysGSHpOTN7L8vzt5KuqGIb0TkVtrbd/Y+Sfi5prqTlKh2Gif0PpCUZD0EAgHSwpw0ACaFpA0BCaNoAkBCaNgAkpKambWZHmdlSM3vFzC6sV1JAs1HbKKqqrx7J7oJ6SdKRklZIekrSGHdfvI15uFQFdeXudb9hiNpGEZSr7Vr2tA+V9Ep2u/SHku5W6TpMIHXUNgqrlqbdRx8dZ2BFFvsIMxtnZgvMbEEN6wLyRG2jsBo+VKe7T5Y0WeJPSLQWahvNUMue9kp9dHCYvlkMSB21jcKqpWk/JWmAme1jZt0lfUfSvfVJC2gqahuFVfXhEXffaGbjJT0oqaukKe7+Qt0yA5qE2kaR5TpgFMf9UG+NuOSvGtQ26q0Rl/wBAHJG0waAhNC0ASAhNG0ASAhNGwASQtMGgITQtAEgITRtAEgITRsAEkLTBoCE0LQBICE0bQBISMMfggAA7Tn44IOD2Pjx44PY2LFjo/NPnz49iF1//fVB7Jlnnqkiu2JhTxsAEkLTBoCE0LQBICE0bQBISE0nIs3sNUnrJW2StNHdh9QjKaDZqG0UVU2PG8sKe4i7v1nh9J36kUxdu3YNYrvssktNy4ydYd9xxx2j0w4cODCInXPOOUHsqquuis4/ZsyYIPaPf/wjiE2aNCk6/2WXXRaN16JRjxujthtj8ODB0fgjjzwSxHr06FHTut59990gtvvuu9e0zDzxuDEAaAG1Nm2X9JCZPW1m4+qREFAQ1DYKqdaba4a5+0oz+6Skh83sRXef13aCrOApeqSG2kYh1bSn7e4rs59rJM2SdGhkmsnuPoQTOUgJtY2iqnpP28x2ktTF3ddn778q6Ud1y6zJ+vXrF8S6d+8exA477LDo/MOGDQtiu+66axA7/vjjq8iuOitWrAhi1113XRA77rjjovOvX78+iD377LNB7LHHHqsiu+Jo9drOy6GHBv+f08yZM6PTxk7Ixy6SiNWgJH344YdBLHbScejQodH5Y7e3x5ZZBLUcHuklaZaZbVnOf7r7H+uSFdBc1DYKq+qm7e6vSjqwjrkAhUBto8i45A8AEkLTBoCE1HRHZIdXVsC7xjpyh1atdy/mZfPmzdH46aefHsQ2bNhQ8XJXrVoVxN55550gtnTp0oqXWatG3RHZUUWs7UaJ3XH7uc99LojdcccdQaxv377RZWbnDz4i1pvKjYf985//PIjdfffdFa1HkiZOnBjEfvrTn0anzQt3RAJAC6BpA0BCaNoAkBCaNgAkhKYNAAnp9E9jX758eTT+1ltvBbG8rh6ZP39+NL527dog9uUvfzmIlbv9dsaMGbUlBki65ZZbglhsrPVGiF2lIkk777xzEIsNpzB8+PDo/AcccEBNeeWJPW0ASAhNGwASQtMGgITQtAEgIZ3+ROTbb78djU+YMCGIHX300UHsL3/5S3T+2DjVMQsXLgxiRx55ZHTa9957L4h99rOfDWLnnntuResGtuXggw+Oxr/xjW8EsXK3h2+t3Fjrv//974NY7AHTf//736Pzx/47jA2x8JWvfCU6f6X5FwF72gCQEJo2ACSEpg0ACaFpA0BC2h1P28ymSDpa0hp33z+L9ZR0j6T+kl6TdKK7h0f9w2UlPeZwjx49gli5B43G7ho744wzgtjJJ58cxO66664qsuucahlPm9r+X7Fx5WNjykvx/w5iHnjggSBW7s7JI444IojF7lK89dZbo/O/8cYbFeW0adOmaPz999+vKKdy43k3Qi3jaU+VdNRWsQslzXH3AZLmZJ+B1EwVtY3EtNu03X2epK2viztW0rTs/TRJo+ucF9Bw1DZSVO112r3cfcuzp16X1KvchGY2TtK4KtcD5I3aRqHVfHONu/u2jue5+2RJk6X0j/uhc6G2UUTVXj2y2sx6S1L2c039UgKaitpGoVW7p32vpFMlTcp+zq5bRgW2bt26iqd99913K5ruzDPPDGL33HNPdNpyT1lHXbV8be+3335BLDZsQ7nx4998880gtmrVqiA2bdq0ILZhw4boMv/whz9UFGuUHXbYIYidf/75Qeykk07KI51tandP28zukvSkpIFmtsLMzlCpoI80s5cljcw+A0mhtpGidve03b3cIylG1DkXIFfUNlLEHZEAkBCaNgAkpNOPp90ol156aRCLjU8cu1V25MiR0WU+9NBDNeeFzuNjH/tYNB4bp3rUqFFBrNwQDWPHjg1iCxYsCGKxk3sp6devX7NTiGJPGwASQtMGgITQtAEgITRtAEhIu+Np13VlnXx8hn333TeIxcbnXbt2bXT+uXPnBrHYCaAbb7wxOn+e/9Z5qWU87XoqYm0PHTo0Gn/88ccrmn/EiPjl6uUezpuCcuNpx/7bePLJJ4PYl770pbrnVE4t42kDAAqCpg0ACaFpA0BCaNoAkBDuiMzRX//61yB22mmnBbHbb789Ov8pp5xSUWynnXaKzj99+vQgFhtSE63hmmuuicbNwvNbsZOLKZ9wLKdLl/h+akrDHrOnDQAJoWkDQEJo2gCQEJo2ACSkkseNTTGzNWa2qE3sUjNbaWYLs1c4riNQcNQ2UlTJ1SNTJd0gaetLD37h7uHAvOiQWbNmBbGXX345Om3saoDYrcZXXHFFdP699947iF1++eVBbOXKldH5W9BUtUhtH3300UFs8ODB0Wljt2zfe++9dc+piMpdJRL7nSxcuLDR6VSl3T1td58n6e0ccgFyRW0jRbUc0x5vZs9lf2LuVreMgOajtlFY1TbtmyXtK2mwpFWSri43oZmNM7MFZhYORwcUD7WNQquqabv7anff5O6bJf2HpEO3Me1kdx/i7kOqTRLIC7WNoqvqNnYz6+3uW+5/Pk7Som1Nj45ZtCj+6zzxxBOD2DHHHBPEyt0Gf9ZZZwWxAQMGBLEjjzyyvRRbVqq1HXuIbvfu3aPTrlmzJojdc889dc8pT7GHGMcerl3OI488EsR+8IMf1JJSw7TbtM3sLknDJe1hZiskXSJpuJkNluSSXpMUdgOg4KhtpKjdpu3uYyLh2xqQC5Arahsp4o5IAEgITRsAEsJ42gmJPfB3xowZQezWW2+Nzr/dduE/9+GHHx7Ehg8fHp3/0Ucf3XaCSMI///nPIJbKuOqxE46SNHHixCA2YcKEILZixYro/FdfHV7ZuWHDhg5mlw/2tAEgITRtAEgITRsAEkLTBoCE0LQBICFcPVJABxxwQDT+rW99K4gdcsghQSx2lUg5ixcvDmLz5s2reH6kJ5Wxs2PjgceuCJGkb3/720Fs9uzZQez444+vPbEmY08bABJC0waAhNC0ASAhNG0ASAgnInM0cODAIDZ+/Pgg9s1vfjM6/5577lnT+jdt2hTEYrcvl3v4KYrLzCqKSdLo0aOD2Lnnnlv3nDrie9/7XhC7+OKLg9guu+wSnf/OO+8MYmPHjq09sQJiTxsAEkLTBoCE0LQBICE0bQBISCXPiNxL0nRJvVR6bt5kd7/WzHpKukdSf5WepXeiu7/TuFSLqdzJwTFjwidZxU469u/fv94pacGCBdH45ZdfHsRSuTuuEVqptt29opgUr9nrrrsuiE2ZMiU6/1tvvRXEhg4dGsROOeWUIHbggQdGl9m3b98gtnz58iD24IMPRue/6aabovFWVMme9kZJ57v7IElDJZ1jZoMkXShpjrsPkDQn+wykhNpGctpt2u6+yt2fyd6vl7REUh9Jx0qalk02TVJ4HRFQYNQ2UtSh67TNrL+kgyTNl9TL3bdc5Pu6Sn9ixuYZJ2lc9SkCjUdtIxUVn4g0s50lzZR0nruva/udlw6eRQ+guftkdx/i7kNqyhRoEGobKamoaZtZN5WK+k53/20WXm1mvbPve0ta05gUgcahtpGaSq4eMUm3SVri7te0+epeSadKmpT9DAevTVivXuFfxIMGDQpiN9xwQ3T+z3zmM3XPaf78+UHsyiuvDGKxcYQlbk/fWmet7a5duwaxs88+O4iVG3t63bp1QWzAgAE15fTEE08Esblz5waxH/7whzWtpxVUckz7i5JOkfS8mS3MYhepVNC/NrMzJC2TdGJjUgQahtpGctpt2u7+uKT4yDPSiPqmA+SH2kaKuCMSABJC0waAhFi5W10bsjKz/FYW0bNnzyB2yy23RKeNPVT005/+dN1zip2Aufrqq6PTxm7h/eCDD+qeU0rcvdzhjVw1u7Zjt4H/5je/iU4bexh0TLnxuCvtGbHb3e++++7otM0ez7uIytU2e9oAkBCaNgAkhKYNAAmhaQNAQpI/Efn5z38+Gp8wYUIQO/TQQ4NYnz596p2SJOn9998PYrExi6+44oog9t577zUkp1bEicjyevfuHY2fddZZQWzixIlBrCMnIq+99togdvPNNwexV155JbpMhDgRCQAtgKYNAAmhaQNAQmjaAJAQmjYAJCT5q0cmTZoUjceuHumIxYsXB7H77rsviG3cuDE6f+xW9LVr19aUE0JcPYJWxdUjANACaNoAkBCaNgAkpN2mbWZ7mdlcM1tsZi+Y2blZ/FIzW2lmC7PXqManC9QPtY0UtXsiMnsadW93f8bMPi7paUmjVXpu3gZ3v6rilXGyBnVWy4lIahtFVq62K3lG5CpJq7L3681siaTGDNgB5IjaRoo6dEzbzPpLOkjS/Cw03syeM7MpZrZbnXMDckNtIxUVN20z21nSTEnnufs6STdL2lfSYJX2VqLPyDKzcWa2wMwW1CFfoO6obaSkoptrzKybpPskPeju10S+7y/pPnffv53lcNwPdVXrzTXUNoqq6ptrrDSo7m2SlrQt6uwkzhbHSVpUa5JAnqhtpKiSq0eGSfqTpOclbc7CF0kao9Kfjy7pNUlnZSd2trUs9kZQVzVePUJto7DK1XbyY4+gc2PsEbQqxh4BgBZA0waAhNC0ASAhNG0ASAhNGwASQtMGgITQtAEgITRtAEhIu0Oz1tmbkpZl7/fIPreSVtumom/P3s1OoI0ttV3031k12Kb8la3tXO+I/MiKzRa4+5CmrLxBWm2bWm178tCKvzO2qVg4PAIACaFpA0BCmtm0Jzdx3Y3SatvUatuTh1b8nbFNBdK0Y9oAgI7j8AgAJCT3pm1mR5nZUjN7xcwuzHv99ZA97HWNmS1qE+tpZg+b2cvZz6QeBmtme5nZXDNbbGYvmNm5WTzp7coTtV1MrVbbuTZtM+sq6UZJX5c0SNIYMxuUZw51MlXSUVvFLpQ0x90HSJqTfU7JRknnu/sgSUMlnZP926S+XbmgtgutpWo77z3tQyW94u6vuvuHku6WdGzOOdTM3edJenur8LGSpmXvp0kanWtSNXL3Ve7+TPZ+vaQlkvoo8e3KEbVdUK1W23k37T6S/tbm84os1gp6tXmO4OuSejUzmVpkTyA/SNJ8tdB2NRi1nYBWqG1ORDaAly7JSfKyHDPbWdJMSee5+7q236W8XaiPlGugVWo776a9UtJebT73zWKtYLWZ9Zak7OeaJufTYWbWTaWivtPdf5uFk9+unFDbBdZKtZ13035K0gAz28fMukv6jqR7c86hUe6VdGr2/lRJs5uYS4eZmUm6TdISd7+mzVdJb1eOqO2CarnadvdcX5JGSXpJ0l8l/d+811+nbbhL0ipJ/1Lp2OUZknZX6Qz0y5L+W1LPMvM+Kunfqlxv1fNWsOxhKv15+JykhdlrVKXbxYvaprbzeeU9NKvc/X5J9+e93npy9zFm9pqkr7v7f7f5akSTUtomM5sj6SuSurn7xtg07v64JCuziEJuV9FQ2/kws/0lXS3pYEm7u3u5upXUerXNicgWZ2YnSerW7DyAOvqXpF+r9FdAp0PTrjMz283M7jOzN8zsnex9360m29fM/sfM1pnZbDPr2Wb+oWb2hJmtNbNnzWx4DbnsIukSSf+n2mUAWxSltt19qbvfJumFGjYnWTTt+usi6XaVnjzRT9IHkm7Yapqxkk6X1Fulu7WukyQz6yPpD5J+IqmnpO9Lmmlmn9h6JWbWLyv+ftvI5QpJN6t0DSpQqyLVdqdF064zd3/L3We6+/teuvvqcklHbDXZDHdf5O7vSbpY0onZbdAnS7rf3e93983u/rCkBSqdNNl6PcvdfVd3Xx7Lw8yGSPqipOvruHnoxIpS251d7iciW52Z7SjpFyqN37BlAJqPm1lXd9+UfW5759wylY4576HSHswJZnZMm++7SZrbwRy6SLpJ0rnuvrF0xRNQmyLUNmjajXC+pIGSPu/ur5vZYEl/0UfPXre9CaOfSidW3lSp4Ge4+5k15tBD0hBJ92QNu2sWX2FmJ7j7n2pcPjqnItR2p8fhkdp0M7Pt27y2k/RxlY71rc1OwlwSme9kMxuU7bn8SNJ/ZXsqd0g6xsy+ZmZds2UOj5zsac+7kj4laXD22vIn6MEqjbkAtKeotS0r2V5S9+zz9mb2sWo3NDU07drcr1IRb3ldKumXknZQae/iz5L+GJlvhkpDYL4uaXtJ/y5J7v43lUYeu0jSGyrtnUxQ5N8pO1mzIXayxkte3/LKliVJq700Ah3QnkLWdmbvLKctV498IGlpB7cvWTxuDAASwp42ACSEpg0ACaFpA0BCaNoAkJCamra1wNOngRhqG0VV9dUj2a2pL0k6UqVxd5+SNMbdF29jHi5VQV21NyxnNahtFEG52q5lT7slnj4NRFDbKKxamnZFT582s3FmtsDMFtSwLiBP1DYKq+Fjj7j7ZEmTJf6ERGuhttEMtexpt/LTp9G5UdsorFqadis/fRqdG7WNwqr68Eg2TvN4SQ+qNPTnFHfvlI//QWuhtlFkuQ4YxXE/1FsjLvmrBrWNemvEJX8AgJzRtAEgITRtAEgITRsAEkLTBoCE0LQBICE0bQBICE0bABJC0waAhNC0ASAhNG0ASAhNGwASQtMGgITQtAEgITRtAEgITRsAEkLTBoCE1PQ0djN7TdJ6SZskbXT3IfVICmg2ahtFVVPTznzZ3d+sw3JQECNGjIjG77zzziB2xBFHBLGlS5fWPacmobYTMXHixCB22WWXBbEuXeIHF4YPHx7EHnvssZrzagQOjwBAQmpt2i7pITN72szG1SMhoCCobRRSrYdHhrn7SjP7pKSHzexFd5/XdoKs4Cl6pIbaRiHVtKft7iuzn2skzZJ0aGSaye4+hBM5SAm1jaKqek/bzHaS1MXd12fvvyrpR3XLrEKHH354NL777rsHsVmzZjU6nZZwyCGHRONPPfVUzpk0R1FqG6HTTjstGr/ggguC2ObNmyterrtXm1Luajk80kvSLDPbspz/dPc/1iUroLmobRRW1U3b3V+VdGAdcwEKgdpGkXHJHwAkhKYNAAmpxx2RTRW7k0mSBgwYEMQ4ERmK3SG2zz77RKfde++9g1h23BfIRawGJWn77bfPOZPmYU8bABJC0waAhNC0ASAhNG0ASAhNGwASkvzVI2PHjo3Gn3zyyZwzSVPv3r2D2Jlnnhmd9o477ghiL774Yt1zAiRp5MiRQey73/1uxfPHavPoo4+OTrt69erKE2sy9rQBICE0bQBICE0bABJC0waAhCR/IrLcgzpRmVtvvbXiaV9++eUGZoLObNiwYUHs9ttvD2K77LJLxcu88sorg9iyZcs6llgB0fEAICE0bQBICE0bABJC0waAhLR7ItLMpkg6WtIad98/i/WUdI+k/pJek3Siu7/TuDRLDjjggCDWq1evRq+2pXXkxM7DDz/cwEzyV6Ta7uxOPfXUIPapT32q4vkfffTRIDZ9+vRaUiqsSva0p0o6aqvYhZLmuPsASXOyz0BqporaRmLabdruPk/S21uFj5U0LXs/TdLoOucFNBy1jRRVe512L3dflb1/XVLZYxRmNk7SuCrXA+SN2kah1Xxzjbu7mfk2vp8sabIkbWs6oGiobRRRtVePrDaz3pKU/VxTv5SApqK2UWjV7mnfK+lUSZOyn7PrltE2jBo1KojtsMMOeay6JcSutCn35PWYlStX1jOdompKbXcWe+yxRzR++umnB7HNmzcHsbVr10bn/8lPflJbYglpd0/bzO6S9KSkgWa2wszOUKmgjzSzlyWNzD4DSaG2kaJ297TdfUyZr0bUORcgV9Q2UsQdkQCQEJo2ACQkqfG0Bw4cWPG0L7zwQgMzSdNVV10VxGInJ1966aXo/OvXr697Tmhd/fv3D2IzZ86saZnXX399ND537tyalpsS9rQBICE0bQBICE0bABJC0waAhCR1IrIjnnrqqWanUHc9evQIYkcdtfXIotLJJ58cnf+rX/1qRev58Y9/HI2XuxsNiInVZmxM/HLmzJkTxK699tqacmoF7GkDQEJo2gCQEJo2ACSEpg0ACWnZE5E9e/as+zIPPPDAIGZm0WlHjhwZxPr27RvEunfvHsROOumk6DK7dAn/H/vBBx8Esfnz50fn/+c//xnEttsuLIGnn346Oj9QzujR4VPZJk2qfIDExx9/PIjFHvb77rvvdiyxFsSeNgAkhKYNAAmhaQNAQmjaAJCQSh43NsXM1pjZojaxS81spZktzF7hwxuBgqO2kaJKrh6ZKukGSdO3iv/C3cMBmhsodqWEu0en/dWvfhXELrrooprWH7sFt9zVIxs3bgxi77//fhBbvHhxEJsyZUp0mQsWLAhijz32WBBbvXp1dP4VK1YEsdiDkV988cXo/C1oqgpS2ylpxDjZr776ahArV8edXbt72u4+T9LbOeQC5IraRopqOaY93syey/7E3K1uGQHNR22jsKpt2jdL2lfSYEmrJF1dbkIzG2dmC8ws/NseKB5qG4VWVdN299XuvsndN0v6D0mHbmPaye4+xN2HVJskkBdqG0VX1W3sZtbb3VdlH4+TtGhb09fL2WefHcSWLVsWnfawww6r+/qXL18exH73u99Fp12yZEkQ+/Of/1z3nGLGjRsXjX/iE58IYrETQJ1Zs2o7JRdccEEQ27x5c03L7Mgt751du03bzO6SNFzSHma2QtIlkoab2WBJLuk1SWc1MEegIahtpKjdpu3uYyLh2xqQC5Arahsp4o5IAEgITRsAEpL8eNo/+9nPmp1C4YwYMaLiaWu9kw2ta/DgwdF4pQ+Ijpk9e3Y0vnTp0qqX2dmwpw0ACaFpA0BCaNoAkBCaNgAkhKYNAAlJ/uoR1GbWrFnNTgEF9dBDD0Xju+1W2cCHsWEbTjvttFpSgtjTBoCk0LQBICE0bQBICE0bABLCiUgAUbvvvns0XunY2TfddFMQ27BhQ005gT1tAEgKTRsAEkLTBoCE0LQBICGVPCNyL0nTJfVS6bl5k939WjPrKekeSf1Vepbeie7+TuNSRa3MLIjtt99+QSyvBxA3G7X9v26//fYg1qVLbft0TzzxRE3zI66Sf5WNks5390GShko6x8wGSbpQ0hx3HyBpTvYZSAm1jeS027TdfZW7P5O9Xy9piaQ+ko6VNC2bbJqk0Y1KEmgEahsp6tB12mbWX9JBkuZL6uXuq7KvXlfpT8zYPOMkjas+RaDxqG2kouKDVma2s6SZks5z93Vtv3N3V+mYYMDdJ7v7EHcfUlOmQINQ20hJRU3bzLqpVNR3uvtvs/BqM+udfd9b0prGpAg0DrWN1FRy9YhJuk3SEne/ps1X90o6VdKk7Gf8McsojNJO40fVeoVAyjprbceesj5y5MggVu529Q8//DCI3XjjjUFs9erVVWSH9lRyTPuLkk6R9LyZLcxiF6lU0L82szMkLZN0YmNSBBqG2kZy2m3a7v64pPAC35IR9U0HyA+1jRR13r+NASBBNG0ASAjjaXdyX/jCF4LY1KlT808Eudl1112D2J577lnx/CtXrgxi3//+92vKCZVjTxsAEkLTBoCE0LQBICE0bQBICF+93XIAAAQPSURBVCciO5HYeNoA0sKeNgAkhKYNAAmhaQNAQmjaAJAQmjYAJISrR1rQAw88EI2fcMIJOWeCInrxxReDWOzJ6cOGDcsjHXQQe9oAkBCaNgAkhKYNAAlpt2mb2V5mNtfMFpvZC2Z2bha/1MxWmtnC7DWq8ekC9UNtI0UWe9jrRyYoPY26t7s/Y2Yfl/S0pNEqPTdvg7tfVfHKzLa9MqCD3L3qe/OpbRRZudqu5BmRqyStyt6vN7MlkvrUNz0gf9Q2UtShY9pm1l/SQZLmZ6HxZvacmU0xs93qnBuQG2obqai4aZvZzpJmSjrP3ddJulnSvpIGq7S3cnWZ+caZ2QIzW1CHfIG6o7aRknaPaUuSmXWTdJ+kB939msj3/SXd5+77t7Mcjvuhrmo5pi1R2yiucrVdydUjJuk2SUvaFnV2EmeL4yQtqjVJIE/UNlJUydUjwyT9SdLzkjZn4YskjVHpz0eX9Jqks7ITO9taFnsjqKsarx6htlFY5Wq7osMj9UJho95qPTxSL9Q26q3qwyMAgOKgaQNAQmjaAJAQmjYAJISmDQAJoWkDQEJo2gCQEJo2ACQk7wf7vilpWfZ+j+xzK2m1bSr69uzd7ATa2FLbRf+dVYNtyl/Z2s71jsiPrNhsgbsPacrKG6TVtqnVticPrfg7Y5uKhcMjAJAQmjYAJKSZTXtyE9fdKK22Ta22PXloxd8Z21QgTTumDQDoOA6PAEBCcm/aZnaUmS01s1fM7MK8118P2cNe15jZojaxnmb2sJm9nP1M6mGwZraXmc01s8Vm9oKZnZvFk96uPFHbxdRqtZ1r0zazrpJulPR1SYMkjTGzQXnmUCdTJR21VexCSXPcfYCkOdnnlGyUdL67D5I0VNI52b9N6tuVC2q70FqqtvPe0z5U0ivu/qq7fyjpbknH5pxDzdx9nqS3twofK2la9n6apNG5JlUjd1/l7s9k79dLWiKpjxLfrhxR2wXVarWdd9PuI+lvbT6vyGKtoFeb5wi+LqlXM5OpRfYE8oMkzVcLbVeDUdsJaIXa5kRkA3jpkpwkL8sxs50lzZR0nruva/tdytuF+ki5BlqltvNu2isl7dXmc98s1gpWm1lvScp+rmlyPh1mZt1UKuo73f23WTj57coJtV1grVTbeTftpyQNMLN9zKy7pO9IujfnHBrlXkmnZu9PlTS7ibl0mJmZpNskLXH3a9p8lfR25YjaLqhWq+3cb64xs1GSfimpq6Qp7n55rgnUgZndJWm4SiOFrZZ0iaTfSfq1pH4qjfZ2ortvfUKnsMxsmKQ/SXpe0uYsfJFKx/6S3a48UdvF1Gq1zR2RAJAQTkQCQEJo2gCQEJo2ACSEpg0ACaFpA0BCaNoAkBCaNgAkhKYNAAn5/6VLQlsOxoxAAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "from mnist import load_dataset\n",
        "X_train, y_train, X_val, y_val, X_test, y_test = load_dataset(flatten=True)\n",
        "\n",
        "plt.figure(figsize=[6,6])\n",
        "for i in range(4):\n",
        "    plt.subplot(2,2,i+1)\n",
        "    plt.title(\"Label: %i\"%y_train[i])\n",
        "    plt.imshow(X_train[i].reshape([28,28]),cmap='gray');"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rKxU-S7Jii0q"
      },
      "source": [
        "We'll define network as a list of layers, each applied on top of previous one. In this setting, computing predictions and training becomes trivial."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "collapsed": true,
        "id": "5UHlM6Fwii0q"
      },
      "outputs": [],
      "source": [
        "network = []\n",
        "network.append(Dense(X_train.shape[1], 100))\n",
        "network.append(ReLU())\n",
        "network.append(Dense(100, 200))\n",
        "network.append(ReLU())\n",
        "network.append(Dense(200, 10))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "collapsed": true,
        "id": "M4kJr-XEii0q"
      },
      "outputs": [],
      "source": [
        "def forward(network, X):\n",
        "    \"\"\"\n",
        "    Compute activations of all network layers by applying them sequentially.\n",
        "    Return a list of activations for each layer. \n",
        "    Make sure last activation corresponds to network logits.\n",
        "    \"\"\"\n",
        "    activations = []\n",
        "    input = X\n",
        "\n",
        "    counter = 0\n",
        "    for layer_net in network:\n",
        "      if counter == 0:\n",
        "        tmp_h = layer_net.forward(input)\n",
        "        activations.append(tmp_h)\n",
        "      else:\n",
        "        tmp_h = layer_net.forward(activations[counter -1])\n",
        "        activations.append(tmp_h)\n",
        "      counter = counter +1\n",
        "    # <your code here>\n",
        "        \n",
        "    assert len(activations) == len(network)\n",
        "    return activations\n",
        "\n",
        "def predict(network, X):\n",
        "    \"\"\"\n",
        "    Use network to predict the most likely class for each sample.\n",
        "    \"\"\"\n",
        "    logits = forward(network, X)[-1]\n",
        "    return logits.argmax(axis=-1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pn5pLDKEii0r"
      },
      "source": [
        "### Backprop\n",
        "\n",
        "You can now define the backpropagation step for the neural network. Please read the docstring."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "collapsed": true,
        "id": "nkET_mz7ii0r"
      },
      "outputs": [],
      "source": [
        "def train(network,X,y):\n",
        "    \"\"\"\n",
        "    Train your network on a given batch of X and y.\n",
        "    You first need to run forward to get all layer activations.\n",
        "    You can estimate loss and loss_grad, obtaining dL / dy_pred\n",
        "    Then you can run layer.backward going from last layer to first, \n",
        "    propagating the gradient of input to previous layers.\n",
        "    \n",
        "    After you called backward for all layers, all Dense layers have already made one gradient step.\n",
        "    \"\"\"\n",
        "    \n",
        "    # Get the layer activations\n",
        "    layer_activations = forward(network,X)\n",
        "    layer_inputs = [X] + layer_activations  #layer_input[i] is an input for network[i]\n",
        "    logits = layer_activations[-1]\n",
        "    \n",
        "    # Compute the loss and the initial gradient\n",
        "    loss = softmax_crossentropy_with_logits(logits,y)\n",
        "    loss_grad = grad_softmax_crossentropy_with_logits(logits,y)\n",
        "    loss_grad_wrt_tmp = loss_grad\n",
        "    counter = len(network)\n",
        "    while (counter != 0):\n",
        "        loss_grad_wrt_tmp = network[counter-1].backward(layer_inputs[counter-1],loss_grad_wrt_tmp)\n",
        "        counter = counter -1\n",
        "\n",
        "\n",
        "    \n",
        "    \n",
        "    # propagate gradients through network layers using .backward\n",
        "    # hint: start from last layer and move to earlier layers\n",
        "    #<YOUR CODE>\n",
        "\n",
        "\n",
        "        \n",
        "    return np.mean(loss)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O54aXPYuii0s"
      },
      "source": [
        "Instead of tests, we provide you with a training loop that prints training and validation accuracies on every epoch.\n",
        "\n",
        "If your implementation of forward and backward are correct, your accuracy should grow from 90~93% to >97% with the default network."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fYmQYHYjii0s"
      },
      "source": [
        "### Training loop\n",
        "\n",
        "As usual, we split data into minibatches, feed each such minibatch into the network and update weights."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "collapsed": true,
        "id": "HmVk1gEmii0s"
      },
      "outputs": [],
      "source": [
        "from tqdm import trange\n",
        "def iterate_minibatches(inputs, targets, batchsize, shuffle=False):\n",
        "    assert len(inputs) == len(targets)\n",
        "    if shuffle:\n",
        "        indices = np.random.permutation(len(inputs))\n",
        "    for start_idx in trange(0, len(inputs) - batchsize + 1, batchsize):\n",
        "        if shuffle:\n",
        "            excerpt = indices[start_idx:start_idx + batchsize]\n",
        "        else:\n",
        "            excerpt = slice(start_idx, start_idx + batchsize)\n",
        "        yield inputs[excerpt], targets[excerpt]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "collapsed": true,
        "id": "4QT9yHR1ii0s"
      },
      "outputs": [],
      "source": [
        "from IPython.display import clear_output\n",
        "train_log = []\n",
        "val_log = []"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "collapsed": true,
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 317
        },
        "id": "NDc03pk2ii0t",
        "outputId": "302ad9e1-b9db-47f1-cc6a-d00c00af714f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 24\n",
            "Train accuracy: 0.99998\n",
            "Val accuracy: 0.9803\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD4CAYAAADrRI2NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU1fn48c+TfV8gIZCFHYSwExYX1KhFse4iglrcqui3Yu23P9uqdV+qbbX9uraiRaUuaFEQFdyQiFZRdpKwhjULSyBkJevM+f1xhzDEkGUyySQzz/v1mldm7vqc3OS5Z84991wxxqCUUsp3+Hk6AKWUUh1LE79SSvkYTfxKKeVjNPErpZSP0cSvlFI+JsDTATQUFxdn+vbt6/L6FRUVhIeHuy+gLkTL7ptlB98uvy+XHY6Xf82aNYeMMfEtWafTJf6+ffuyevVql9fPyMggPT3dfQF1IVr2dE+H4TG+XH5fLjscL7+I7GnpOtrUo5RSPkYTv1JK+RhN/Eop5WM08SullI/RxK+UUj6m2cQvInNF5KCIZJ1kvojIcyKSIyIbRWSs07wbRGS743WDOwNXSinlmpbU+F8HpjQx/0JgkOM1C/gHgIh0Ax4CJgITgIdEJLYtwSqllGq7ZvvxG2NWiEjfJha5DJhnrPGdV4pIjIj0AtKBL4wxRQAi8gXWCeSdtgatlOp6jDFU19mprLFRWet41diorrNRWWOnstZGVa2NOrsdmx3sdkOd3WAzpv693fHZZj/+2rW7hrU1W0EEPwFBEAEB66cIHHuPtYwBjAGD4djI9MYYx7Tj8+zHPrRFC+LyE0GAhKgQLh+T1Lb9tYA7buBKAnKdPuc5pp1s+k+IyCysbwskJCSQkZHhcjDl5eVtWr8r07JneDoMj2nP8htjKKoy5JfbqbJBdZ2xftoM1XVQZTNU26CqrsFPm6HGBjXHftrbJTzLjpx23LiVrF3R2lNG/2g/Ykq2t2odV459p7hz1xgzB5gDMG7cONOWu/B8+S4+LXu6p8PocJl5JbzyzU5y91Vx9shEhiVGk5oYRWJ0SH2NsrVqbXY27ytl9e4jrNl7hDW7j7C/tKrRZf0EwoMDCA8KICzYn/CgAGIi/AkPDiA0yJ+wQH9Cg/wJDfQnxPEKDfQjNMj5s7VMSIA/IYF+BPj74S+Cv7/gL4KfHwT4+Z3w3s8Paxk/4euvv64/9sZYtXRjzElq9WA3pr72f+xXJD+pkTtq6S7+DhtqaVx+IoQG+bdq26787bsj8ecDKU6fkx3T8rGae5ynZ7hhf0r5vDV7jvDCV9tZvrWQqJAAwv3tPLtse30iiQ0LJDUximGJ0QxLjGJYYhT94iLw9/tpIis5WsvavUdYs+cIq/cUsSG3hMpaGwBJMaGM79eNcX1iSU2MIiokkLAgK7GHBfkTHODntuToDiKCv4DrdfT20dnickfiXwzMFpH5WBdyS4wx+0TkM+BPThd0zwfudcP+lPJZK3ce5vmvtvPfnMPEhgXyuwtOYeZpfVi78r9MOH0Sm/eVsamghOyCUjbtK+X173ZTU2e1sYQE+jGkp3US6B8fQc7BMtbsOcK2A+UA+PsJqb2imD4+hXF9Y0nrE0uv6FBPFle1k2YTv4i8g1VzjxORPKyeOoEAxph/AkuAnwM5wFHgJse8IhF5DFjl2NSjxy70KuULjDHkFlXSIyqYkMDWfX1vuJ1vcw7x/LIcftxdRFxEMH/8+VCundib8ODj/8JhQQGk9bES9jG1Njs7CsvJzi91nAxKWLyhgLKqOqJCAhjbJ5ZLRyUytk8so1NiCAvqFK2/qp21pFfPNc3MN8AdJ5k3F5jrWmhKdU1lVbUsXJfPWyv3svVAGQF+wik9IxmZHM3I5BhGJEVzSs9IAv2b7k1tjOGrLQd5/qsc1ucW0zMqhIcvSWXGhN4tPpEE+lu1/CE9o5iadny7h8pr6B4ehF8jTT/K++npXSk32VRQyps/7GHRunyO1tgYkRTNAxencri8msz8EpZk7uedH62ObkEBfgztFcXIpGhGJEczMjmagfERBPj7YbcbPt+0n+e/yiG7oJTk2FD+dMUIpqYlERzg+jeHY0SE+MjgNm9HdV2a+JVqg6paG0uz9vHmyr2s2XOE4AA/LhmVyMxT+zAqJeaEZY0x7C06ysa8EjLzS9iYV8zCdfn8e6U1jHpIoB/DEqMpraxl+8Fy+sWF89erRnL5mKRmvx0o1Rqa+JVywZ7DFbz9w17eW53LkaO19IsL5/6LhnJVWjIxYUGNriMi9OkeTp/u4VwyKhGwblLadbiCzLwSxwmhmLDgAJ6dMZqLRyY22gtHqbbSxK9UC9nsVpv7myv38PW2Qvz9hMlDE/jFqX04fUB3l9rL/fyEAfERDIiP6JA7NpUCTfxKNWvbgTI+WJvPonX57C+tIiEqmN/8bBAzxvemZ3SIp8NTqtU08SvViMKyahZvKOCDtXlkF5Ti7yecPTiehy9N5byhCdrmrro0TfxKOVTW2Ph8034Wrsvnm+2HsNkNI5KieeiSVC4ZlUhchPaEUd5BE7/yaXa7YeWuwyxcm8/SrP2UV9eRGB3CbWf158qxSQzsEenpEJVyO038yiflHTnK2z/s5cP1BeQXVxIRHMCFw3tyxdgkTu3n2oVapboKTfyqUzDGkF9cyfrcYrLySxnaK5KLRvQiwM1t6YfLq3lheQ5vrdxLnd3OWYPj+f2UUzg/tWerR0VUqqvSxK88ory6jo25xazLLWZ9bjHr9hZzqLwasIb6tRt45vNt3HZ2f6aOTW7TWDfH9vfqNzt5ZcVOKmttXD0uhV+fN4jEGB2ETPkeTfyq3dnshu0Hy1i/10rw63OL2XawrH4I4f5x4Zw1KI4xvWMYnRLL4J4RfL21kBczdvDHhVk8++V2bjmzH9dO7ENEcOv+ZKvrbLz9w15e+CqHwxU1TBnWk7svOIWBPSLaoaRKdQ2a+FW7qaq18dTSLfxndS4VNdb47tGhgYxOieHCET0ZnRLD6JSYRu90PX9YTyanJvDdjsO8lJHDn5Zs4cXlO7jx9L7ceHpfYsMbvzv2GLvd8OGGfJ75fBt5Ryo5rX93/nDhEEY3GEZBKV+kiV+1ix2F5cx+ex2b95Vy5dgkJg2MY0zvWPp2D2vxgztEhDMGxnHGwDjW5xbz0vIcnl22nVe+2cm1E3pzy5n9f3IDlTGG5VsP8pdPt7JlfxnDEqP40xUjOHNQXKd6YIhSnqSJX7ndwnV5/HFhFsEBfsy9cRznDklo8zZHp8Qw5/pxbDtQxj8zdvDad7uZ9/0epqYlcdtZAwBYs6eIPy/dyo+7i+jTPYznrhnDxSN6aQ8dpRrQxK/c5mhNHQ99mM1/1uQxoW83nr1mtNuf4DQ4IZK/TR/N/04ezJwVO3l3dS7vrsolJdKPPZ9+T3xkMI9dPpwZ41P07lqlTkITv3KLLftLmf32OnYUlnPnuQO567xBbu+K6SylWxiPXT6cO88byNxvd/PRmp3cff5gbp7UT58ipVQz9D9EtYkxhvmrcnl4cTaRIYH8++aJTBoU12H77xEZwj0XDuHU0P2kpw/qsP0q1ZW1qEomIlNEZKuI5IjIPY3M7yMiy0Rko4hkiEiy07w/i0iW4zXdncErzyqrquXX89dz7weZjO/bjSV3TerQpK+Uck1LHrbuD7wITAbygFUistgYs8lpsaeBecaYN0TkXOBJYKaIXASMBUYDwUCGiCw1xpS6uyCqY2Xll3DH22vJLTrK7y44hf85e4BeRFWqi2hJjX8CkGOM2WmMqQHmA5c1WCYV+MrxfrnT/FRghTGmzhhTAWwEprQ9bOUpxhhe++8urnzpO6pr7cyfdRp3nDNQk75SXYiYY7dPnmwBkauAKcaYWxyfZwITjTGznZZ5G/jBGPOsiFwJvA/EAWnAQ1jfFsKAH4EXjTHPNNjHLGAWQEJCQtr8+fNdLlB5eTkREb55V2Z7l728xjA3q5q1B22MivfnlhHBRAZ1joTvy8cdfLv8vlx2OF7+c845Z40xZlxL1nHXxd27gRdE5EZgBZAP2Iwxn4vIeOA7oBD4HrA1XNkYMweYAzBu3DiTnp7uciAZGRm0Zf2urD3L/mnWfh5elEnxUTv3XzSUX07q16luiPLl4w6+XX5fLju4Vv6WJP58IMXpc7JjWj1jTAFwJYCIRABTjTHFjnlPAE845r0NbGtVhMqjiipqeGhxNh9tKCC1VxTzbp5IamKUp8NSSrVBSxL/KmCQiPTDSvgzgGudFxCROKDIGGMH7gXmOqb7AzHGmMMiMhIYCXzuxvhVO/o0ax/3L8qipLKW304ezP+kD9CbopTyAs0mfmNMnYjMBj4D/IG5xphsEXkUWG2MWQykA0+KiMFq6rnDsXog8I2jSaAU+IUxps79xVDuVFRRw4MfZvHxxn0MS4zi37+cyNBeWstXylu0qI3fGLMEWNJg2oNO7xcACxpZrwqrZ4/qIpZmWrX80qpa/t/kwdyutXylvI7euasA68lUDy7O5pON+xieFMVb0yYypKfW8pXyRpr4FUsy9/GAo5Z/9/mDue1sreUr5c008fsw51r+iKRo3p52Kqf0jPR0WEqpdqaJ38sYYyitqqOwrNp6lVdzyPGzfprj8+Hyavz9hN9dcAqzzuqvtXylfIQmfi9Qa7Pz56VbWLj6KGVffkpNnf0nywT6C3ERwcRHBtMrOoSRydHERwZzyahEBidoLV8pX6KJv4s7XF7Nr95ayw+7ihjbw5/xQ/oQH2kl+HhHoo+PDCY6NLBT3WmrlPIcTfxdWHZBCbPmraGwvJq/Tx9FbEkO6elDPR2WUqqT00bdLuqTjfu46h/fY7MbFtx+GleMSW5+JaWUQmv8XY7dbvj7l9t4/qscxvaO4Z8z0+gRGeLpsJRSXYgm/i6krKqW/313PV9uPsj0cSk8evkwggP8PR2WUqqL0cTfRew6VMGt81az61AFj1w6jOtP66MXa5VSLtHE3wWs2FbI7LfX4u8n/PuXEzh9gD7XVinlOk38nZgxhn99u4s/LdnM4IRIXrl+HCndwjwdllKqi9PE30lV1dq4b2EmH6zN58LhPXl62ijCg/VwKaXaTjNJJ1RUUcNNr/3IhrwSfjt5MLP1YeZKKTfSxN8JPfHJZjbvK+PlmWlcMKynp8NRSnkZvYGrk1mfW8z7a/O4eVI/TfpKqXahib8TsdsNDy/OJj4ymNnnDvR0OEopL9Wiph4RmQI8i/XM3VeNMU81mN8H6wHr8UAR1rN18xzz/gJchHWS+QK4yxhj3FYCL/LhhnzW5xbz9LRRROiFXKUaV1UCuasgdyXsXcmEAzlQdDb0OxP6ngmxfTwdYafXbHYREX/gRWAykAesEpHFxphNTos9DcwzxrwhIucCTwIzReR04AxgpGO5b4GzgQz3FcE7VFTX8dTSLYxKjubKMUmeDkepzqMkD/auhL3fw94f4EAWYED8oddIKsJ7E5bzJWycby0f0xv6ngX9zrJOBlGJrd+nMVBaAEU74HAOFO0C/0AIiYaQGMfPaAiNOT4tOAr8m0ipddVQWWyduOpfxY5XiTUvogecfqdLv6bWaEm1cgKQY4zZCSAi84HLAOfEnwr81vF+ObDI8d4AIUAQIEAgcKDtYXuff2Ts4EBpNS9dl6Y9eJTvstvg4CZHone8SvOseUERkDwe0u+F3qdCUhoER5CdkUH62WdD4RbYtcJ6bf0E1r9prddtwPFvA33PhMiE4/s7WgSHHcm9/rXDSvi1R48v5x9kxWZsTccfFHn8pBAYCtVlxxN7XVXT6waEQO/TOiTxS3OtLiJyFTDFGHOL4/NMYKIxZrbTMm8DPxhjnhWRK4H3gThjzGEReRq4BSvxv2CM+WMj+5gFzAJISEhImz9/vssFKi8vJyIiwuX1PaHwqJ17v61kfE9/bhvp+oBrXbHs7uLLZYcuUn5jI6immODqIoKrDxNcfZigmsOO98emHcLfXg1AdVA3SqJTKYkeSkl0KhXhfTB+Px2bqtGyGzsR5buJKc50vLIJsFmJvCIsmbqAcMKOFhBYV3Z8FfyoDE2gMjSRo2GJVIYmOX4mUh3cHRD8bZUE1FU4vcob/RxYW4GfvZq6gDDqAiKoCwh3vJzfn/jZ7h/k0q/1WPnPOeecNcaYcS1Zx10NyXcDL4jIjcAKIB+wichAYChwbMzgL0TkTGPMN84rG2PmAHMAxo0bZ9LT010OJCMjg7as7wn/8+YaAv1r+PsN6fSMdj3xd8Wyu4svlx06Ufmry6Fwq1VrL9wCR3ZD2T4o3QflB35aY/YLgMhe1itxHEQlQeIYSJlIcExveojQo5ldtqjsdhvs2wC7vyF897dQWwmDTofuAx2vAUhMH8ICgggDurv+G+hwrhz7liT+fCDF6XOyY1o9Y0wBcCWAiEQAU40xxSJyK7DSGFPumLcUOA04IfH7su92HGJp1n7uPn9wm5K+Uh2qttJK8IVbrCR/cAsc3Awle48vExACMX2sNvYBQ6zkHtULIhOtaVGJEBYHfh3QudDPH5LGWq8z7mr//XVyLUn8q4BBItIPK+HPAK51XkBE4oAiY4wduBerhw/AXuBWEXkSq6nnbOD/3BR7l1dns/PoR5tIjg3lljP7ezocpX7q2EXO/DVWjfngZijcbF3sxNFM7BcIcYMhZTykXQ/xQ6HHUIjtayVc1ek0m/iNMXUiMhv4DKs751xjTLaIPAqsNsYsBtKBJ0XEYDX13OFYfQFwLpCJ9VfyqTHmI/cXo2uavyqXLfvL+Md1YwkJ1H8Qr1FXAyW5ULzXauYIDG3QEyS2+R4gnlJVAgXrrESfvxbyVkP5fmue+FvNIj1HwMjpED/ESvDd+ls9XlSX0aK/PGPMEmBJg2kPOr1fgJXkG65nA25rY4xeqeRoLc98vpVT+3djynC9Q7dLsdVBWQEc2WMl9+I9J74vLaC+NtyUoIjGuwZG9ICEEdBrpJVo26vWXFdjdY08luTz18Chbcdj7z4Q+p9t9Z5JGgc9h0NAcPvEojpUJ6xy+Ia/f7mNkspaHrx4mHc/UOVoERTtPJ7cQqK7ZvI4sAlW/MVKkKX5YK9zmilWe3VMn+M3EMX0Pt6+XVd1vN92o/24He9LcmF/llXDttVYmw4IhYRh1kmg50jrZ49hENjC60HO3z6KHSenI3usY3IgG2xWDxrC463kPmLa8bbw0Fi3/gpV56GJ3wO2Hyjj3yv3cM2E3qQmRnk6nPZRsB5+nAOZC44nl2MCQk88EYTGNPgcC0MuspoQPO3Iblj+JGx8F4IjYdBkiJnqSO6OBB+dAgGudcVrlK3Wqnnv2wj7N1o/M9+H1Y5LZ+Jvtak7nQxijmTCuvwTk3tj3z7EH6KTrfgnznLU5tOsMnhzBUSdQBN/BzPG8OjHmwgP8ue3kwe7Z6N2G3x6D8N3rIPxwyHcQ0/oqquBzYvhh5ch70cIDIMx18HAyVBTcWLt1vl9RSEc2n78s7FBxp/hin/C0Is9U5byg7Dir7D6Naup5fQ7YdL/Qli39t+3f6BVy08YBlxjTTPGOgntzzx+Mti1wjohAaMBNoD17SPJOiH1O+v4N4+Y3layj0zsnNcWVIfSv4AOtmzzQb7ZfogHL06le4QbmjzsNvjwDtjwDt3EH+akw/Q3IXF027fdUqX7YM3rsOY1q692t/5wwZMw+lqrNt8axxLcgpvh3etg0m/h3Ps7rndIVQn89zlY+Q+riWbsTDj7D67d9u9OItCtn/VKvfT49PJC2L+RDRszGXX2Je7/9qG8kib+DlRdZ+PxTzYxID6cmae5YSApuw0W/coao+ScP7K2tBvjtv8d5l4Alz4PI69u+z5OxhjI/cFqztn0odXmPXAyTLwNBpznet/sYwnupqWw9Pfw7d+sXiZT/wXh7XhbTW2lVZZv/w6VR2DYlXDOHyGuk4+SGhEPA8/jSJ4/dB/g6WhUF6GJvwO9/t/d7D58lDdunkCgfxtvWjkh6d8PZ/+O8owMmJUB/7kBPrjV6nf9s0fc+9W+ttJqt/9xjtXkEBwNE26D8b90b+IJDIFLn4PkcfDJ3TDnbLh6nnXR0Z1sddaYLhl/tnrqDPwZnPtAx35jUqqDaeLvIAfLqnj+qxzOG9KDswfHt21jdhss+h+rfffc++Gs3x2fFxEP138In94L379gdde76rW2t01XHoEf5sAP/4TKIusmnYv/DiOuhuB2HCNm7PWQMBzeux7mToGLnramtZXdDpsWwVePWwNyJU+Aqa9A30lt37ZSnZwm/g7y9Gdbqa6zcf/FqW3bkN0GC2+HzPesmulZd/90Gf9AK0H2Ggmf/D+r3X/G21Y/7NaqOAwrX4QfX4HqUhh8IZx2h5UgO6oXSNJYmPU1vH8zLL7Tuqnowr+0vEujs0M51glz47tWr5ceqTDjHTjlQu3VonyGJv4OsDGvmP+syWPWmf3pFxfu+oZsdbDodsj8D5z3IJz5/5pefqzj9vl3fwH/mgyX/wOGXd6yfZUdgO+fh1VzreFpUy+1vln0HOF6/G0R3h1+8YFVQ//2b1Yz09X/hpiU5tctL4TsD6xkn78GxA/6nW39DoddocMKKJ+jib+dVdXaeGhxNt3Dg9r2OEVbHSy8DbIWwHkPwZm/bX4dsMZPue1reHem1fa///9ZFy1PluxK8uG/z8LaN6ybiIZfZZ1gegxxPXZ38fOHnz1k9TtfeLvV7n/VXOif/tNla47C1iVWss9ZZnUR7TkCzn/cKlNUr46OXqlOQxN/O8otOsrtb64hu6CUZ2eMJjLExfFMbHWwcBZkvQ8/e9jqT94akT3hxo9hyd3wzTNWX/ArXzmxq+WR3VaPlnVvAQZGzbC6UnbGniJDL4b4DKu757+vsJq8zBirGWzXCtj4nnU/QU251af99DutsWUS2tjMppSX0MTfTlZsK+TX89dhsxv+dcM4zhua0PxKjTkh6T8Ck37j2nYCguGS56DXaKub5CvnwjXvWOOhf/MMbJhv1ajHzoQzftP5n1saNxBuWQaLZ8OyRxgdnQprj1iDogVHWU1aI6dDn0kdM+yvUl2IJn43M8bwUsYOnv58K4N7RPLyzDT6utqub6uDD26B7IUw+dG2jyMuYnW77DHU6iXz8tnWcAr+QTBhllUzju5Cz/sNjrB6LCWPJ+yrP0O/02HKkzB4ijUiplKqUZr43aisqpa7/7OBz7IPcMmoRP48dQRhQS7+im218P4tVpfDyY/BGb92X6B9Trf6+39yN8QNshJ+RHPPOeqkROC0O/iueljneAKVUl2AJn43yTlYxqx/r2HP4aM8cHEqN5/R1/VRN52T/vmPt8/Dl6OT4VrXn22slOq6NPG7wdLMfdz9nw2EBvnz1i0TObV/G4YWOFoE/7kRdn0N5z8Bp89udhWllGoNTfxtUGez8/Tn2/jn1zsYnRLDP34xll7RbWhbLtwK78yAkjy47CVrZEullHIzTfwuKqqo4dfvrOPbnENcO7E3D12SSnBAG24E2v6FNSJlQDDc8DH0nui+YJVSykmL+rmJyBQR2SoiOSJyTyPz+4jIMhHZKCIZIpLsmH6OiKx3elWJSAtvHe28MvNKuOT5b/lxdxF/mTqSP10xwvWkbwx89wK8fbXVhfLW5Zr0lVLtqtkav4j4Ay8Ck4E8YJWILDbGbHJa7GlgnjHmDRE5F3gSmGmMWY7jGREi0g3IAT53cxk61NLMfdz17nriI4JZcPtpjExu5Xjzzuqq4eP/hfVvwdBLrQePBLVhSAellGqBljT1TAByjDE7AURkPnAZ4Jz4U4FjYwgsBxY1sp2rgKXGmKOuh+tZdrv19KzBCRG8cdOEtj1IpfygNYZO7g/Wgz7OvkdvNFJKdYiWJP4kINfpcx7QsC1iA3Al8CxwBRApIt2NMYedlpkB/K2xHYjILGAWQEJCAhkZGS0KvjHl5eVtWr8pmw/b2FdSxWV9DJmrv3d5O+HluxiR+QSBtSVsSf09hXI6rFjR5vjas+ydnS+XHXy7/L5cdnCx/MaYJl9YNfVXnT7PBF5osEwi8AGwDiv55wExTvN7AYVAYHP7S0tLM22xfPnyNq3flD8s2GBSH1hqjlbXub6RTYuNebynMc8MNSZ/nfuCM+1b9s7Ol8tujG+X35fLbszx8gOrTTP59dirJTX+fMB57NtkxzTnk0cBVo0fEYkAphpjip0WuRpYaIypbc1JqTOpqrXxSeY+Lhjek9AgFy7kGmM9vHv5E5A8Hqa/BZEujt+jlFJt0JJG5VXAIBHpJyJBWE02i50XEJE4ETm2rXuBuQ22cQ3wTluD9aTlWw5SVlXHFWNcGMum5qjVVXP5EzByhtVdU5O+UspDmk38xpg6YDbwGbAZeM8Yky0ij4rIpY7F0oGtIrINSACeOLa+iPTF+sbwtVsj72AL1+UTHxnM6QPiWr6SMbAzA+aebw209rNHrJ47rjw5Siml3KRFN3AZY5YASxpMe9Dp/QJgwUnW3Y11gbjLKj5aQ8bWQmae1gd/vxaOv7P7v1YNf89/ITIRrpkPp0xp30CVUqoF9M7dFliSuZ8am71lzTx7f7AS/q6vISLBejbs2Bu0lq+U6jQ08bfAonX5DOwRwbDEqJMvlLcalv8JdiyD8Hi44E8w7mYdF14p1elo4m9GbtFRftxdxO8uOKXxYZYL1sHyJ2H7ZxDazXpgyvhb9A5cpVSnpYm/GYs3FABw6ajEE2fsz4SMp2DLxxASA+c9aD3FKjjSA1EqpVTLaeJvgjGGhevyGd83lpRuYdbEwztg2SOw6UMIjoZz/ggTb4OQaM8Gq5RSLaSJvwnZBaXkHCzniSuGWxNqq+DNK6HiMJz1ezjtVxAa69kglVKqlTTxN2HRunwC/YWLRvSyJnz/PBzZDdd/CP3TPRiZUkq5ToeDPAmb3fDhhgLOOaUHMWFB1lOxvvmbNXxy/3RPh6eUUi7TxH8S3+04RGFZNZcf67v/+f3WnbgXPNH0ikop1clp4j+JResKiAwO4NwhPWDXCmvIhUn/CzG9PR2aUkq1iSb+RlTW2Pg0ax8/H9GLED8DS/9gJfwzfu3p0JRSqs304m4jvth8gIoam9XMs+pVOLjJGkZZ78JVSvsZYc0AABmuSURBVHkBrfE3YtG6fHpFhzCxh80ahmHAuTDkIk+HpZRSbqGJv4HD5dV8va2QS0cn4vfVI1BbAVP+DI0N16CUUl2QJv4GPsnch81uuCbxIKx7E079H4gf7OmwlFLKbTTxN7BwXT5DE8Lp+8PD1rDKZ/3e0yEppZRb6cVdJ7sPVbBubzGvj94CW9bCFS9DSBNDMSulVBekNX4ni9bnEyUVnLnnRUiZCCOnezokpZRyuxYlfhGZIiJbRSRHRO5pZH4fEVkmIhtFJENEkp3m9RaRz0Vks4hscjyDt9MxxvDh+gL+3O0T/CuLrCdn6QVdpZQXajbxi4g/8CJwIZAKXCMiqQ0WexqYZ4wZCTwKPOk0bx7wV2PMUGACcNAdgbvbhrwSAg9v4YKjH0HajZA42tMhKaVUu2hJjX8CkGOM2WmMqQHmA5c1WCYV+Mrxfvmx+Y4TRIAx5gsAY0y5MeaoWyJ3s0Vr83gkcB4SHGk9VEUppbxUSy7uJgG5Tp/zgIkNltkAXAk8C1wBRIpId2AwUCwiHwD9gC+Be4wxNueVRWQWMAsgISGBjIyM1pfEoby8vNXr19kNZWu+4jS/bLal3EbBjxtd3r8nuVJ2b+HLZQffLr8vlx1cLL8xpskXcBXwqtPnmcALDZZJBD4A1mEl/zwgxrFuCdAf6yTzPvDLpvaXlpZm2mL58uWtXicjc5fJe7CfKfnbBGNsdW3avye5UnZv4ctlN8a3y+/LZTfmePmB1aaZfH7s1ZIafz6Q4vQ52THN+eRRgFXjR0QigKnGmGIRyQPWG2N2OuYtAk4F/tW601P7qlr+NElymNrL5oGfv6fDUUqpdtWSNv5VwCAR6SciQcAMYLHzAiISJyLHtnUvMNdp3RgRiXd8PhfY1Paw3adifw7ph+azPmYygf0neTocpZRqd80mfmNMHTAb+AzYDLxnjMkWkUdF5FLHYunAVhHZBiQATzjWtQF3A8tEJBMQ4BW3l6INShbdTR1+yORHPR2KUkp1iBbduWuMWQIsaTDtQaf3C4AFJ1n3C2BkG2JsP7tWkLh/Of8InMntqUM9HY1SSnUIn75zt2LLl9Qaf6rH3orozVpKKR/h02P1FO9aT5lJ5Odp/T0dilJKdRifrvFHHNnCNvowMD7C06EopVSH8d3EX3mE6NoDHAofiJ+fNvMopXyH7yb+A1av0qruelFXKeVbfDbx1xZkAhCQ2Dk7HCmlVHvx2Yu7FXvXYzOR9Ezs4+lQlFKqQ/lsjd8cyGaLvTf99cKuUsrH+Gbit9uIKNnGFtOb/vHhno5GKaU6lG8m/qJdBNqr2BcygLAgn23tUkr5KN9M/AeyAO3Ro5TyTT5Z3TUHsrAZP4J7auJXSvken0z81fmZ7DW96J3Q3dOhKKVUh/PNpp79WWwxvRmgPXqUUj7I9xJ/VQkhFXmOrpzao0cp5Xt8L/Ef3AzATv++9IwK8XAwSinV8Xwv8e+3hmo42m2oDs6mlPJJLUr8IjJFRLaKSI6I3NPI/D4iskxENopIhogkO82zich6x2txw3U73IFsSokguocO1aCU8k3N9uoREX/gRWAykAesEpHFxhjnh6Y/DcwzxrwhIucCTwIzHfMqjTGj3Ry3y+z7s9hkT2FAD72wq5TyTS2p8U8AcowxO40xNcB84LIGy6QCXzneL29kfudgt2MObmKzjtGjlPJhLUn8SUCu0+c8xzRnG4ArHe+vACJF5Fgn+RARWS0iK0Xk8jZF21bFu/GvrWCz6c0A7dGjlPJR7rqB627gBRG5EVgB5AM2x7w+xph8EekPfCUimcaYHc4ri8gsYBZAQkICGRkZLgdSXl5+0vXjCr9nOLDF3pu8TWsp3OZdF3ebKru38+Wyg2+X35fLDq6VvyWJPx9Icfqc7JhWzxhTgKPGLyIRwFRjTLFjXr7j504RyQDGADsarD8HmAMwbtw4k56e3qpCOMvIyOCk62esxI4fZZEDueBn57i8j86qybJ7OV8uO/h2+X257OBa+VvS1LMKGCQi/UQkCJgBnNA7R0TiROTYtu4F5jqmx4pI8LFlgDMA54vCHWt/JgX+iSTrUA1KKR/WbOI3xtQBs4HPgM3Ae8aYbBF5VEQudSyWDmwVkW1AAvCEY/pQYLWIbMC66PtUg95AHcocyCarLpn+cdq+r5TyXS1q4zfGLAGWNJj2oNP7BcCCRtb7DhjRxhjdo7ocObKLrLpx2pVTKeXTfOfO3YPWF40tpjf94zTxK6V8l+8kfsfDVzbr4GxKKR/nQ4k/myq/cI4EJejgbEopn+Y7D2I5kM2ewH70i4nQwdmUUj7NN2r8xsCBbLLqUvThK0opn+cbNf7ivVBdypraRG3fV0r5PN+o8R/IBtDB2ZRSCp9J/FkYhK0mRQdnU0r5PJ9J/CUhyRwlhH56165Sysf5SOLPZk9AP5JiQgkL8o3LGkopdTLen/hrjsLhHWTZUvTCrlJK4QuJ/+BmwLCyopcOzqaUUvhC4ncM1bC+NkkHZ1NKKXwi8WdjCwgnz8Tr4GxKKYVPJP4sDkcMxODHgB7a1KOUUt6d+I2BA1nsCehPWJC/Ds6mlFJ4e+IvzYeqErJtKfSLC0dEB2dTSinvTvz7rQu7P1T01MHZlFLKoUWJX0SmiMhWEckRkXsamd9HRJaJyEYRyRCR5Abzo0QkT0RecFfgLeLo0fNtWQ/tw6+UUg7NJn4R8QdeBC4EUoFrRCS1wWJPA/OMMSOBR4EnG8x/DFjR9nBb6UA2NZG9KTNhWuNXSimHltT4JwA5xpidxpgaYD5wWYNlUoGvHO+XO88XkTQgAfi87eG20oFsiiIGAWiNXymlHFoycE0SkOv0OQ+Y2GCZDcCVwLPAFUCkiHQHjgDPAL8AfnayHYjILGAWQEJCAhkZGS0M/6fKy8vJyMjAz1bNmYe2syFyuBX0prUUbvPui7vHyu6LfLns4Nvl9+Wyg2vld9eIZXcDL4jIjVhNOvmADfgVsMQYk9dUjxpjzBxgDsC4ceNMenq6y4FkZGSQnp4OBevgGzv5kSNJqg3lgp+d4/I2u4r6svsgXy47+Hb5fbns4Fr5W5L484EUp8/Jjmn1jDEFWDV+RCQCmGqMKRaR04AzReRXQAQQJCLlxpifXCB2O8fDV1Ye7aXNPEop5aQliX8VMEhE+mEl/BnAtc4LiEgcUGSMsQP3AnMBjDHXOS1zIzCuQ5I+wP4sTGAY3xdFMrW/XthVSqljmr24a4ypA2YDnwGbgfeMMdki8qiIXOpYLB3YKiLbsC7kPtFO8bbcgSxquw+hrMZojV8ppZy0qI3fGLMEWNJg2oNO7xcAC5rZxuvA662O0BXGWD16kiYD6OBsSinlxDvv3C3bD5VF7A3sD6CDsymllBPvTPyOO3az65J1cDallGrAOx9A60j8P1Qm0j8+UAdnU0opJ16a+LMhOoXMQ5DWR9v3lWqJ2tpa8vLyqKqq8nQorRIdHc3mzZs9HUaHCQkJITk5mcDAQJe34Z2Jf38WtvhUCrIruTo+pfnllVLk5eURGRlJ3759u9S35LKyMiIjIz0dRocwxnD48GHy8vLo16+fy9vxujZ+sdfCoW0URQzCGB2jR6mWqqqqonv37l0q6fsaEaF79+5t/lbmdYk/vCIXjI29QVaPHk38SrWcJv3Ozx3HyAsT/24ANtl6A9qHXymlGvK6xB9RvhsCQlhbHkNSTCihQf6eDkkp1QLFxcW89NJLLq3785//nOLiYjdH5L28LvGHV+yGHkPJOVStzTxKdSFNJf66urom112yZAkxMTHtEVabGGOw2+2eDuMnvK5XT0T5bkyfS9i5tpxp47RHj1KueOSjbDYVlLp1m6mJUTx0ybCTzr/nnnvYsWMHo0ePZvLkyVx00UU88MADxMbGsmXLFrZt28bll19Obm4uVVVV3HXXXcyaNQuAvn37snr1asrLy7nwwguZNGkS3333HUlJSXz44YeEhoaesK+PPvqIxx9/nJqaGrp3785bb71FQkIC5eXl3HnnnaxevRoR4aGHHmLq1Kl8+umn3HfffdhsNuLi4li2bBkPP/wwERER3H333QAMHz6cjz/+GIALLriAiRMnsmbNGpYsWcJTTz3FqlWrqKys5KqrruKRRx4BYNWqVdx1111UVFQQHBzMsmXLuOiii3juuecYPXo0AJMmTeLFF19k1KhRbjsW3pX4yw4QVFtCafQpVNTYtMavVBfy1FNPkZWVxfr16wFrnPm1a9eSlZVV33Vx7ty5dOvWjcrKSsaPH8/UqVMJCgo6YTvbt2/nnXfe4ZVXXuHqq6/m/fff5xe/+MUJy0yaNImVK1ciIrz66qv85S9/4ZlnnuGxxx4jOjqazMxMAI4cOUJhYSG33norK1asoF+/fhQVFTVblu3bt/PGG29w6qmnAvDEE0/QrVs3bDYb5513Hhs3bmTIkCFMnz6dd999l/Hjx1NaWkpoaCi//OUvef311/m///s/tm3bRlVVlVuTPnhb4nfcsbs30Poj0efsKuWapmrmHWnChAkn9Fd/7rnnWLhwIQC5ubls376dYcNOjLVfv371teW0tDR27979k+3m5eUxffp09u3bR01NTf0+vvzyS+bPn1+/XGxsLB999BFnnXVW/TLdunVrNu4+ffrUJ32A9957jzlz5lBXV8e+ffvYtGkTIkKvXr0YP348AFFRUQBMmzaNxx57jL/+9a/MnTuXG2+8sdn9tZZ3tfE7Hr6SbbOaeLTGr1TXFh5+/H84IyODL7/8ku+//54NGzYwZsyYRvuzBwcH17/39/dv9PrAnXfeyezZs8nMzOTll192qV98QEDACe33zttwjnvXrl08/fTTLFu2jI0bN3LRRRc1ub+wsDAmT57Mhx9+yHvvvcd111130mVd5WWJP4vqoO5sLgnUwdmU6mIiIyMpKys76fySkhJiY2MJCwtjy5YtrFy50uV9lZSUkJSUBMAbb7xRP33y5Mm8+OKL9Z+PHDnCqaeeyooVK9i1axdAfVNP3759Wbt2LQBr166tn99QaWkp4eHhREdHc+DAAZYuXQrAKaecwr59+1i1ahVg3YF87CR1yy238Otf/5rx48cTGxvrcjlPxssSfzblEX3ZUVhO//hwvRlFqS6ke/funHHGGQwfPpzf/e53P5k/ZcoU6urqGDp0KPfcc88JTSmt9fDDDzNt2jTS0tKIi4urn37//fdz5MgRhg8fzqhRo1i+fDnx8fHMmTOHK6+8klGjRjF9+nQApk6dSlFREcOGDeOFF15g8ODBje5r1KhRjBkzhiFDhnDttddyxhlnABAUFMS7777LnXfeyahRo5g8eXL9N4G0tDSioqK46aabXC5jU8QY0y4bdtW4cePM6tWrW79iXQ38KZE9yZdy7cFfkNYnlueuGeP+ADsxX37otC+XHdxT/s2bNzN06FD3BNSBvHGsnoKCAtLT09myZQt+fj+tnzsfq2PHXkTWGGPGtWT73lPjP3oYeo3iSPgACkoq9cKuUqpLmjdvHhMnTuSJJ55oNOm7Q4u2KiJTRGSriOSIyE8eli4ifURkmYhsFJEMEUl2mr5WRNaLSLaI3O7uAtSL6gW3LmNj6Kk6OJtSqsu6/vrryc3NZdq0ae22j2YTv4j4Ay8CFwKpwDUiktpgsaeBecaYkcCjwJOO6fuA04wxo4GJwD0ikuiu4Buzv8JqutIav1JKNa4lNf4JQI4xZqcxpgaYD1zWYJlU4CvH++XH5htjaowx1Y7pwS3cX5vsq7C6V/WL0xq/Uko1piU3cCUBuU6f87Bq7842AFcCzwJXAJEi0t0Yc1hEUoBPgIHA74wxBQ13ICKzgFkACQkJZGRktLYc9XJLauge4scP333j8ja6qvLy8jb97royXy47uKf80dHRTXan7KxsNluXjLstqqqq6o+3K8feXXfu3g28ICI3AiuAfMAGYIzJBUY6mngWicgCY8wB55WNMXOAOWD16mlL74SHv1tKako30tMbnpu8ny/3bPHlsoP7evV0xd4x3tirpzkhISGMGWP1WnTl2Lek6SUfcB7tLNkxrZ4xpsAYc6UxZgzwR8e04obLAFnAma2KsBWMMeyvsGv7vlI+IiJC/9dd0ZLEvwoYJCL9RCQImAEsdl5AROJE5Ni27gXmOqYni0io430sMAnY6q7gGzpQWk2VDQZojx6lVAdobrjozqrZph5jTJ2IzAY+A/yBucaYbBF5FFhtjFkMpANPiojBauq5w7H6UOAZx3QBnjbGZLZDOQDYUVgOQH+t8SvVNkvvgf1u/lftOQIufOqks++55x5SUlK44w4rfRwb9vj222/nsssu48iRI9TW1vL4449z2WUN+5ec6GTDNzc2vPLJhmKOiIigvNzKKQsWLODjjz/m9ddf58YbbyQkJIR169ZxxhlnMGPGDO666y6qqqoIDQ3ltdde45RTTsFms/GHP/yBTz/9FD8/P2699VaGDRvGc889x6JFiwD44osveOmll+oHnusoLWrjN8YsAZY0mPag0/sFwIJG1vsCGNnGGFtsZ33i1xq/Ul3N9OnT+c1vflOf+N977z0+++wzQkJCWLhwIVFRURw6dIhTTz2VSy+9tMkhWRobvtlutzc6vHJjQzE3Jy8vj++++w5/f39KS0v55ptvCAgI4Msvv+S+++7j/fffZ86cOezevZv169cTEBBAUVERsbGx/OpXv6KwsJD4+Hhee+01br75Zjf89lrHq4Zl3lFYQbA/OjibUm3VRM28vYwZM4aDBw9SUFBAYWEhsbGxpKSkUFtby3333ceKFSvw8/MjPz+fAwcO0LNnz5Nuq7HhmwsLCxsdXrmxoZibM23aNPz9rce6lpSUcMMNN7B9+3ZEhNra2vrt3n777QQEBJywv5kzZ/Lmm29y00038f333zNv3rzW/qrazMsSfzk9w/10cDaluqhp06axYMEC9u/fXz8Y2ltvvUVhYSFr1qwhMDCQvn37NjmssfPwzWFhYaSnp7s07LJzHmm4vvOwyw888ADnnHMOCxcuZPfu3c32sLnpppu45JJLCAkJYdq0afUnho7kPWP1ADsLK+gVrklfqa5q+vTpzJ8/nwULFtQPWVBSUkKPHj0IDAxk+fLl7Nmzp8ltnGz45pMNr9zYUMxg3VO0efNm7HZ7k23wzkM8v/766/XTJ0+ezMsvv1x/AfjY/hITE0lMTOTxxx9vt9E3m+M1ib+yxkZ+cSW9wr2mSEr5nGHDhlFWVkZSUhK9evUC4LrrrmP16tWMGDGCefPmMWTIkCa3cbLhm082vHJjQzGD9SjIiy++mNNPP70+lsb8/ve/595772XMmDEn9PK55ZZb6N27NyNHjmTUqFG8/fbb9fOuu+46UlJSPDcaqjGmU73S0tKMKwrLqsydb681z//nS5fW9wbLly/3dAge48tlN8Y95d+0aVPbA/GA0tJST4fQanfccYd59dVXXV7f+VgdO/ZYvSxblGe9pnocFxHMc9eMYXicv6dDUUqpk0pLS2Pjxo0/eQB8R/Kqi7tKKdXZrVmzxtMheE+NXynVdqaTPZFP/ZQ7jpEmfqUUYA38dfjwYU3+nZgxhsOHDxMS0rZ7lbSpRykFQHJyMnl5eRQWFno6lFapqqpqcyLsSkJCQkhOTm7TNjTxK6UACAwMrL+rtSvJyMioH6JYtYw29SillI/RxK+UUj5GE79SSvkY6WxX8EWkEGh6MI6mxQGH3BROV6Nl912+XH5fLjscL38fY0x8S1bodIm/rURktTFmnKfj8AQtu2+WHXy7/L5cdnCt/NrUo5RSPkYTv1JK+RhvTPxzPB2AB2nZfZcvl9+Xyw4ulN/r2viVUko1zRtr/EoppZqgiV8ppXyM1yR+EZkiIltFJEdE7vF0PB1NRHaLSKaIrBeR1Z6Opz2JyFwROSgiWU7TuonIFyKy3fEz1pMxtqeTlP9hEcl3HP/1IvJzT8bYXkQkRUSWi8gmEckWkbsc073++DdR9lYfe69o4xcRf2AbMBnIA1YB1xhjNnk0sA4kIruBccYYr7+RRUTOAsqBecaY4Y5pfwGKjDFPOU78scaYP3gyzvZykvI/DJQbY572ZGztTUR6Ab2MMWtFJBJYA1wO3IiXH/8myn41rTz23lLjnwDkGGN2GmNqgPnAZR6OSbUTY8wKoKjB5MuANxzv38D6h/BKJym/TzDG7DPGrHW8LwM2A0n4wPFvouyt5i2JPwnIdfqch4u/kC7MAJ+LyBoRmeXpYDwgwRizz/F+P5DgyWA8ZLaIbHQ0BXldU0dDItIXGAP8gI8d/wZlh1Yee29J/AomGWPGAhcCdziaA3ySsdovu34bZuv8AxgAjAb2Ac94Npz2JSIRwPvAb4wxpc7zvP34N1L2Vh97b0n8+UCK0+dkxzSfYYzJd/w8CCzEav7yJQccbaDH2kIPejieDmWMOWCMsRlj7MArePHxF5FArMT3ljHmA8dknzj+jZXdlWPvLYl/FTBIRPqJSBAwA1js4Zg6jIiEOy72ICLhwPlAVtNreZ3FwA2O9zcAH3owlg53LOk5XIGXHn8REeBfwGZjzN+cZnn98T9Z2V059l7RqwfA0YXp/wB/YK4x5gkPh9RhRKQ/Vi0frMdpvu3N5ReRd4B0rOFoDwAPAYuA94DeWMN6X22M8coLoCcpfzrWV30D7AZuc2rz9hoiMgn4BsgE7I7J92G1dXv18W+i7NfQymPvNYlfKaVUy3hLU49SSqkW0sSvlFI+RhO/Ukr5GE38SinlYzTxK6WUj9HEr5RSPkYTv1JK+Zj/D6taLKHaMK8QAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "for epoch in range(25):\n",
        "\n",
        "    for x_batch,y_batch in iterate_minibatches(X_train, y_train, batchsize=32, shuffle=True):\n",
        "        train(network, x_batch, y_batch)\n",
        "    \n",
        "    train_log.append(np.mean(predict(network, X_train) == y_train))\n",
        "    val_log.append(np.mean(predict(network, X_val) == y_val))\n",
        "    \n",
        "    clear_output()\n",
        "    print(\"Epoch\",epoch)\n",
        "    print(\"Train accuracy:\",train_log[-1])\n",
        "    print(\"Val accuracy:\",val_log[-1])\n",
        "    plt.plot(train_log,label='train accuracy')\n",
        "    plt.plot(val_log,label='val accuracy')\n",
        "    plt.legend(loc='best')\n",
        "    plt.grid()\n",
        "    plt.show()\n",
        "    "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JZJ2QfqGii0t"
      },
      "source": [
        "__What should you see:__ `train accuracy` should increase to near-100%. Val accuracy will also increase, allbeit to a smaller value.\n",
        "\n",
        "__What else to try:__ You can try implementing different nonlinearities, dropout or composing neural network of more layers. See how this affects training speed, overfitting & final quality.\n",
        "\n",
        "Good hunting!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "Dt1-Yn04ii0t"
      },
      "outputs": [],
      "source": [
        "# and yes, it's perfectly legal to reuse your code from this seminar in homework01."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.2"
    },
    "toc": {
      "colors": {
        "hover_highlight": "#DAA520",
        "navigate_num": "#000000",
        "navigate_text": "#333333",
        "running_highlight": "#FF0000",
        "selected_highlight": "#FFD700",
        "sidebar_border": "#EEEEEE",
        "wrapper_background": "#FFFFFF"
      },
      "moveMenuLeft": true,
      "nav_menu": {
        "height": "264px",
        "width": "252px"
      },
      "navigate_menu": true,
      "number_sections": true,
      "sideBar": true,
      "threshold": 4,
      "toc_cell": false,
      "toc_section_display": "block",
      "toc_window_display": false,
      "widenNotebook": false
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}