chapter5/5.3-Fashion-MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.3.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch,math\n",
    "from pathlib import Path\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt  \n",
    "import torchvision.datasets as dsets\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "\n",
    "import torch.nn.functional as F\n",
    "import torch.nn as NN\n",
    "torch.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fashion MNIST进行分类 \n",
    "## Fashion MNIST 介绍\n",
    "Fashion MNIST数据集 是kaggle上提供的一个图像分类入门级的数据集，其中包含10个类别的70000个灰度图像。如图所示，这些图片显示的是每件衣服的低分辨率(28×28像素)\n",
    "\n",
    "数据集的下载和介绍：[地址](https://www.kaggle.com/zalando-research/fashionmnist/)\n",
    "\n",
    "\n",
    "Fashion MNIST的目标是作为经典MNIST数据的替换——通常被用作计算机视觉机器学习程序的“Hello, World”。\n",
    "\n",
    "MNIST数据集包含手写数字(0-9等)的图像，格式与我们将在这里使用的衣服相同，MNIST只有手写的0-1数据的复杂度不高，所以他只能用来做“Hello, World”\n",
    "\n",
    "而Fashion MNIST 的由于使用的是衣服的数据，比数字要复杂的多，并且图片的内容也会更加多样性，所以它是一个比常规MNIST稍微更具挑战性的问题。\n",
    "\n",
    "Fashion MNIST这个数据集相对较小，用于验证算法是否按预期工作。它们是测试和调试代码的好起点。\n",
    "\n",
    "## 数据集介绍\n",
    "\n",
    "### 分类\n",
    "```\n",
    "0 T-shirt/top\n",
    "1 Trouser\n",
    "2 Pullover\n",
    "3 Dress\n",
    "4 Coat\n",
    "5 Sandal\n",
    "6 Shirt\n",
    "7 Sneaker\n",
    "8 Bag\n",
    "9 Ankle boot \n",
    "```\n",
    "### 格式\n",
    "\n",
    "fashion-mnist_test.csv\n",
    "\n",
    "fashion-mnist_train.csv\n",
    "\n",
    "存储的训练的数据和测试的数据，格式如下：\n",
    "\n",
    "label是分类的标签\n",
    "pixel1-pixel784是每一个像素代表的值 因为是灰度图像，所以是一个0-255之间的数值。\n",
    "\n",
    "为什么是784个像素？ 28 * 28 = 784\n",
    "\n",
    "### 数据提交\n",
    "\n",
    "Fashion MNIST不需要我们进行数据的提交，数据集中已经帮助我们将 训练集和测试集分好了，我们只需要载入、训练、查看即可，所以Fashion MNIST 是一个非常好的入门级别的数据集\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#指定数据目录\n",
    "DATA_PATH=Path('./data/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>pixel1</th>\n",
       "      <th>pixel2</th>\n",
       "      <th>pixel3</th>\n",
       "      <th>pixel4</th>\n",
       "      <th>pixel5</th>\n",
       "      <th>pixel6</th>\n",
       "      <th>pixel7</th>\n",
       "      <th>pixel8</th>\n",
       "      <th>pixel9</th>\n",
       "      <th>...</th>\n",
       "      <th>pixel775</th>\n",
       "      <th>pixel776</th>\n",
       "      <th>pixel777</th>\n",
       "      <th>pixel778</th>\n",
       "      <th>pixel779</th>\n",
       "      <th>pixel780</th>\n",
       "      <th>pixel781</th>\n",
       "      <th>pixel782</th>\n",
       "      <th>pixel783</th>\n",
       "      <th>pixel784</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>43</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>203</td>\n",
       "      <td>214</td>\n",
       "      <td>166</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 785 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   label  pixel1  pixel2  pixel3  pixel4  pixel5  pixel6  pixel7  pixel8  \\\n",
       "0      2       0       0       0       0       0       0       0       0   \n",
       "1      9       0       0       0       0       0       0       0       0   \n",
       "2      6       0       0       0       0       0       0       0       5   \n",
       "3      0       0       0       0       1       2       0       0       0   \n",
       "4      3       0       0       0       0       0       0       0       0   \n",
       "5      4       0       0       0       5       4       5       5       3   \n",
       "6      4       0       0       0       0       0       0       0       0   \n",
       "7      5       0       0       0       0       0       0       0       0   \n",
       "8      4       0       0       0       0       0       0       3       2   \n",
       "9      8       0       0       0       0       0       0       0       0   \n",
       "\n",
       "   pixel9  ...  pixel775  pixel776  pixel777  pixel778  pixel779  pixel780  \\\n",
       "0       0  ...         0         0         0         0         0         0   \n",
       "1       0  ...         0         0         0         0         0         0   \n",
       "2       0  ...         0         0         0        30        43         0   \n",
       "3       0  ...         3         0         0         0         0         1   \n",
       "4       0  ...         0         0         0         0         0         0   \n",
       "5       5  ...         7         8         7         4         3         7   \n",
       "6       0  ...        14         0         0         0         0         0   \n",
       "7       0  ...         0         0         0         0         0         0   \n",
       "8       0  ...         1         0         0         0         0         0   \n",
       "9       0  ...       203       214       166         0         0         0   \n",
       "\n",
       "   pixel781  pixel782  pixel783  pixel784  \n",
       "0         0         0         0         0  \n",
       "1         0         0         0         0  \n",
       "2         0         0         0         0  \n",
       "3         0         0         0         0  \n",
       "4         0         0         0         0  \n",
       "5         5         0         0         0  \n",
       "6         0         0         0         0  \n",
       "7         0         0         0         0  \n",
       "8         0         0         0         0  \n",
       "9         0         0         0         0  \n",
       "\n",
       "[10 rows x 785 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = pd.read_csv(DATA_PATH / \"fashion-mnist_train.csv\");\n",
    "train.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>pixel1</th>\n",
       "      <th>pixel2</th>\n",
       "      <th>pixel3</th>\n",
       "      <th>pixel4</th>\n",
       "      <th>pixel5</th>\n",
       "      <th>pixel6</th>\n",
       "      <th>pixel7</th>\n",
       "      <th>pixel8</th>\n",
       "      <th>pixel9</th>\n",
       "      <th>...</th>\n",
       "      <th>pixel775</th>\n",
       "      <th>pixel776</th>\n",
       "      <th>pixel777</th>\n",
       "      <th>pixel778</th>\n",
       "      <th>pixel779</th>\n",
       "      <th>pixel780</th>\n",
       "      <th>pixel781</th>\n",
       "      <th>pixel782</th>\n",
       "      <th>pixel783</th>\n",
       "      <th>pixel784</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>103</td>\n",
       "      <td>87</td>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>53</td>\n",
       "      <td>99</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>63</td>\n",
       "      <td>53</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>137</td>\n",
       "      <td>126</td>\n",
       "      <td>140</td>\n",
       "      <td>0</td>\n",
       "      <td>133</td>\n",
       "      <td>224</td>\n",
       "      <td>222</td>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>44</td>\n",
       "      <td>105</td>\n",
       "      <td>44</td>\n",
       "      <td>10</td>\n",
       "      <td>...</td>\n",
       "      <td>105</td>\n",
       "      <td>64</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>174</td>\n",
       "      <td>136</td>\n",
       "      <td>155</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>57</td>\n",
       "      <td>70</td>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 785 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   label  pixel1  pixel2  pixel3  pixel4  pixel5  pixel6  pixel7  pixel8  \\\n",
       "0      0       0       0       0       0       0       0       0       9   \n",
       "1      1       0       0       0       0       0       0       0       0   \n",
       "2      2       0       0       0       0       0       0      14      53   \n",
       "3      2       0       0       0       0       0       0       0       0   \n",
       "4      3       0       0       0       0       0       0       0       0   \n",
       "5      2       0       0       0       0       0      44     105      44   \n",
       "6      8       0       0       0       0       0       0       0       0   \n",
       "7      6       0       0       0       0       0       0       0       1   \n",
       "8      5       0       0       0       0       0       0       0       0   \n",
       "9      0       0       0       0       0       0       0       0       0   \n",
       "\n",
       "   pixel9  ...  pixel775  pixel776  pixel777  pixel778  pixel779  pixel780  \\\n",
       "0       8  ...       103        87        56         0         0         0   \n",
       "1       0  ...        34         0         0         0         0         0   \n",
       "2      99  ...         0         0         0         0        63        53   \n",
       "3       0  ...       137       126       140         0       133       224   \n",
       "4       0  ...         0         0         0         0         0         0   \n",
       "5      10  ...       105        64        30         0         0         0   \n",
       "6       0  ...         0         0         0         0         0         0   \n",
       "7       0  ...       174       136       155        31         0         1   \n",
       "8       0  ...         0         0         0         0         0         0   \n",
       "9       0  ...        57        70        28         0         2         0   \n",
       "\n",
       "   pixel781  pixel782  pixel783  pixel784  \n",
       "0         0         0         0         0  \n",
       "1         0         0         0         0  \n",
       "2        31         0         0         0  \n",
       "3       222        56         0         0  \n",
       "4         0         0         0         0  \n",
       "5         0         0         0         0  \n",
       "6         0         0         0         0  \n",
       "7         0         0         0         0  \n",
       "8         0         0         0         0  \n",
       "9         0         0         0         0  \n",
       "\n",
       "[10 rows x 785 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = pd.read_csv(DATA_PATH / \"fashion-mnist_test.csv\");\n",
    "test.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "label         9\n",
       "pixel1       16\n",
       "pixel2       36\n",
       "pixel3      226\n",
       "pixel4      164\n",
       "           ... \n",
       "pixel780    255\n",
       "pixel781    255\n",
       "pixel782    255\n",
       "pixel783    255\n",
       "pixel784    170\n",
       "Length: 785, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ubyte文件标识了数据的格式\n",
    "\n",
    "其中idx3的数字表示数据维度。也就是图像为3维，\n",
    "idx1 标签维1维。\n",
    "\n",
    "具体格式详解：http://yann.lecun.com/exdb/mnist/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2051, 60000, 28, 28)\n"
     ]
    }
   ],
   "source": [
    "import struct\n",
    "from PIL import Image \n",
    "\n",
    "with open(DATA_PATH / \"train-images-idx3-ubyte\", 'rb') as file_object:\n",
    "    header_data=struct.unpack(\">4I\",file_object.read(16))\n",
    "    print(header_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2049, 60000)\n"
     ]
    }
   ],
   "source": [
    "with open(DATA_PATH / \"train-labels-idx1-ubyte\", 'rb') as file_object:\n",
    "    header_data=struct.unpack(\">2I\",file_object.read(8))\n",
    "    print(header_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(28, 28)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAASbElEQVR4nO3dbYyV5ZkH8P8FjA4wCIyADDABLBA0vgw6EiNo1GpD+QJVa8oHZaMujWmTNqlxjfuhJmYj0bXdmpAmUyXQpbVpUogYRUsICduglSNhB9zZXYFgGZjMCCjlTZiXaz/MYTPCPNd1OM85z3Pa6/9LJjNzrrnPuefM/OecM9dzP7eoKojo79+IvCdARNlg2ImCYNiJgmDYiYJg2ImCGJXljYlIqn/9i0hirdpdhVtvvTWx1tnZaY49fvx4pafzNXV1dYm13t7e3G47i9uny6nqsEGRNCERkSUAfgFgJIDXVXW18/WpEllfX59Y836p+vv709w0enp6EmvPPvusOXbdunWpbtszY8aMxJr3h6iat53F7dPlksJe9tN4ERkJYA2AbwO4EcAKEbmx3OsjoupK85p9IYD9qnpQVS8A+B2AZZWZFhFVWpqwTwdweMjnncXLvkZEVolIQUQKKW6LiFJK8w+64V4XXPaaXFXbALQB6V+zE1H50jyydwJoHvL5DABH002HiKolTdh3AZgrIrNF5CoA3wOwuTLTIqJKS9t6Wwrg3zDYelurqv/ifD2fxhNVWVX67FeKYSeqvor32YnobwvDThQEw04UBMNOFATDThQEw04UBMNOFATDThQEw04UBMNOFATDThQEw04UBMNOFATDThQEw04UBMNOFATDThQEw04UBMNOFATDThQEw04URKZbNgPV23Z53LhxZn3x4sVmfcuWLWXftvU9AcDIkSPNel9fX9m3nZY3d0+WZyemdPjIThQEw04UBMNOFATDThQEw04UBMNOFATDThRE5n32ESOS/7709/ebY+fMmZNYe+qpp8yx586dM+tnzpwx61999VVi7aOPPjLHpu2je71w6z71xqadm3cMgfczpeykCruIHAJwCkA/gD5Vba3EpIio8irxyH6fqh6rwPUQURXxNTtREGnDrgD+KCIfi8iq4b5ARFaJSEFECilvi4hSSPs0fpGqHhWRKQC2ish/q+qOoV+gqm0A2gBARLhqgignqR7ZVfVo8X0PgE0AFlZiUkRUeWWHXUTGisi4ix8D+BaAfZWaGBFVVpqn8dcB2FTs444C8FtVfc8aICJmX9bryd5///2JtQceeMAc29nZadavvvpqsz5mzJjE2oMPPmiOff311816d3e3WffWjKfpZTc0NJj1gYEBs3727Nmyb5uyVXbYVfUggFsrOBciqiK23oiCYNiJgmDYiYJg2ImCYNiJgsh0iauq4sKFC2WPv+OOOxJrs2bNMsd6SzGtZaIA8P777yfWFixYYI59+eWXzXqhYB9JvHfvXrPe0dGRWFu40D7OybpPAWDnzp1m/YMPPjDrJ0+eNOt05co9HTsf2YmCYNiJgmDYiYJg2ImCYNiJgmDYiYJg2ImCkCy33PXOVOMtFbX61RMmTDDH9vb2mnVvKadl165dZn3//v1mPc2xBwDQ1NSUWPO+b2/ujzzyiFlfs2aNWd++fbtZjyrtVtlJVBWqOuyV85GdKAiGnSgIhp0oCIadKAiGnSgIhp0oCIadKIia6rN/+OGH5nhvzbpz22bd27o4TS/c2u4Z8Hv8u3fvNutWH9/7vpYsWWLWr7/+erM+ffp0s07ZY5+dKDiGnSgIhp0oCIadKAiGnSgIhp0oCIadKIhMzxvv+eKLL8y6tW773Llz5lhvS+ZRo+y7wtra2Oujjx492qx7ffa7777brN91112JNe98+FOmTDHr771n7sJNf0PcR3YRWSsiPSKyb8hljSKyVUQ+Lb6fWN1pElFapTyNXwfg0sOsngOwTVXnAthW/JyIapgbdlXdAeDEJRcvA7C++PF6AMsrPC8iqrByX7Nfp6pdAKCqXSKS+MJPRFYBWFXm7RBRhVT9H3Sq2gagDfAXwhBR9ZTbeusWkSYAKL7vqdyUiKgayg37ZgArix+vBPBWZaZDRNXiPo0XkTcB3Atgkoh0AvgpgNUAfi8iTwL4C4DvVmIyY8aMMetWz9jrJ589e9ase/uIHz9+PLHmrbP3zhngrbX3vjfrfuvv7zfHej3+5uZms07lsX7m3s/b+5kmccOuqisSSt8s6xaJKBc8XJYoCIadKAiGnSgIhp0oCIadKIhMl7iOGDHCbBNNmzbNHH/+/PmyaoC/xNU7VbTVuvO2i7badoDfcrzqqqvM+qlTpxJr48ePN8e2t7ebdWtpLwC0traa9UKhYNajstqxI0eONMeW23rjIztREAw7URAMO1EQDDtREAw7URAMO1EQDDtREJn22RsbG7Fs2bLE+tSpU83xn3/+eWIt7emax44da9atpZ5ej97r8ff29pp17zTX1vd+7bXXmmPXrFlj1ltaWsy6NzcannW/pdke3MJHdqIgGHaiIBh2oiAYdqIgGHaiIBh2oiAYdqIgxDvNcSU1NDToTTfdlFh/5513zPHWtsxp1wCPGzfOrFvbMnvr1evq6lLVvWMAvK2uLd5206+88opZ37BhQ9m37fFOse39zK1TMnvX7R374B23kSdVHfab4yM7URAMO1EQDDtREAw7URAMO1EQDDtREAw7URCZLkaeM2cO3n333cT6mTNnzPFWn907t7p3PIG1Vh6we7r19fXmWK9n60nT8/V60bfccotZ97ayTiPtsRF9fX2VnE6m7rnnnsTaww8/bI5dtGhRYu2xxx5LrLmP7CKyVkR6RGTfkMteEJEjIrKn+LbUux4iylcpT+PXAVgyzOU/V9WW4lvywzUR1QQ37Kq6A8CJDOZCRFWU5h90PxSR9uLT/IlJXyQiq0SkICIF7xhyIqqecsP+SwDfANACoAvAq0lfqKptqtqqqq3eyQ+JqHrKCruqdqtqv6oOAPgVgIWVnRYRVVpZYReRpiGffgfAvqSvJaLa4PbZReRNAPcCmCQinQB+CuBeEWkBoAAOAfh+KTc2MDBg7nPuvaa3eune+cu9nq01L8DuCXvnhff68N4xAN7cLN7xB6dPnzbrDz30kFl/++23r3hOF5W7z3ipGhsbE2vTpk0zx86dO9ese+O9+23evHmJtfPnz5tjrXX61u+pG3ZVXTHMxW9444iotvBwWaIgGHaiIBh2oiAYdqIgGHaiIDJd4uq13rwj7I4ePZpY806J7LWgpkyZYtatbXTHjBljjt25c6dZb2hoMOvWckjAXuLqLVH1ls/eeeedZj0N77pffPFFsz558mSzPmHChMSa1/bzlt9++eWXZt1bfnvq1KnEmrdls3UabOt3gY/sREEw7ERBMOxEQTDsREEw7ERBMOxEQTDsREFk2mfv7+83txf2+tGPPvpoYq1QKJhjvS2ZvWWFs2bNMuuWm2++2ax7czt8+LBZt45dGD16tDnW6/HPnDnTrHusfvVrr71mjm1qajLrXq/cqqdZNgz4x214c7NOi+4ZP358Ys1a/spHdqIgGHaiIBh2oiAYdqIgGHaiIBh2oiAYdqIgMu2z19fXY/78+Yn11atXm+Ot3uTTTz9tjrXWwgP+evht27Yl1g4ePGiO9U5L7K3j99Y319XVJdasvivgr2f3trL2PP7444k1r4d/4MABs+4dI2DVrdNMl8K6zwG7Fw7Yx054v6vW+ROs05LzkZ0oCIadKAiGnSgIhp0oCIadKAiGnSgIhp0oiEz77L29veju7k6sr1+/3hy/fPnyxJq3dbC3Ht3r2d5+++2Jtfvuu88c6/W6vT66tyW0t7ba4q279vrJzc3NZr2npyex5q3TT3sOAuv6vZ+3d59ec801Zv3EiRNm/bPPPkuseXOzjjdJdd54EWkWke0i0iEin4jIj4qXN4rIVhH5tPh+onddRJSfUp7G9wH4iareAOBOAD8QkRsBPAdgm6rOBbCt+DkR1Sg37Krapaq7ix+fAtABYDqAZQAuPu9eDyD5OTYR5e6K/kEnIrMALADwZwDXqWoXMPgHAcCwm6WJyCoRKYhIwTr/HBFVV8lhF5EGAH8A8GNV/Wup41S1TVVbVbV14kS+rCfKS0lhF5E6DAb9N6q6sXhxt4g0FetNAJL/7UpEuXNbbzK4P+wbADpU9WdDSpsBrASwuvj+Le+6+vv7cfz48cS6t83tpk2bEmt79+41x3qtN2/Jo9Ue87bv9ZaRet+31U4B7PaYN9ba/hfwW1Dz5s0z60eOHEmsWcsxAaCzs9Osjx071qxPmjQpseb9zI4dO2bWvaW/o0bZ0bLaqV67s76+PrFmtXlL6bMvAvAYgL0isqd42fMYDPnvReRJAH8B8N0SrouIcuKGXVX/BCDpz/83KzsdIqoWHi5LFATDThQEw04UBMNOFATDThREpktcR40ahcmTJyfWvaWgVu/zhhtuMMeeOXPGrHvLLa1Dfb0lqF7PNm0f3hrvbdk8depUs37y5Emz3tLSYtZfffXVxNrGjRsTawDwxBNPmHXvlMvWKb69U4d7y0y9Xrh3v1vHL1jbXAP20t5US1yJ6O8Dw04UBMNOFATDThQEw04UBMNOFATDThREpn32gYEBs7/prW8+e/ZsYq2rq8sc6123d0pla31y2p6tdyppb+21VU/bw589e7ZZt04N7nnppZfM+p49e8z6M888Y9atcxh4xz5497l33IbXK7f67N5aeOu6rd9jPrITBcGwEwXBsBMFwbATBcGwEwXBsBMFwbATBZFpn72vr8/cwtfrhVtriL31xd75z73x3vnV04z1etVen94657133nhvPXt7e7tZ37Bhg1m3zlHgzW3Lli2p6tZW2l6Pf+bMmWZ9/PjxZt07N4PVK/f67FYv3TqXPh/ZiYJg2ImCYNiJgmDYiYJg2ImCYNiJgmDYiYIQr7ctIs0Afg1gKoABAG2q+gsReQHAPwK4uFH186r6rnVdt912m+7YsSOxbvXgAbsPb+0DDvg93dOnT5t1b32yxbuPvTXn1jp+wO7pbt261Rzb0dFh1nfu3GnWo5o/f75Zt/aGB+z18jNmzDDHHjp0yKydO3du2AM7Sjmopg/AT1R1t4iMA/CxiFz8Dfq5qv5rCddBRDkrZX/2LgBdxY9PiUgHgOnVnhgRVdYVvWYXkVkAFgD4c/GiH4pIu4isFZGJCWNWiUhBRAreqYCIqHpKDruINAD4A4Afq+pfAfwSwDcAtGDwkX/YTb1UtU1VW1W11XsdQ0TVU1LYRaQOg0H/japuBABV7VbVflUdAPArAAurN00iSssNuwwu2XoDQIeq/mzI5U1Dvuw7APZVfnpEVCmltN4WA/gPAHsx2HoDgOcBrMDgU3gFcAjA94v/zEtUX1+vVlvhwIEDpc6biBKoanmtN1X9E4DhBps9dSKqLTyCjigIhp0oCIadKAiGnSgIhp0oCIadKAi3z17RGxPJ7saIgkrqs/ORnSgIhp0oCIadKAiGnSgIhp0oCIadKAiGnSiITLdsBnAMwGdDPp9UvKwW1ercanVeAOdWrkrOLXGv6UwPqrnsxkUKqtqa2wQMtTq3Wp0XwLmVK6u58Wk8URAMO1EQeYe9Lefbt9Tq3Gp1XgDnVq5M5pbra3Yiyk7ej+xElBGGnSiIXMIuIktE5H9EZL+IPJfHHJKIyCER2Ssie0SkkPNc1opIj4jsG3JZo4hsFZFPi++H3WMvp7m9ICJHivfdHhFZmtPcmkVku4h0iMgnIvKj4uW53nfGvDK53zJ/zS4iIwH8L4AHAXQC2AVghar+V6YTSSAihwC0qmruB2CIyD0ATgP4tareVLzsZQAnVHV18Q/lRFX9pxqZ2wsATue9jXdxt6KmoduMA1gO4B+Q431nzOtRZHC/5fHIvhDAflU9qKoXAPwOwLIc5lHzVHUHgBOXXLwMwPrix+sx+MuSuYS51QRV7VLV3cWPTwG4uM14rvedMa9M5BH26QAOD/m8E7W137sC+KOIfCwiq/KezDCuu7jNVvH9lJzncyl3G+8sXbLNeM3cd+Vsf55WHmEf7vxYtdT/W6SqtwH4NoAfFJ+uUmlK2sY7K8NsM14Tyt3+PK08wt4JoHnI5zMAHM1hHsNS1aPF9z0ANqH2tqLuvriDbvF9T87z+X+1tI33cNuMowbuuzy3P88j7LsAzBWR2SJyFYDvAdicwzwuIyJji/84gYiMBfAt1N5W1JsBrCx+vBLAWznO5WtqZRvvpG3GkfN9l/v256qa+RuApRj8j/wBAP+cxxwS5nU9gP8svn2S99wAvInBp3W9GHxG9CSAawFsA/Bp8X1jDc3t3zG4tXc7BoPVlNPcFmPwpWE7gD3Ft6V533fGvDK533i4LFEQPIKOKAiGnSgIhp0oCIadKAiGnSgIhp0oCIadKIj/A+CVYyMlYQqWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with open(DATA_PATH / \"train-images-idx3-ubyte\", 'rb') as file_object:\n",
    "    raw_img = file_object.read(28*28)\n",
    "    img = struct.unpack(\">784B\",raw_img)\n",
    "    image = np.asarray(img)\n",
    "    image = image.reshape((28,28))\n",
    "    print(image.shape)\n",
    "    plt.imshow(image,cmap = plt.cm.gray)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0,)\n"
     ]
    }
   ],
   "source": [
    "with open(DATA_PATH / \"train-labels-idx1-ubyte\", 'rb') as file_object:\n",
    "    raw_img = file_object.read(1)\n",
    "    label = struct.unpack(\">B\",raw_img)\n",
    "    print(label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里好像有点错误，显示的错位了，但是我的确是按照格式进行处理的。这种格式处理起来比较复杂，并且数据集中的csv直接给出了每个像素的值，所以这里我们可以直接使用csv格式的数据。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据加载\n",
    "\n",
    "为了使用pytorch的dataloader进行数据的加载，需要先创建一个自定义的dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FashionMNISTDataset(Dataset):\n",
    "    def __init__(self, csv_file, transform=None):\n",
    "        data = pd.read_csv(csv_file)\n",
    "        self.X = np.array(data.iloc[:, 1:]).reshape(-1, 1, 28, 28).astype(float)\n",
    "        self.Y = np.array(data.iloc[:, 0]);\n",
    "        del data;  #结束data对数据的引用,节省空间\n",
    "        self.len=len(self.X)\n",
    "\n",
    "    def __len__(self):\n",
    "        #return len(self.X)\n",
    "        return self.len\n",
    "        \n",
    "    \n",
    "    def __getitem__(self, idx):\n",
    "        item = self.X[idx]\n",
    "        label = self.Y[idx]\n",
    "        return (item, label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于自定义的数据集，只需要实现三个函数：\n",
    "\n",
    "`__init__`： 初始化函数主要用于数据的加载，这里直接使用pandas将数据读取为dataframe，然后将其转成numpy数组来进行索引\n",
    "\n",
    "`__len__`： 返回数据集的总数，pytorch里面的datalorder需要知道数据集的总数的\n",
    "\n",
    "`__getitem__`：会返回单张图片，它包含一个index，返回值为样本及其标签。\n",
    "\n",
    "创建训练和测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = FashionMNISTDataset(csv_file=DATA_PATH / \"fashion-mnist_train.csv\")\n",
    "test_dataset = FashionMNISTDataset(csv_file=DATA_PATH / \"fashion-mnist_test.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在使用Pytorch的DataLoader读取数据之前，需要指定一个batch size 这也是一个超参数，涉及到内存的使用量，如果出现OOM的错误则要减小这个数值，一般这个数值都为2的幂或者2的倍数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#因为是常量，所以大写，需要说明的是，这些常量建议都使用完整的英文单词，减少歧义\n",
    "BATCH_SIZE=256 # 这个batch 可以在M250的笔记本显卡中进行训练，不会oom"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们接着使用dataloader模块来使用这些数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset,\n",
    "                                           batch_size=BATCH_SIZE,\n",
    "                                           shuffle=True) # shuffle 标识要打乱顺序"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_loader = torch.utils.data.DataLoader(dataset=test_dataset,\n",
    "                                           batch_size=BATCH_SIZE,\n",
    "                                           shuffle=False) # shuffle 标识要打乱顺序，测试集不需要打乱"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看一下数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([1, 28, 28]), torch.Size([28, 28]))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a=iter(train_loader)\n",
    "data=next(a)\n",
    "img=data[0][0].reshape(28,28)\n",
    "data[0][0].shape,img.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAR6klEQVR4nO3dbYxUVZ4G8OehedNu0Oalu3lvQKIubVYMggloUGRkMQQnihmiG9YYMXGMQzIflrgf4IMhZrMzk00kE3uCGWaddUKcAYkiO0hQ10QILUHAQUYgLNM00CDIiyCv//3Q10mLff+nrVtVt+A8v6RT3fXvU/d00Q+3qs8959DMICLXvx55d0BEykNhF4mEwi4SCYVdJBIKu0gkepbzYCT1p3+REjMzdnV/pjM7yZkkd5PcQ3JRlscSkdJioePsJKsA/BXADACtALYAmGdmf3Ha6MwuUmKlOLNPArDHzPaZ2QUAfwAwJ8PjiUgJZQn7MAB/6/R1a3Lfd5BcQLKFZEuGY4lIRln+QNfVS4XvvUw3s2YAzYBexovkKcuZvRXAiE5fDwfQlq07IlIqWcK+BcA4kqNJ9gbwEwBritMtESm2gl/Gm9klks8D+B8AVQBeM7PPitYzESmqgofeCjqY3rOLlFxJLqoRkWuHwi4SCYVdJBIKu0gkFHaRSCjsIpFQ2EUiobCLREJhF4mEwi4SCYVdJBIKu0gkFHaRSJR1KWnpGtnlJKW/K+XMxDyPnVWfPn3c+oULF1JrWX+usWPHuvX77rvPrb/++uuptYsXL7ptvX8z7+fSmV0kEgq7SCQUdpFIKOwikVDYRSKhsItEQmEXiYRWl5VMQuP0PXoUfj65fPlywW1DGhsb3XponDykvr7erZ86dSq19uqrr2Y6tlaXFYmcwi4SCYVdJBIKu0gkFHaRSCjsIpFQ2EUiofnsFSA0Vh2S55zz0LFLOVbe1NTk1h966KHUWv/+/d22e/fudev9+vVz6zfeeKNbP3PmjFsvhUxhJ7kfwGkAlwFcMrOJxeiUiBRfMc7s95vZsSI8joiUkN6zi0Qia9gNwJ9JfkJyQVffQHIByRaSLRmPJSIZZH0ZP8XM2kjWAVhP8nMz+7DzN5hZM4BmQBNhRPKU6cxuZm3JbTuAVQAmFaNTIlJ8BYedZDXJft9+DuBHAHYWq2MiUlxZXsbXA1iVjBH3BPDfZrauKL2KTJ7rwr/wwgtuPTTevHHjRrfuzRsfP36823b06NFuvaGhwa23tbWl1g4ePOi2Dc1HHzp0qFtfvny5W9+5s/znxYLDbmb7APxjEfsiIiWkoTeRSCjsIpFQ2EUiobCLREJhF4mElpIug1Jvizx48ODU2uTJk922gwYNcuuhvoeGoOrq6lJrffv2ddu2tra6dW85ZsCfxjpq1Ci37bp1/ijym2++6dazyPr7oqWkRSKnsItEQmEXiYTCLhIJhV0kEgq7SCQUdpFIaCnpRJ8+fdz6pUuXUmuh5ZJD46Kh6ZSzZs1y68OHD0+thaZSnjx50q1XV1e79ZDDhw+n1q5cueK2vfnmm916aDlmb4rsypUr3bahcfaQqqoqt+797KW69kVndpFIKOwikVDYRSKhsItEQmEXiYTCLhIJhV0kEprPXgShMdXQksgTJ/qb3/bs6V8O4W0PvGHDBrftyJEj3fo999zj1kNLMg8cODC1Fvq5Tpw44dZvv/12t/7BBx+k1tasWeO2LfUaBFm26dZ8dhFxKewikVDYRSKhsItEQmEXiYTCLhIJhV0kEprP3k3enPO7777bbRva1jg0L/ubb75x697xly1b5rZ955133PrZs2fdemhOutf3UNvQ89Kjh3+u2rJli1vPolevXm499LOF1kDweGP03hh88MxO8jWS7SR3drpvAMn1JL9Ibmt/aIdFpLy68zL+twBmXnXfIgAbzGwcgA3J1yJSwYJhN7MPARy/6u45AFYkn68A8EiR+yUiRVboe/Z6MzsEAGZ2iGTqhl4kFwBYUOBxRKRISv4HOjNrBtAMXL8TYUSuBYUOvR0hOQQAktv24nVJREqh0LCvATA/+Xw+gLeK0x0RKZXgy3iSbwCYBmAQyVYAiwG8DGAlyacBHAAwt5SdLIfbbrvNrXtrs7/99tuZjj1u3Di3vnbtWrd+xx13pNY+/vhjt+2jjz7q1pcuXerW58yZ49a3b9+eWvv666/dtr1793br3jx+IDwO7wnNGb948WLBjx0SWt9g8uTJqTVvPfxg2M1sXkppeqitiFQOXS4rEgmFXSQSCrtIJBR2kUgo7CKRuKamuBY6tQ8Aamv9iXmhobfVq1e7dY+3nDIQnsLqbXsM+Fsbh4b1Fi3y5zBt2rTJrR84cMCtz549O7V2/vx5t+3x41dPyfhhxx41alRqLbQEdkho+fARI0a49Ycffji1dsstt7htjx07VlC/dGYXiYTCLhIJhV0kEgq7SCQUdpFIKOwikVDYRSJxTY2ze1MWQ0vzetNAgfBU0CxCY/yhqZ79+/d363379k2tXbp0yW0bGkcP+fzzz916U1NTai20ZXN1dbVbb2hocOuhKbKemTOvXmP1u2699Va33tjY6Na93+XQ78unn36aWvOm3urMLhIJhV0kEgq7SCQUdpFIKOwikVDYRSKhsItE4poaZw9tg+tpa2tz6xMmTHDr69atS62F5i7X1aXujgUge9+GDh2aWluyZInbttS85Z5D4+DefHQgvA7AkCFDUmvPPvus23bKlCmZjn3ixAm37v3socd+//33U2unT59OrenMLhIJhV0kEgq7SCQUdpFIKOwikVDYRSKhsItE4poaZw+tDe/Zs2dPpmPff//9qTVvPjkAvPvuu249tGa9Nycc8Ofy19TUuG2zmjp1qlv35mafO3fObXvmzBm3vm/fPrfuXX8wb17a5sQdNm/e7NZDa/kPGDDArXu/M6G1GbwxfK9t8MxO8jWS7SR3drpvCcmDJLclH7NCjyMi+erOy/jfAuhq2Y5fmdmdycfa4nZLRIotGHYz+xCAvw+PiFS8LH+ge57k9uRlfuobM5ILSLaQbMlwLBHJqNCw/xrAWAB3AjgE4Bdp32hmzWY20cwmFngsESmCgsJuZkfM7LKZXQHwGwCTitstESm2gsJOsvPcwR8D2Jn2vSJSGYLj7CTfADANwCCSrQAWA5hG8k4ABmA/AH9y8DUgNA6/d+/e1FqW8X8gPBburTEO+OuIb9u2zW3r7Z8OhNfTnzt3rlvfsWNHaq1fv35u29Ca9944OgDMmDEjtbZw4UK37UcffeTWQ7zrMgDgqaeeSq3t3r3bbXvy5MmC+hQMu5l1dfXB8oKOJiK50eWyIpFQ2EUiobCLREJhF4mEwi4SibJPcSWZWss6hJWF1y8gW98GDx7s1kNDa8OGDXPr3vDWSy+95LZ9/PHH3fr06dPdemgZbG+r7NbWVrdtaEvmBx980K1PmzYttRZa6jmrrVu3uvVnnnkmtRZaSrpQOrOLREJhF4mEwi4SCYVdJBIKu0gkFHaRSCjsIpEo+zh7nmPpnlL2KzQV88svv3Troe2Dx48fn1p75ZVX3LaTJ09266ElkUP19vb21Jq3nTMA3HvvvW79gQcecOveWHrouoqQrL8v3lh6aAntQunMLhIJhV0kEgq7SCQUdpFIKOwikVDYRSKhsItE4prasrlS58J749wAMGbMGLd+9OhRt15fX19w+9B20MOHD3froTHfr776yq17y0E/9thjbttQPXR9Qp8+fVJr58+fd9tmHYc/e/asW7948WJq7eDBg5mOnUZndpFIKOwikVDYRSKhsItEQmEXiYTCLhIJhV0kEtfUOHspx9Jra2vdujcnPbT18Llz59z6sWPH3PqyZcvcujcW3qtXL7dtaJw8NOc8tMb5k08+mVpbvHix29bbJhsI/2zeWHrWcfSsvGsAQltVFyp4Zic5guRGkrtIfkbyZ8n9A0iuJ/lFcuunRURy1Z2X8ZcA/NzMbgdwD4CfkvwHAIsAbDCzcQA2JF+LSIUKht3MDpnZ1uTz0wB2ARgGYA6AFcm3rQDwSKk6KSLZ/aD37CQbAUwAsBlAvZkdAjr+QyBZl9JmAYAF2bopIll1O+wkawD8EcBCMzvV3T9wmFkzgObkMSpztUmRCHRr6I1kL3QE/fdm9qfk7iMkhyT1IQDSlxEVkdwFz+zsOIUvB7DLzH7ZqbQGwHwALye3b3XngFmGPLyht6qqKrdtU1OTW+/Z038qvH6Hhp9C00RD00xPnjzp1r1hnIEDB7ptQz/3hQsX3Hqo7++9915qbfXq1W7bEG+aaEhoGDfr0Fyob97zevjw4UzHTtOdl/FTAPwzgB0ktyX3vYiOkK8k+TSAAwDmlqSHIlIUwbCb2UcA0v6bm17c7ohIqehyWZFIKOwikVDYRSKhsItEQmEXicR1s2VzaOvg0DTU0Fh47969U2s1NTVu21C9b9++bj00Fu5NQ62urnbbXrlyJdOxhw0b5tZDy0F7QmPdlbr9N5BtnD40JbpQOrOLREJhF4mEwi4SCYVdJBIKu0gkFHaRSCjsIpGoqKWks4yrjhw50m3bo4f//1poLNwbbw71OzRWfcMNN7j1EO8agNBc+5C5c/2Zy88991zBj30tj6Nn5f2bHT9+vCTH1JldJBIKu0gkFHaRSCjsIpFQ2EUiobCLREJhF4lEWcfZa2pqcNddd6XWn3jiCbe9N2YcmhN+9uxZt+6NewLAqVOnUmuhbY9DQtcAhDQ0NKTWbrrpJrftmDFj3PrSpUvdektLi1vPe2vkQmUd4w+1b2xsTK1l/X1IfdySPKqIVByFXSQSCrtIJBR2kUgo7CKRUNhFIqGwi0SC3dinegSA3wFoAHAFQLOZ/SfJJQCeAXA0+dYXzWxt4LHcg9XW1rp98cbSx48f77YdNGiQWx81apRbr6urS62FxrL79+/v1kNzzkNj1e3t7am13bt3u21XrVrl1o8ePerWr1dVVVVu/fLly249y7/Z7Nmz3babNm1y62bW5cG7c1HNJQA/N7OtJPsB+ITk+qT2KzP7j248hojkrDv7sx8CcCj5/DTJXQD8bUBEpOL8oPfsJBsBTACwObnreZLbSb5GssvX4CQXkGwh6V9XKSIl1e2wk6wB8EcAC83sFIBfAxgL4E50nPl/0VU7M2s2s4lmNrEI/RWRAnUr7CR7oSPovzezPwGAmR0xs8tmdgXAbwBMKl03RSSrYNjZ8WfF5QB2mdkvO90/pNO3/RjAzuJ3T0SKpTtDb1MB/C+AHegYegOAFwHMQ8dLeAOwH8CzyR/zvMe6ftcGFqkQaUNvwbAXk8IuUnppYdcVdCKRUNhFIqGwi0RCYReJhMIuEgmFXSQSCrtIJBR2kUgo7CKRUNhFIqGwi0RCYReJhMIuEgmFXSQSZd2yGcAxAP/X6etByX2VqFL7Vqn9AtS3QhWzb6lropd1Pvv3Dk62VOradJXat0rtF6C+FapcfdPLeJFIKOwikcg77M05H99TqX2r1H4B6luhytK3XN+zi0j55H1mF5EyUdhFIpFL2EnOJLmb5B6Si/LoQxqS+0nuILkt7/3pkj302knu7HTfAJLrSX6R3Pr7XJe3b0tIHkyeu20kZ+XUtxEkN5LcRfIzkj9L7s/1uXP6VZbnrezv2UlWAfgrgBkAWgFsATDPzP5S1o6kILkfwEQzy/0CDJL3ATgD4Hdm1pTc9+8AjpvZy8l/lLVm9q8V0rclAM7kvY13slvRkM7bjAN4BMC/IMfnzunX4yjD85bHmX0SgD1mts/MLgD4A4A5OfSj4pnZhwCOX3X3HAArks9XoOOXpexS+lYRzOyQmW1NPj8N4NttxnN97px+lUUeYR8G4G+dvm5FZe33bgD+TPITkgvy7kwX6r/dZiu5rcu5P1cLbuNdTldtM14xz10h259nlUfYu9qappLG/6aY2V0A/gnAT5OXq9I93drGu1y62Ga8IhS6/XlWeYS9FcCITl8PB9CWQz+6ZGZtyW07gFWovK2oj3y7g25y255zf/6ukrbx7mqbcVTAc5fn9ud5hH0LgHEkR5PsDeAnANbk0I/vIVmd/OEEJKsB/AiVtxX1GgDzk8/nA3grx758R6Vs4522zThyfu5y3/7czMr+AWAWOv4ivxfAv+XRh5R+jQHwafLxWd59A/AGOl7WXUTHK6KnAQwEsAHAF8ntgArq23+hY2vv7egI1pCc+jYVHW8NtwPYlnzMyvu5c/pVludNl8uKREJX0IlEQmEXiYTCLhIJhV0kEgq7SCQUdpFIKOwikfh/PMG3KVbuIeoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(img,cmap = plt.cm.gray)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这回看着就没问题了，是一个完整的图了，所以我们还是用csv吧\n",
    "\n",
    "## 创建网络\n",
    "\n",
    "三层的简单的CNN网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CNN(NN.Module):\n",
    "    def __init__(self):\n",
    "        super(CNN, self).__init__()\n",
    "        self.layer1 = NN.Sequential(   \n",
    "            NN.Conv2d(1, 16, kernel_size=5, padding=2),\n",
    "            NN.BatchNorm2d(16), \n",
    "            NN.ReLU()) #16, 28, 28\n",
    "        self.pool1=NN.MaxPool2d(2) #16, 14, 14\n",
    "        self.layer2 = NN.Sequential(\n",
    "            NN.Conv2d(16, 32, kernel_size=3),\n",
    "            NN.BatchNorm2d(32),\n",
    "            NN.ReLU())#32, 12, 12\n",
    "        self.layer3 = NN.Sequential(\n",
    "            NN.Conv2d(32, 64, kernel_size=3),\n",
    "            NN.BatchNorm2d(64),\n",
    "            NN.ReLU()) #64, 10, 10\n",
    "        self.pool2=NN.MaxPool2d(2)  #64, 5, 5\n",
    "        self.fc = NN.Linear(5*5*64, 10)\n",
    "    def forward(self, x):\n",
    "        out = self.layer1(x)\n",
    "        #print(out.shape)\n",
    "        out=self.pool1(out)\n",
    "        #print(out.shape)\n",
    "        out = self.layer2(out)\n",
    "        #print(out.shape)\n",
    "        out=self.layer3(out)\n",
    "        #print(out.shape)\n",
    "        out=self.pool2(out)\n",
    "        #print(out.shape)\n",
    "        out = out.view(out.size(0), -1)\n",
    "        #print(out.shape)\n",
    "        out = self.fc(out)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以上代码看起来很简单。这里面都是包含的数学的含义。我们只讲pytorch相关的：在函数里使用torch.nn提供的模块来定义各个层，在每个卷积层后使用了批次的归一化和RELU激活并且在每一个操作分组后面进行了pooling的操作（减少信息量，避免过拟合），后我们使用了全连接层来输出10个类别。\n",
    "\n",
    "view函数用来改变输出值矩阵的形状来匹配最后一层的维度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.9031,  0.1854, -1.2564,  0.0946, -0.9428,  0.9311, -0.4686, -0.5068,\n",
       "         -0.3318, -0.6995]], grad_fn=<AddmmBackward>)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cnn = CNN();\n",
    "#可以通过以下方式验证，没报错说明没问题，\n",
    "cnn(torch.rand(1,1,28,28))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CNN(\n",
      "  (layer1): Sequential(\n",
      "    (0): Conv2d(1, 16, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))\n",
      "    (1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (2): ReLU()\n",
      "  )\n",
      "  (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (layer2): Sequential(\n",
      "    (0): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1))\n",
      "    (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (2): ReLU()\n",
      "  )\n",
      "  (layer3): Sequential(\n",
      "    (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1))\n",
      "    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (2): ReLU()\n",
      "  )\n",
      "  (pool2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (fc): Linear(in_features=1600, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "#打印下网络，做最后的确认\n",
    "print(cnn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从定义模型开始就要指定模型计算的位置，CPU还是GPU，所以需要加另外一个参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "DEVICE=torch.device(\"cpu\")\n",
    "if torch.cuda.is_available():\n",
    "        DEVICE=torch.device(\"cuda\")\n",
    "print(DEVICE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "#先把网络放到gpu上\n",
    "cnn=cnn.to(DEVICE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 损失函数\n",
    "多分类因为使用Softmax回归将神经网络前向传播得到的结果变成概率分布 所以使用交叉熵损失。\n",
    "在pytorch中 \n",
    "NN.CrossEntropyLoss 是将 `nn.LogSoftmax()` 和 `nn.NLLLoss()`进行了整合，[CrossEntropyLoss](https://pytorch.org/docs/stable/nn.html#crossentropyloss) ,我们也可以分开来写使用两步计算，这里为了方便直接一步到位\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "#损失函数也需要放到GPU中\n",
    "criterion = NN.CrossEntropyLoss().to(DEVICE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 优化器\n",
    "Adam 优化器：简单，暴力，最主要还是懒"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#另外一个超参数，学习率\n",
    "LEARNING_RATE=0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#优化器不需要放GPU\n",
    "optimizer = torch.optim.Adam(cnn.parameters(), lr=LEARNING_RATE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 开始训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "#另外一个超参数，指定训练批次\n",
    "TOTAL_EPOCHS=50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch : 1/50, Iter : 100/234,  Loss: 0.4569\n",
      "Epoch : 1/50, Iter : 200/234,  Loss: 0.3623\n",
      "Epoch : 2/50, Iter : 100/234,  Loss: 0.2648\n",
      "Epoch : 2/50, Iter : 200/234,  Loss: 0.3044\n",
      "Epoch : 3/50, Iter : 100/234,  Loss: 0.2107\n",
      "Epoch : 3/50, Iter : 200/234,  Loss: 0.3022\n",
      "Epoch : 4/50, Iter : 100/234,  Loss: 0.2583\n",
      "Epoch : 4/50, Iter : 200/234,  Loss: 0.2837\n",
      "Epoch : 5/50, Iter : 100/234,  Loss: 0.2377\n",
      "Epoch : 5/50, Iter : 200/234,  Loss: 0.2422\n",
      "Epoch : 6/50, Iter : 100/234,  Loss: 0.1537\n",
      "Epoch : 6/50, Iter : 200/234,  Loss: 0.2270\n",
      "Epoch : 7/50, Iter : 100/234,  Loss: 0.1485\n",
      "Epoch : 7/50, Iter : 200/234,  Loss: 0.1740\n",
      "Epoch : 8/50, Iter : 100/234,  Loss: 0.3264\n",
      "Epoch : 8/50, Iter : 200/234,  Loss: 0.2096\n",
      "Epoch : 9/50, Iter : 100/234,  Loss: 0.1844\n",
      "Epoch : 9/50, Iter : 200/234,  Loss: 0.1927\n",
      "Epoch : 10/50, Iter : 100/234,  Loss: 0.1343\n",
      "Epoch : 10/50, Iter : 200/234,  Loss: 0.2225\n",
      "Epoch : 11/50, Iter : 100/234,  Loss: 0.1251\n",
      "Epoch : 11/50, Iter : 200/234,  Loss: 0.1789\n",
      "Epoch : 12/50, Iter : 100/234,  Loss: 0.1439\n",
      "Epoch : 12/50, Iter : 200/234,  Loss: 0.1290\n",
      "Epoch : 13/50, Iter : 100/234,  Loss: 0.2017\n",
      "Epoch : 13/50, Iter : 200/234,  Loss: 0.1130\n",
      "Epoch : 14/50, Iter : 100/234,  Loss: 0.0992\n",
      "Epoch : 14/50, Iter : 200/234,  Loss: 0.1736\n",
      "Epoch : 15/50, Iter : 100/234,  Loss: 0.0920\n",
      "Epoch : 15/50, Iter : 200/234,  Loss: 0.1557\n",
      "Epoch : 16/50, Iter : 100/234,  Loss: 0.0914\n",
      "Epoch : 16/50, Iter : 200/234,  Loss: 0.1508\n",
      "Epoch : 17/50, Iter : 100/234,  Loss: 0.1273\n",
      "Epoch : 17/50, Iter : 200/234,  Loss: 0.1982\n",
      "Epoch : 18/50, Iter : 100/234,  Loss: 0.1752\n",
      "Epoch : 18/50, Iter : 200/234,  Loss: 0.1517\n",
      "Epoch : 19/50, Iter : 100/234,  Loss: 0.0586\n",
      "Epoch : 19/50, Iter : 200/234,  Loss: 0.0984\n",
      "Epoch : 20/50, Iter : 100/234,  Loss: 0.1409\n",
      "Epoch : 20/50, Iter : 200/234,  Loss: 0.1286\n",
      "Epoch : 21/50, Iter : 100/234,  Loss: 0.0900\n",
      "Epoch : 21/50, Iter : 200/234,  Loss: 0.1168\n",
      "Epoch : 22/50, Iter : 100/234,  Loss: 0.0755\n",
      "Epoch : 22/50, Iter : 200/234,  Loss: 0.1217\n",
      "Epoch : 23/50, Iter : 100/234,  Loss: 0.0703\n",
      "Epoch : 23/50, Iter : 200/234,  Loss: 0.1383\n",
      "Epoch : 24/50, Iter : 100/234,  Loss: 0.0916\n",
      "Epoch : 24/50, Iter : 200/234,  Loss: 0.0685\n",
      "Epoch : 25/50, Iter : 100/234,  Loss: 0.0947\n",
      "Epoch : 25/50, Iter : 200/234,  Loss: 0.1244\n",
      "Epoch : 26/50, Iter : 100/234,  Loss: 0.0615\n",
      "Epoch : 26/50, Iter : 200/234,  Loss: 0.0478\n",
      "Epoch : 27/50, Iter : 100/234,  Loss: 0.0280\n",
      "Epoch : 27/50, Iter : 200/234,  Loss: 0.0459\n",
      "Epoch : 28/50, Iter : 100/234,  Loss: 0.0213\n",
      "Epoch : 28/50, Iter : 200/234,  Loss: 0.0764\n",
      "Epoch : 29/50, Iter : 100/234,  Loss: 0.0391\n",
      "Epoch : 29/50, Iter : 200/234,  Loss: 0.0899\n",
      "Epoch : 30/50, Iter : 100/234,  Loss: 0.0541\n",
      "Epoch : 30/50, Iter : 200/234,  Loss: 0.0750\n",
      "Epoch : 31/50, Iter : 100/234,  Loss: 0.0605\n",
      "Epoch : 31/50, Iter : 200/234,  Loss: 0.0766\n",
      "Epoch : 32/50, Iter : 100/234,  Loss: 0.1368\n",
      "Epoch : 32/50, Iter : 200/234,  Loss: 0.0588\n",
      "Epoch : 33/50, Iter : 100/234,  Loss: 0.0253\n",
      "Epoch : 33/50, Iter : 200/234,  Loss: 0.0705\n",
      "Epoch : 34/50, Iter : 100/234,  Loss: 0.0248\n",
      "Epoch : 34/50, Iter : 200/234,  Loss: 0.0751\n",
      "Epoch : 35/50, Iter : 100/234,  Loss: 0.0449\n",
      "Epoch : 35/50, Iter : 200/234,  Loss: 0.1006\n",
      "Epoch : 36/50, Iter : 100/234,  Loss: 0.0281\n",
      "Epoch : 36/50, Iter : 200/234,  Loss: 0.0418\n",
      "Epoch : 37/50, Iter : 100/234,  Loss: 0.0547\n",
      "Epoch : 37/50, Iter : 200/234,  Loss: 0.1003\n",
      "Epoch : 38/50, Iter : 100/234,  Loss: 0.0694\n",
      "Epoch : 38/50, Iter : 200/234,  Loss: 0.0340\n",
      "Epoch : 39/50, Iter : 100/234,  Loss: 0.0620\n",
      "Epoch : 39/50, Iter : 200/234,  Loss: 0.1004\n",
      "Epoch : 40/50, Iter : 100/234,  Loss: 0.0588\n",
      "Epoch : 40/50, Iter : 200/234,  Loss: 0.0309\n",
      "Epoch : 41/50, Iter : 100/234,  Loss: 0.0387\n",
      "Epoch : 41/50, Iter : 200/234,  Loss: 0.0136\n",
      "Epoch : 42/50, Iter : 100/234,  Loss: 0.0149\n",
      "Epoch : 42/50, Iter : 200/234,  Loss: 0.0448\n",
      "Epoch : 43/50, Iter : 100/234,  Loss: 0.0076\n",
      "Epoch : 43/50, Iter : 200/234,  Loss: 0.0593\n",
      "Epoch : 44/50, Iter : 100/234,  Loss: 0.0267\n",
      "Epoch : 44/50, Iter : 200/234,  Loss: 0.0308\n",
      "Epoch : 45/50, Iter : 100/234,  Loss: 0.0150\n",
      "Epoch : 45/50, Iter : 200/234,  Loss: 0.0764\n",
      "Epoch : 46/50, Iter : 100/234,  Loss: 0.0221\n",
      "Epoch : 46/50, Iter : 200/234,  Loss: 0.0325\n",
      "Epoch : 47/50, Iter : 100/234,  Loss: 0.0190\n",
      "Epoch : 47/50, Iter : 200/234,  Loss: 0.0359\n",
      "Epoch : 48/50, Iter : 100/234,  Loss: 0.0256\n",
      "Epoch : 48/50, Iter : 200/234,  Loss: 0.0374\n",
      "Epoch : 49/50, Iter : 100/234,  Loss: 0.0198\n",
      "Epoch : 49/50, Iter : 200/234,  Loss: 0.0300\n",
      "Epoch : 50/50, Iter : 100/234,  Loss: 0.0465\n",
      "Epoch : 50/50, Iter : 200/234,  Loss: 0.0558\n",
      "Wall time: 7min 18s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#记录损失函数\n",
    "losses = [];\n",
    "for epoch in range(TOTAL_EPOCHS):\n",
    "    for i, (images, labels) in enumerate(train_loader):\n",
    "        images = images.float().to(DEVICE)\n",
    "        labels = labels.to(DEVICE)\n",
    "        #清零\n",
    "        optimizer.zero_grad()\n",
    "        outputs = cnn(images)\n",
    "        #计算损失函数\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        losses.append(loss.cpu().data.item());\n",
    "        if (i+1) % 100 == 0:\n",
    "            print ('Epoch : %d/%d, Iter : %d/%d,  Loss: %.4f'%(epoch+1, TOTAL_EPOCHS, i+1, len(train_dataset)//BATCH_SIZE, loss.data.item()))\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练后操作\n",
    "### 可视化损失函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEVCAYAAAAYZ2nCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd5xU1fk/8M/0XnZ2tksvAsqCNFEECygCAhEwCkaJCqiJiGJBUAnGGEwwmvizxKCxRaKIX3sBDTZEUFSasCBIZ/vM7Mzs9LnP74/1XrcBOzOXhes879eLF7Mz95499+yd57nnlnNURERgjDHGUqA+0RVgjDGmPJw8GGOMpYyTB2OMsZRx8mCMMZYy7YmuQDYiItTV1aG2thZ1dXWor69HXV0dvF4vamtrEQgEEI1GEYvFEIvFEI/HEQqFUF9fj3A4jFgshkQigWQy2aRclUoFjUYDrVYLvV4PnU4HrVYLnU4HnU4Hs9kMl8sFu90Om80Gh8MBi8UCp9MJh8MBo9EIo9EIi8UCh8MBnU53glro+EokEvD5fAgGg6ivr4ff75faNhwOIxKJIBgMIhAIIBQKSf9isRii0SgikQji8TgSiYT0TxAECIIA8f4TlUoFAFK7N25bg8EAnU4Hq9UKh8MBh8MBu90Ou90uvc7Pz4fD4ZDKUZpAIACPx4P6+nrpXygUQiAQQCAQkNpXfC22aSQSQTQaRTweRywWa7KPq1Qqad/W6/UwmUyw2WzSv8bt53Q64XQ6pdc5OTm/iP05Go3i8OHD8Hq98Hg8qKyslPbfSCQi7avRaFTap8V9NZlMQhAElJaWYsmSJRnXRZHJY86cOdi6dStMJhOcTidcLpcUDE0mE6xWK3JycqQdyeVyweVywWKxQKuVZ5MFQUA4HEYgEIDf70coFILf74ff70cwGERlZSUqKytRUVGB2tpa6TOv14vy8nJEIpGjlq9SqaQvifhFsVgsMJlMMBgM0Gg00Gg0UKlUUKlUICIkk0lEo1EkEgkp6SQSCcTjcSkB+Xw+CILQpm00Go1wOp3Izc2F1WqFxWKBy+WC2+2WvpT5+fnIzc2FxWKRvrzil9ZkMske/GKxGKqrq+HxeKTAU1tbi9raWikIBYNBeL1e+P1+1NXVIRAISAEsGAyipqamzW0AACaTCSaTCXq9HgaDAUajUUrM4j+1Wi39AxoOEMR9pLKyUkpKoVBICpSxWOyov1ev1yM/Px95eXnIz89HUVERCgoKUFBQALPZDKfTCbfbjZycHLjdbjidTlitVqkOmSIiRKNR6cBFTADigU95eTkqKiqk/ysqKuDxeKS/RVsYDAZYrVaYTCZotVoYjUYpuer1emkfB4BkMolIJCIdVEUiEen7Fw6Hj/m7zGYzrFYrbDab1Ka5ublwuVwwm83Iy8uD2+2W9nWHw4GcnBwpEcnRrkSEWCyGUCiEYDAIv9+P6upqeL1e6Wdxm8QDyvLyclRXV6OqqgrV1dVHLV+j0cBsNsNgMEjxovG+qtFoEAqFMt4OAFAp8VbdOXPmYMOGDYhEIvB4PPD5fAgEAi2OxFuj0+lgMBig1+thNpulo0KDwSA1rlqthiAISCaT0pc8Ho9LwUcMAMei0WiQn5+P/Px8Kbk5nU4UFhaiqKgIbrdbOvp3OBxwuVzIycmB3W6HVqs9LkedgiBIR4A+nw/19fXw+Xyoq6tDJBJBJBKRekLi0aPH45GO0mtra+HxeOD3+xGNRo+5/RaLRUp+YoAQe0JqtVpKguIXM5lMIplMSglQrFMsFkMwGGxTUBIDq3hUb7PZYDabYbFYYLPZpL+JxWKR3hO/aOI/McgYjUbZgnFz8Xgcfr8fPp9PChp1dXWoq6tDZWUlqqqqUFVVhZqaGilAV1VVIR6PH7FMlUolJW4xAOt0OmkfF4OxWq2GSqWSekyxWAzhcFgKauJR67HCg1qtRn5+PoqLi1FYWAi32w2Xy4Xi4mLk5uZK7W6xWGA2m6Ver9VqhdVqla03kEwmmxws+Hw+qV19Ph+8Xq8UJwKBgNSu1dXV8Pl8xwyoYrtaLBapXcU4IgZnjUYj1UXch6PRKKLRKMLhsNTbbUvI1Wq1UrwoKCiQ2rakpAQlJSXSQUNBQQEcDocUx3Q6Xbv1VhWZPFpDRAiFQgiHw9KRZ11dHfx+P2pqauD1eqUjJ/GUkNjFE7vKYteOiKRTQI2/gOIOL/YCzGaz1GUWj7ztdjusVivy8vKQm5ur2NMObREKhVBVVSW1rRj4GgfDYDAoBSbxiFv8JyZosc0BSAlFPD0hnu7R6/WwWq1wuVzSEaIYhHJycpCXlweLxXJcg/3JQBAE6TSFeOpC7Hk1bn/xdIV44CPu42Jbi//ERGIwGJokTnH/Fvd18WdxP8/NzZWS8C+hvQVBQE1NjdRranwq2efzSQep9fX10v4rHtSIPXyxN9t4HzYYDDAYDNIBjdVqhdFolGKH2JYulwtWq1VKrsej1y4qLi7GxIkT8eSTT2ZUjmKTR8+ePXHuuedi6dKlJ7oqjDGmGF27dsXZZ5+N//znPxmVo9hDBr1eD4/Hc6KrwRhjimI2m9t0jehYFJs8TCaTLA3AGGPZRK7Yqdjkodfrj3nBljHGWFNyxU7FJg/xjijGGGNtJ1fsVHTyUOi1fsYYO2Hkip2KTR7ibYbprLe7OoikwImHMZZ90o2dzSk2eQiCkFYD/Gf9foz826eY99rm41Arxhg7uaUbO5tTbPJIN3u+/NV+AMCKbw7KXSXGGDvpZX3PI5lMSsMBpCLfZjgOtWGMMWVIN3Y2p9jkEY1GYTCknggKHcbjUBvGGFOGdGNnc4pNHpFIBEZj6omg2GE6DrVhjDFlSDd2NqfY5BGPx9MakbOT23IcasMYY8qQbuxs7oQkj0Qigb1792b0iHwsFoNer095vSI+bcUYy2Lpxs7m2j15/L//9/+Qk5ODLl26wOVyYdWqVWmVk272dFv5gjljLHspsufx0Ucf4dZbb8XSpUtx6NAhLFmyBBMmTMCuXbtSLiscDsNkSv36ReOeRzzJw5swxrJLurGzuXZNHitWrMDZZ5+NK664AsXFxZgxYwai0Si2bt2aUjnihDhOpzPlOhh1P9+iVh3ggRUZY9kjk9jZXLvOYV5SUoJXX30VGzduRGlpKZ5++mkAQIcOHVosu337dpSVlUmznDmdTpjNZnTt2hWCIICI4HA4MqqPpz6GYifffcUYyw7iNLiZxk6gnZPHnDlz8Morr2DAgAFwu92orq7Gww8/jIEDB7ZYdvny5Vi0aFGL96+++mrcf//9AJBxA1QHuefBGMsePp8PQOaxE2jn01aHDx9GRUUF+vTpgxtuuAEdO3bE4sWLsXlzy3Gmkslkq2W43W7U1NQAAHJzczOqT0VdJKP1GWNMSeSKnUA79zzuvfdenH766Vi1ahX0ej1uv/12DBw4EHfffTfefvvtJsv27t0bEydORDKZRDQahdfrRTAYREFBAbxeL4DMG+Cwj2ciZIxlD7liJ9DOyeN///sfFi9eLN1jbLfbcdFFF+H9999vsezUqVMxderUVst55ZVXAAAulyuj+hzycvJgjGUPseeRaewE2vm0lcPhwMaNG6WfBUHA1q1bW71gfjTiebucnJyM6sPXPBhj2USu2Am0c89jwYIFmDVrFn744Qd0794d3377LTZu3IgPPvggpXJCoRAAwGLJbKgRfySR0fqMMaYkcsVOoJ17HjNnzsT69esxcOBAxONxjB8/Hj/88AMuuOCClMqprKyETqeD3W7PqD7e+lhG6zPGmJLIFTuBdu55AMCQIUMwZMiQjMqorKxEfn4+1OrMcp8/Es9ofcYYUxK5Yieg0FF1y8vLUVhYmHE5oWjrtwMzxtgvkVyxE1Bo8qiqqkJRUVHa62vUDVMwxnhsK8ZYFsk0djamyORRXV0Nt9ud9vomXeZTMDLGmNJkGjsbU1zyICJUVVUhPz8/7TL0WsVtNmOMZUSO2NmY4qJoXV0dYrFYRg0gnrZijLFsIUfsbExxyaOqqgoAUFBQkHYZGhUnD8ZYdpEjdjamuOTh9/sBZDYqJOcOxli2kSN2Nqa45FFXVwdAvgZgjLFsIHfsVFzyELOnzWZLuwwiuWrDGGPKIEfsbEyxySOTx+sTAmcPxlh2kSN2Nqa45CF2vTKZgzch8MOBjLHsIkfsbEyxySOT7JlMcs+DMZZd5IidjSkueQSDQej1euh0urTLCMV5TCvGWHaRI3Y2prjkEY/HM9745E/XPIw6xW0+Y4ylRY7Y2Zjiomc0GoXRaJSlLLtRvoZkjLGTmZyxE1Bg8qivr4fZbJalrByzXpZyGGPsZCdn7AQUmDwikYhs2dNl4eTBGMsOcsZOQKHJw2QyyVJWns0gSzmMMXaykzN2AgpMHqFQSLYGyLVyz4Mxlh3kjJ2AApOHnHcM5PJpK8ZYlsj6u60AyDJ5OwAU2OU7/8cYYyc7uWInoMDkQRmOaig0Gtcqn5MHYyxLZBo7m1Nc8siUNxSTXjtM/JwHY4ylQ3HJQ6VSQchgYENP/c/Jw6LXyFElxhg76WUaO5tTXPJQq9UZNYA/Epde2/gJc8ZYlsg0drYoT7aS2kmmDRCIJKTXFgP3PBhj2SHrk4dWq0UikTj2gkdQ5Y9Kry16rRxVYoyxk16msbO5rEseu6uD0mu1WiVHlRhj7KTHySPDBjjoC8tYG8YYU4asTx46nQ7xePzYCx7Bvtp6GWvDGGPKkGnsbE5xycNoNCISiaS9/p5qTh6MseyTaexsTnHJw2AwIBqNHnvBI6iPNUxByyPqMsaySaaxsznFJQ+9Xo9YLHbsBY+hk0u+SVEYY+xkJ1fsFCkueZjNZoTDmV/07p5vlaE2jDGmDHLFTtEJSx4fffQRHnjggZTXExsg04ddihzyjWvPGGMnO7lip+iEJI+VK1di3LhxCAaDx164GXEO3kwv/ORYeGgSxlj2kCt2ito9eWzYsAGTJk3CrFmz8Oc//znl9W02GwAgEAhkVA+ey4Mxlk3kip2idh+f44477sCUKVPw6KOPQqU68hPe27dvR1lZGdRqNQwGA5xOJ3r27AmrteFaRTAYREFBQdr1KHHyaSvGWPaQK3aK2jV5rF+/HmvXrsVvf/tb3HXXXSgsLMSUKVPQoUOHFssuX74cixYtavLe448/DrfbDQBpXfhpPBlKp1y+24oxlj2MxoazLXJdNG/X01aPPPIIYrEYbrzxRqxYsQJ33XUXTj31VKxdu7bFsslkssV74XBYmsA9nQbwhX5+utJq4EERGWPZI5PY2Zp2jaCbN2/GlClT8O9//xs2mw01NTUYNWoU/vrXv+KNN95osmzv3r0xceJEJJNJRKNReL1eWCyWjBrgcN3P6xztlBljjP3SKDp5BAIBnH322dKFG7fbjV//+td48sknWyw7depUTJ06tcX7X375JQCgvj71YUbKffI9ms8YY0pisVgApBc7W9Oup62Ki4uxe/fuJu95PB7k5ua2uQy73Q4gvTsGKvycPBhj2SmT2Nmadk0el1xyCV5++WXs3bsXALB37168+OKLmDRpUpvLyCR7VgXkG9eFMcaURNE9j5tuugmdOnXCaaedhsGDB6NPnz7o1asX5s6d2+YyxNvN0mmAQES+4YgZY0xJMomdrWnXax45OTn4+uuv8f7772PPnj3o378/hg0bltLFa6fTCbVajaqqqpR/fyTe8g4uxhjLBpnEzta0+/2qarUa48aNS3t9rVYLt9udVgOEY5w8GGPZKZPY2RrFjaoLNHS/0rnoE4xy8mCMZa90Y2drFJk8LBZLWuftwnH55u9ljDGlSTd2tkaxySMUCqW8Xiwhz1DEjDGmROnGztYoMnnYbLa0ul6xJB17IcYY+4VKN3a2RpHJw+FwoK6uLuX1kjJNgsIYY0qUbuxsjSKTh91uT6sBiDsejLEslm7sbI0ik0dOTg58Pl/K6/FYiIyxbJZu7GyNIpOH1WpFKBRKeS5erVqRm8sYY7JIN3a2RpHRVJzUJNW5eA1aRW4uY4zJIt3Y2RpFRtN0x2ix8ARQjLEsJuf4VopMHuIQ7tXV1Smtx8mDMZbN0o2drVF08vB6vamtZ9Efj+owxpgipBs7W6PI5CF2vYLBYErr2Y3c82CMZa90Y2drFJk8xGlsU31S0mnmngdjLHulGztbo8jk4XK5AAA1NTUprZdr5eTBGMte6cbO1igyeeTl5QFI/aKP3ag7HtVhjDFFSDd2tkaRyUOv18NqtcLj8aS0nrXRNY9Ekse5Yoxll3RjZ2sUmTyAhgs/qV700Wt+3twwT0nLGMtC6cTO1ig2eej1esRisdTWafSEeYinpGWMZaF0YmdrUkoe8XhctrHgM2U0GlN+xN7a6CHBQCQud5UYY+ykl07sbE2bk8df/vIXWCwW9O/fHwBw6NAhTJ06FZ07d8add94py0BbqUinARzmny+Ye0OcPBhj2afdk4fFYkE8Hscbb7wBALj66qvx8ccf47LLLsOSJUvw5ptvZlyZVKTT9bI16nkc9oXlrhJjjJ302v201aRJkwAAu3btwo8//ojVq1fjX//6F5YsWYI+ffrgiy++yLgyqdBqtUgkEimto2o0oceeGnkmgWeMMSVJJ3a2ps3Jo7i4GEOHDsX999+PFStWQKvV4vzzzwcA6HQ6RKPRjCuTCo1Gg2Qy/Yveezl5MMayUKaxU5TSBfPFixfjxx9/xLx58zBhwgTYbDbs27cP27ZtwxlnnJFxZVKh0Wgyus6yqzrzW9UYY0xpMo2dopRGCjzvvPOwY8cObN++HWeeeSaAhtNYF154IaZNm5ZxZdrTzgpOHowxlq42J49YLIaysjIIgoDzzjsPACAIAgYNGoR33nmnyfWE9iAIArTa1EfJLXGacMgXRoyfMGeMZaF0Y2dzbT5ttWDBAvTr1w9XXHEFAOCbb75B37594XQ6cfHFFyMcbt+7l5LJJDQaTcrr9SywHofaMMaYMqQbO5trc/IQn+/45JNPQESYPn069Ho9XnzxRaxevRovv/xyxpVJRfrJwya9JiI5q8QYYye9dk8e48ePh1arxbvvvostW7bg+++/x6OPPorf/OY3OP300/Hdd99lXJlUCIIAtTr10VXybAbpNY9vxRjLNunGzubafOLL4XBg9OjRWLRoEcaOHQuz2YwhQ4YAAKLRqCzn0FIRj8eh06U+xHrjIUrCsSTMep5dkDGWPdKNnc2llH4efvhhOBwO/Otf/8KNN94Ig8GAb775BmVlZRg+fHjGlUlFug2Q02gecx4ckTGWbeRKHikddvfs2RObNm2C3+9HTk5OQwFaLebPn4+JEydmXJlUJBKJtBogv9Fpq/pY5k9ZMsaYkqQbO5tL+ZyNWq1GeXk51q1bB5fLhb59++KBBx5IuwLpnn8Lh8MwGo0pr+e2/pw8uOfBGMs26cbO5lKK2j/++CMGDBiA0047DePHj8fZZ5+NTp06YenSpSn/Yp/Ph8GDB6Nfv34prws0NIDJZEp5vcbzmAcj3PNgjGWXdGNncyklj1tuuQU+nw8rV66U5vaYO3cuZs2aha+++iqlXzx79mxs2LABlZWVKa0nisVi0Ov1x16wGZPu51vU/DynB2Msy6QbO5tLKXls27YNkydPxkUXXQSVSgWz2Yz58+ejR48eWLZsWZvLee2117Bs2TJcd911KVcYaHg+o76+HlZr6g/8NX4SvtLfvoM5MsbYiZRJ7GwupWseI0eOxLJly3DrrbeipKQEAFBRUYHKykrp52OprKzE9ddfj7lz56Jr16546623Wl1u+/btKCsrg1qthsFggNPphNlsRpcuXaRRIW02W6vrtlWlP/MJURhjTCnC4bAssRNIMXksXLgQq1atQrdu3TBy5EhYrVb873//g8vlwsyZM4+5PhFh5syZcDqduO+++/D8888fcdnly5dj0aJFLd6/4oor8MgjjwAA7HZ7KtVvoTrAPQ/GWPbw+/0AMo+dQIqnrUpKSrB161YsWbIEdrsdwWAQ8+bNw4YNG+B0Oo+5/tKlS/HOO+/gueeeg9lsPuqyRxpv3uFwwOfzAUCbfufRVNRxz4Mxlj3kip1AGrfqWiwWzJ49G7NnzwbQcNpq3bp1GDdu3FHX2759O2666SYYjUbMnz8fer0eFRUV8Pl8mDZtGu6++26cdtpp0vK9e/fGxIkTkUwmEY1G4fV6EQwGkZ+fj7q6OgANiSQT3lDmUzEyxphSyBU7gTSSR3Pvv/8+rrvuOhw8eBDFxcVHXM7lcmHhwoUIBAKIx+OIRqOIRqNQqVTQaDQtpkWcOnUqpk6d2mpZH374IYDMGyDAt+oyxrKIeNrqpEgew4cPBxHhww8/xPTp04+4XEFBAe65554m7z355JPYuXMnXnzxxZR+Z319wxSyFosl9Qo34uOeB2Msi8gVO4EUr3m0plu3bujQoQM2bNiQ8rrpzqNbW1sLANIQKanSqBtu163nJ8wZY1kk09jZWMbJQ6VS4bTTTsPOnTtTXnfo0KG4+OKLU16vqqoKQENvJh0OU+bjujDGmNJkGjsbO+ppq1AohHfffRdEBI1GA41GA4PBALPZDIvFAqvViry8POTn52Pz5s0p//JBgwbhhRdeSHk9n88Hg8GQ9iP2TpMOnno+ZcUYyy6Zxs7Gjpo8vvnmG1x++eVtmnHv3HPPzbgybeX3+zO6T9mgy3wWLcYYU5pMY2djR00ew4cPRzweRyKRQDKZlG6bDYVCCIVCCAaDqKurQ319PYYOHSpLhdqipqYGLpcr7fUN2sxn0WKMMaXJNHY2dsy7rcTTVSI5HmvPlMfjQW5ubtrr6zWcPBhj2SfT2NmYIqNofX19ZreaqY69CGOM/dJkHDsbUWTyCAaDGY0K2ZZrOIwx9kuTaexsTJHJo7a2NqPzdtGEIGNtGGNMGTKNnY0pMnn4fL7MkkeckwdjLPtkGjsbU1zyiMfjiEQiGV24D0Z5TCvGWHaRI3Y2prjkIceokDymFWMs28g5oi6gwOQhx8Be4phWOg3fdsUYyw5yDooIKDB5RCINEzgZjcaMy8q1GDIugzHGlEDO2AlkefLIs3HyYIxlh6xPHnKet8u16jMugzHGlCDrr3nIOQevy8zJgzGWHeSMnYACk4ecF33y7HzaijGWHbL+grnY9ZIjexbY5Dn3xxhjJzs5YyegwOQRCAQAyDO6b0lO5hOiMMaYEsgZOwEFJg+/3w+1Wg2z2ZxxWR1dmZfBGGNKIGfsBBSYPDweD5xOJ9Tq9KreeERd7nkwxrJFprGzOcUlj1AolFHm9Ibi0mub4ZhzYTHG2C9CprGzOcUlj3g8Dp1Ol/b65XVh6bVKxcOTMMayQ6axs7msSx7VgaiMtWGMMWXI+uSRSCSg1aZ/uumQL3zshRhj7Bcm09jZnOKSR6bZc19tSMbaMMaYMmR9zyMWi0GvT39YkYNeTh6MseyTaexsTnHJI9Ou134PJw/GWPbJ+tNWyWQSGo0m7fUr/XzBnDGWfTKNnc0pLnkQUUYPuYh3W/FduoyxbJJp7GxOcckDkOf5jBInP13OGMsucj7bpsjk0XiIkVSN6l0AALhnXB+5qsMYY4qQSexsLuuSRzieAABYDPKd+2OMMSXI6uSh0WiQTCbTXj8QEZMHj2vFGMsemcbO5hSXPLRabUYNUBduGBjRaZLvYRnGGDvZZRo7m1Nc8tDr9YhG07/dVux5WI3c82CMZY9MY2dzJyR5JJNJ/PDDD6isrEx5XZPJhHA4vfGpiKhRz0O+Jy0ZY+xkl0nsbE27J4933nkH3bp1Q8+ePVFYWIgxY8agpqamzetbLBZpIvdUhWJJJAWCUaeGXqu4ThdjjKUtk9jZmnaNoK+++iomTJiAiy++GBs3bsSKFSuwbt06/OEPf2hzGWazOe3sGY43nO8z6/mUFWMsu2QSO1vTrlG0d+/eePPNNzF+/HgAQL9+/bB06VLs3r27zWXodDrEYrG0fn8i2XCbmlbNj5czxrJLJrGzNe2aPE4//XScfvrp0s8//PADPv30U9x5550tlt2+fTvKysqgVqthMBjgdDpRWloKvV6fdgPEkwIAQKfhU1aMseySSexszQk7f/Pxxx9j2rRp6NSpE2655ZYWny9fvhyLFi1q8t7nn38uNQARpfyofeyn5MHXOxhj2SaT2Nmado+iiUQC8+fPx8iRIzF06FCsWbMGOTk5LZZr7X7kaDQKg8EAIkIikUj9d/NpK8ZYlsokdramXXsegiBg8uTJWL16NV544QVceeWVR8yAvXv3xsSJE5FMJhGNRuH1egEANpsNAOD3+5Gbm5vS708IDT0PDScPxliWySR2tqZdk8enn36Kt956CytXrsRFF1101GWnTp2KqVOntnj/8OHDAACv15t68vip58HXPBhj2UaMl+nEzta0a/JYvXo1OnfujDPPPBO7d+9GMBiERqNBr1692jzDlXiKy+PxpPz7xSHBuOPBGMs2mcTO1rRr8lCpVNi7dy+cTmeT9y+66CKsXLmyTWU4HA4AQF1dnez1Y4yxXyq5Y2e7Jo/bbrsNgwYNgt1uh9PphNVqRSQSSekCjsViAYC0npQUexxJGYclZowxJcgkdramXZOHw+HAhAkTMi4DSC97ihfKxWsfjDGWLeTueSjuyrF4oSeV8bBE2p/m700KnDwYY9klk9jZGsUlD4fDAaPRiPLy8pTXFR8OFB8WZIyxbJFJ7GyN4pKHSqVCUVERKioqUl7X8FPyiMY5eTDGsksmsbM1ikseQMMtZz6fL+X1jLqGecvF0XUZYyybpBs7W6PI5GG329O66GP7afbAQCQu60TwjDGmBOnGztYoNnkEAoGU19Np1DDq1BCIex+MseyTbuxsjSKTR25uLqqqqtJa1ySeuopx8mCMZZdMYmdzikwehYWFqKqqSuvUk8XQcOqqPsrJgzGWXTKJnc0pMnkUFBQgmUyitrY25XWtPyUPfyQud7UYY+yklknsbE6xyQMAqqurU17XYdIBAPxhTh6MseySSexsTpHJw2q1AgCCwWDK67osegBATb180zEyxpgSZBI7m1Nk8rDb7QAaJjVJVZ7NAADwBKOy1okxxk52mcTO5rIueYjXPEcR+MMAACAASURBVAIReaZiZIwxpcj65GE2mwGkN7RwzU89jk0H5XnKkjHGlCKT2NmcIpOHmD3TedhlxTcHAQAfba/Cxzvkud+ZMcaUIJPY2Zwik4c4kXs6DeCyGKTXT3/+o2x1Yoyxk10msbM5RSYPk8kEAAiFQimvW3qKQ3rNo+syxrJJJrGzOUUmD7VaDaPRmNZ5uyKHUXodSfBT5oyx7JFJ7GxRlgz1OSHMZjPC4XDK67218bD0Osh3XDHGsky6sbM5xSYPq9Wa1oMugejPCWNvbQgBHqaEMZZF0o2dzSk2eVgsFlkaoO+iVQhGuQfCGMsOcsVOxSYPnU6HeDz1XkP3fGuL93ZXZd6QjDGmBOnGzuYUmzz0ej1isdTHpxJaGYpYo1bJUSXGGDvppRs7m1Ns8kg3e+6vbXmLGicPxli2yPqeh0ajQTKZ+q22f760b4v36qMJntOcMZYV0o2dzSk2eajV6rQC/q8Hd2jx3pR/fon73t6G6kAU2w5nPmAYY4ydrNKNnc1pZajLCSEIArRa+ar/3Nq9+HRnNfbU1GNc3yL8+dK+cJh1spXPGGMnA7lip2J7HslkEhqNRtYy99Q0PHX57pZyPPDeNsSTAuYu34h3Nh8+xpqMMaYMcsVOxfY8EomErD2P5pZvOAhfKI5V2yrxf98egkDAG98dwuqyKrz5+2F4bu1efLyjCu/MPgen5JiPWz0YY0xOcsVOxSaPaDQKg8Fw7AUzsGpbpfT65v9+J72e+PgX0uu3N5XjxvO6AQCICN/u92Hu8o3YVxvC9ed2xczhXeG2/lzPRFKARq2CSsV3eDHG2p9csVOxySMSicBoNB57wVbcefGp+OsHO2Spx18+KMNfPihr9bOnPv0RL6zdhyWXlWJEzzzMfWUTPtpeielndcK8Mb3wwLvbcU53N8b0LWqx7q6qAAQCehbYEIwmYNJpZL2l+MNtlVi2fh/mjOqJ/h2cspXLGDu5ZRI7G1ORQu9R7dChA0aNGoVnn3025XWJCDsqA7j4758fh5q1jUWvQX2s4Xa5S0qLsPCSPvhfWRWCkQQO+cJ4bu3eJsuXOE34/fndYTdpcffrW/HklQNwdnc3ACCaSEKtUmHt7lp8XFaFPkX2FneVEVGT3s6M57/GR9sbJsP6528GYPRphSd1b4iI8OXuWiQEwoieeSe6OowpViaxszHF9jxCoZA0pWKqVCoVOrpO7HUKMXEAwDuby/HO5vKjLn/IF8aC17dIP097ej1cFj3mj+mFO1ZsbrH8M2v2IBhN4MwuLry/tQLheBIT+hXjD+P7INdqQKU/Ki37+Me7cdvyTZh4Rgku7F2ALm4LfqgKIhCJY9KAU5BICli+4SA+2VGFP/3qdOTZGrq8KpUKiaSAz3+oQe8iOwodRiSSAjz1MeTbj3xkkxQopV7UpgM+3Pf29/h2vw8atQqb/nCRNBc9Yyw1mcTOxhTb87DZbJgxYwYeeeSRtMt4c+MhCERY8H9bEY5nz9we/To4semA/HO4vzxrKG5bvgmHfGEM6OjE9ed2Q498K4qdJhh1DXd3iD2eOy8+Fb87rzsAoCoQwWc7a3BJaREOeELItxvhMOngrY9h2Vf78Z91+1BeF5F+zzf3jILLokelP4oCu6HNPaZ4UsCGvV6c0dEJo06Dz3+oxruby3HVWZ1wWrEDRIRoQoBRp0EolsClj69FrlWPmy7ojrO65srSM2veA2TtxxeKYb8nhL4ljqz+G8gRO4ETkDwEQcDLL7+M1atXo6ioCLNnz0Z+fn7K5ej1esydOxcPPvigLPXaXR2Etz6Gv63aiS9/rJWlTPaz0lMc2Hywrs3LTx5wCl779mCrn9034TT84a3vpZ+fvnoQzuqWi5fW78MHWytw++hTcVqxA3ZjQ+9EpVKhOhDF4Ac+AgAM6pSDFTeejc53vdtq+fPH9MLZ3dwY/9ga6b1rhnWG1aDF9vIAbr2wB04rdrRYL5EUEE8STPqGRHnYF8YrXx/AuNIiFDtNOP0PKwEAbqseG+65sM1t0RgRwVMfQ641tQue4tf8SEGzvC6MP79XhhE93Jg04JSTbsieXVUB6DRqdMq1NHnfF4ohKVCb2uOaZ7/CxzuqcWqBDX+Y0Adnd3Mfr+qmJBJPIhxLIseib5ffJ1fsbNfkEQqFMGbMGKxduxYXXHABdu7ciZqaGqxatQpnnXVWm8tJJBLQ6XS47777sHDhwuNWXyLC9vIAookkLn1i7XH7Paz9FdqNqPBHjr1gG+x6YAz++M42vPDlPum9Uwts2FF59HmiZ5zTBU+v2QMAGNU7H49NG4B/ffYjuudbMap3AXQaFZ74ZDeWrNwBjVqFgR1z8NVeDwCgZ4EVq249FwCw7bAfnvoYXvhyL/4yuRRrd9di8yEfbh3VE0mB8M9Pd+P/rd4FAJg6pAN+d153HPaF8d0BH2YN74ofa+ox6uFPpXq5rQZcUlqEYDSBUb3zYTFoYTfqcMW/1mFIFxeev3YIAKAuFMcbGw+hT7EdnXMtSAqE61/cgO3lAXy78EKYdRrs84Tw9R4PJp5RDIO2Iak2732FY0kp4TZ3yBfGPa9vwcc7qgEAi8b3wW+HdQEArPjmIG5/dRMA4K2bhqFviQPxJEGvbf3xtZF/+wS7q3+eQe+JKwdgRM88aNUqlFUE8MZ3h3DrhT3hMOmkei14fQuuHdYFfX+avjqRFOCPJOD6KdC3pSf52c5qXPvc13jqqoEY2btAej8pEF75+oB0Ovo/152Jc3q0ntDK68L4cFslfnVGCezG9B9eljN2tmvyuOuuu7B06VJ88skn6Nu3L+LxOH7961/j8OHDWLduXZu7knV1dXA6nXjooYdw2223HedaN9h0wIccsx4dcxvOFUbiSTzx8S48unoX/nZZP/xYE8TjH+9ul7owdiKtuOEsDOrsOmLP7XibP6YXFr/f+h2Orfnq7pHIMevR4+7327zOb8/u3OKmlQ9vHYELH/lM+rl7vhW7fprO4bpzumBs3yJMfrLhILNrngWdXGZMHdIRs178RlpnysBTsOKb1nvUor4lDmw51NBL37LoIsQSAiY89gUO+X6e/e/SM0rwyOX927w9IjljZ7slDyJCcXExZs+ejQULFkjvf/755xgxYgS+//579OnTR3p/+/btKCsrg1qthsFggNPphNlsRrdu3VBVVYWuXbvimWeewbXXXtse1c9ITTAKnVoNrUaF/5VV4Zk1e7DpgA+TB5yCmSO64OOyatiMWlx0WgG+3F2LOS9vxGs3ni3tiIC8txczxpRvzbzzU35Aec+ePbLFzna7ZaWsrAwVFRUYN25ck/dPP/10AMCuXbuaJI/ly5dj0aJFLcqZO3cupk2bBgBwu0+Oc5bH0vghwQn9ijGhX3GTz3sV2qXXE/uXYGL/EgDA7j+PRVL4uRsuXmAGGpKxQD8PJ//NPi/mLt+Im87vjkkDToEv9PN58W/2eXD361vx1FUD0SnXgo0HfHCYdOica8bjH+/CQ6t2oovbgotPL8STn+xG/w5OeEMx7Ptp+PolU0qRZzPgt89+fcRtNOrUiMSFTJqJMZYC8a7HVHg8Dac95Yid7ZY8qqsbzlkWFBQ0ed/haDiX6PM1vfvnSEMG2+12+P1+6fUvmUatOuKFS5VKBU2jjwZ2ysGnd5wv/dz4AuLATi58cMsI6efGDwXedEEP3HRBD+nneRf3OmJ9fvzzWACAug0XUwWBQADUqoabEXItBny114MLeuVDp1EjlhAQTwrwhmJwWw2IJQXo1GqY9A13OvVZuBKDOuXg1RvOgkqlwvtbyrH4/TIM7uzCml3VWDZzKNwWAw75whj76OcY3sONv0wuRVIg/PbZr7C7uh63juqJzm4z5ry8EQBwx+hTMbZvEc5/6BOpngM6OvHt/iPfeXbZwFPwaqPTDB1dZuz3tJwThrH2Jl5DSoWcsbPdkkdOTg6AhsoXFhZK7x9pY3r37o2JEycimUwiGo3C6/UiGAwiPz8fdXUN5wPFxMPaR1uSRmvLds+3AQBGn/bz312vVUOvVcPy0/Ma4q28AGDWa7H3waY91DF9i1p9Et9h1rVY9n+3ndfkZ7EnJ2q+/LEsuaxfSssDgD8Sx3f7fRjUKQdmveaY1/NCsQQMWg3qY4k2XxD9sTqI2voYDFo1Sk9xojoQRSwpoMRpkpYJROKw6LWIJQXoNWp8f9iP+lgCvQptcJh0UKlUqApEYNRpYNJpoNOoUR9NwGLQIp4UoNOoUROMwqTTSH+rz3ZW47AvjO75VvQptiOWELBqWyUu7F2AaEJArlWPSn8E/nACFf4weuTbsLs6iN8++zUenNQXF5/esB/c/852vPbtQZSe4sDj0wbAbtRhxbcHcf872zBnZA9cdVYnfLazGuefmo+NB33o5DLju/0+/HVlGV6aMRSReBI9Cqz47/r9WPT2NuSYdXjxujNRFYhg3mtbMLyHG2d0cOLeNxvuzHviygEodBhh0Wsx+u8N1y5KnCa8f8twbNzvQ+8iOw54G27lfeDd7Xhu7V5cPqgDOrnN0inj6Wd1QvcCG+59YysK7UbMG3MqzHotqgJR3PvGVkwd0gEDOubg2/1e/PerAwCALm4LVCrgx58u1hfYDbh1VE8c9oXxQ1UQE/uXoNBhxPofa5tcy3l06hlNhkUCgOevHYJ4QsDwnun1HOSMne12zaOmpgZ5eXlYtWoVLrzw59sU165di2HDhmHPnj3o3Llzm8p69tlnce2116a0DmOMZTs5Y2e7DcnudrvRv39/vPHGG03ef+ONN5Cfn49OnTq1uaxgsOEOB6vVKmsdGWPsl0zO2Nmu83nMmDEDTz/9NJYuXYrKyko8+uijeOSRRzBr1qyUnvgMhxtuWTOZTMdYkjHGmEjO2NmuAwTdcMMNqK2txU033YRZs2ZBq9Vi1qxZuPfee1Mqp66uDhqNRpbxWRhjLFvIGTvbNXloNBosXLgQs2bNwu7du9GlSxcUFxcfe8VmAoEAbDZbVo9PwxhjqZIzdipyYMTp06fjs88+w549e050VRhjTDHi8TgAQKdLf4gTkSKTB3B85jBvL0SEuro61NbWoq6uDvX19airq4PX60VtbS0CgQCi0ShisRhisRji8ThCoRDq6+sRDocRi8WQSCRaPAujUqmg0Wig1Wqh1+uh0+mg1Wqh0+mg0+lgNpvhcrlgt9ths9ngcDhgsVjgdDrhcDhgNBphNBphsVjgcDhk2cFORolEAj6fD8FgEPX19fD7/VLbhsNhRCIRBINBBAIBhEIh6V8sFkM0GkUkEkE8HkcikZD+CYIAQRBaDEAotnvjtjUYDNDpdLBarXA4HHA4HLDb7bDb7dLr/Px8OBzKHf01EAjA4/Ggvr5e+hcKhRAIBBAIBKT2FV+LbRqJRBCNRhGPxxGLxZrs4yqVStq39Xo9TCYTbDab9K9x+zmdTjidTul1Tk7OL2J/jkajOHz4MLxeLzweDyorK6X9NxKJSPtqNBqV9mlxX00mkxAEAaWlpViyZEnGdVHkpAhz5szB1q1bYTKZ4HQ64XK5pGBoMplgtVqRk5Mj7UgulwsulwsWi0W2ec8FQUA4HEYgEIDf70coFILf74ff70cwGERlZSUqKytRUVGB2tpa6TOv14vy8nJEIkcflE+lUklfEvGLYrFYYDKZYDAYoNFooNE0PD+gUqlARNIzMYlEQko6iUQC8XhcSkA+nw+C0LYnwY1GI5xOJ3Jzc2G1WmGxWOByueB2u6UvZX5+PnJzc2GxWKQvr/ilNZlMsge/WCyG6upqeDweKfDU1taitrZWCkLBYBBerxd+vx91dXUIBAJSAAsGg6ipqWlzGwANFxdNJhP0ej0MBgOMRqOUmMV/arVa+gf8NALAT/tIZWWllJRCoZAUKGOx2FF/r16vR35+PvLy8pCfn4+ioiIUFBSgoKAAZrMZTqcTbrcbOTk5cLvdcDqdsFqtUh0yRUSIRqPSgYuYAMQDn/LyclRUVEj/V1RUwOPxSH+LtjAYDLBarTCZTNBqtTAajVJy1ev10j4ONBwwRiIR6aAqEolI3z/xQvDRmM1mWK1W2Gw2qU1zc3PhcrlgNpuRl5cHt9st7esOhwM5OTlSIpKjXYkIsVgMoVAIwWAQfr8f1dXV0nNsfr9f2ibxgLK8vBzV1dWoqqqSHrY+EvF6hsFgkOJF431Vo9EgFJLnIVdF9jzmzJmDDRs2IBKJwOPxwOfzIRAIHPGp9MZ0Oh0MBgP0ej3MZrN0VGgwGKTGVavVEAQByWRS+pLH43Ep+IgB4Fg0Gg3y8/ORn58vJTen04nCwkIUFRXB7XZLR/8OhwMulws5OTmw2+3QarXH5ahTEATpCNDn86G+vh4+nw91dXWIRCKIRCJST0g8evR4PNJRem1tLTweD/x+P6LR6FF/l0ajgcVikZKfGCDEnpBarZaSoPjFTCaTSCaTUgIU6xSLxRAMBtsUlMTAKh7V22w2mM1mWCwW2Gw26W9isVik98QvmvhPDDJGo1G2YNxcPB6H3++Hz+eTgkZdXR3q6upQWVmJqqoqVFVVoaamRgrQVVVV0qmH1qhUKilxiwFYp9NJ+7gYjNVqNVQqldRjisViCIfDUlATj1qPFR7UajXy8/NRXFyMwsJCuN1uuFwuFBcXIzc3V2p3i8UCs9ks9XqtViusVqtsvYFkMtnkYMHn80nt6vP54PV6pTgRCASkdq2urobP5ztmQBXb1WKxSO0qxhExOItnQhrvw9FoFNFoFOFwWOrttiXkarVaKV4UFBRIbVtSUoKSkhLpoKGgoAAOh0OKYzqdrt16q4pMHq0hIoRCIYTDYenIs66uDn6/HzU1NfB6vdKRk3hKSOziiV1lsWsnDrOs0WiafAHFHV7sBZjNZqnLLB552+12WK1W5OXlITdXngmEjiQej0OvbxgaWqPRIJFIHLff1ZpQKISqqiqpbcXA1zgYBoNBKTCJR9ziPzFBi20OQEoo4ukJ8XSPXq+H1WqFy+WSjhDFIJSTk4O8vDxYLJbjGuxTFY/HYbPZpGR04MCBjMsUBEE6TSGeuhB7Xo3bXzxdIR74iPu42NbiPzGRGAyGJolT3L/FfV38WdzPc3NzpSR8srR3JgRBQE1NjdRranwq2efzSQep9fX10v4rHtSIPXyxN9t4HzYYDDAYDNIBjdVqhdFolGKH2JYulwtWq1VKrqn22hOJBMaOHSudet2wYcPxairJLyZ5tFUikcCWLVukYJTKw4knG6/XC5fLBQCwWCxtPlVwMhMTSiKRUPxzPL+0v080GsUTTzwhXa/461//eqKrlBGfz4chQ4bA4XDA7Xbj/ffbPmT7yeZE7GuKvOaRiW3btmHAgAEAgF69emH79u0nuEbpa3wk26FDhxNYE3ls3rwZ/fo1jCOl9L8N8Mv7++zYsQNz584F0PD3UXry2L9/P3744QcADdujZCdiX1N+fzNFjc9tKn14k1/StgC8PSc73p6T14nYlqxLHo2HfhdH+lWqX9K2ALw9JzvenpPXidiWrEseje8QMhhSn0zlZPJL2haAt+dkx9tz8joR25J1yaPx/f1KfchQ9EvaFoC352TH23PyOhHbknUXzHv16oX77rsPyWRS8RfJfknbAvD2nOx4e05eJ2Jbsu5WXcYYY5nLutNWjDHGMsfJgzHGWMqyKnl4PB7ceuutGDRoECZOnIivvvrqRFdJkkgk8O9//xtTp07FzJkz8emnn7ZYZu3atRg3bhwGDRqEefPmwev1Nvk8HA7jgQcewJAhQzBq1Ci89957LcbR2bRpEyZNmoSBAwfipptuQmVl5XHdrtraWowbNw533HFHk/eJCK+++irOPfdcnHnmmfjHP/7RYqys6upqzJ49GwMHDsSkSZPw3XfftSh/5cqVGDVqFAYPHow//elPqK+vP27bEg6H8Zvf/AavvfZai89isRj+/ve/Y+jQoTj33HPx6quvtmj7nTt3YurUqRg4cCBmzJiB/fv3N/k8mUzimWeewbBhwzBs2DA8++yzbRqvLR3r1q3DHXfcgZtvvhmff/55i89rampw8803Y+DAgfjVr37V6nAXH374IS666CIMHjwYf/zjH1s81ez3+7FgwQIMHjwYY8eObXWfTpcgCHjwwQdxzTXXNHmfiPDuu+9i5MiRGDx4MB588MEWgyb6fD7ccccdGDRoEC655BKsXbu2RfmfffYZxowZg0GDBuGee+5BXV1dk8/r6+uxaNEiDB48GBdddBE+/PDDjLYnHA7jd7/7HRYvXtzisw8//BDXXnstfvOb3+DFF19ssU/s3bsX06dPx8CBA3HVVVdJDz6KiAjLli3D8OHDMXToUDz++OMtxkerqKjAjTfeiIEDB+Kyyy7D5s2bj11pyhL79u2jwsJCOuWUU+iuu+6isWPHEgB64403TnTVqKamhs455xwymUw0adIkGjhwIAGgZcuWScs8/vjjBIDGjBlD8+bNow4dOlC3bt2orq6OiIgCgQD17duXHA4HzZ07l6ZOnUpqtZqWLFkilbF8+XJSq9U0YsQImj9/PvXs2ZMKCgqovLz8uGyXIAh06aWXEgAaPnx4k89mzpxJKpWKrr76apozZw7Z7XYaPXo0CYJARES7du2i3Nxc6ty5M82fP58uvPBCUqlU9MEHH0hl3HPPPQSApkyZQrfffjvl5eXRoEGDKBaLyb4t0WiURo8eTW63m3744YcWnw0bNozMZjPNnj2brrnmGtLpdDRv3jxpmQ8//JB0Oh0NHjyYFixYQP369SO73U67du0iIqJkMkkTJkwgnU5H119/Pd1www1kMpnot7/9rezb8uSTT5JOp6Nf/epXNGnSJLLZbLRy5Urp8x9//JHy8vKoU6dOdNddd9Ho0aNJpVLRu+++Ky2zaNEiAkCTJ0+m22+/nfLz8+mMM86gaDRKRERVVVXUuXNnys/PpzvvvFPaD5577rmM619fX0+TJ08mAHTuuec2+ezOO+8kAHT55ZfTbbfdRrm5uXTWWWdRIpEgIqJDhw5RSUkJFRUV0bx58+iSSy4hALR8+XKpjL/97W8EgMaPH0933nknlZSUUO/evam+vp6IiLxeL/Xq1YtcLhfdfvvt9Otf/5pUKhU99thjaW3PoUOHpO/8nDlzpPfj8Thdf/31pFKpaNSoUXTxxRcTAPr9738vLfPll1+S0Wik0tJSWrBgAQ0ZMoSMRiNt3LiRiBq+g1dddRWp1Wq65ppraPbs2WS1WmnixIlSGdu3byen00ndunWj+fPn0/nnn08ajYY+/vjjo9Y7a5LH5ZdfTr1795aCLVFDAOvdu7cUsE6Up59+mgYMGEA7d+4kooZAct5559F5551HREQVFRVkNBrpvvvuk9bxeDyUk5NDDz/8MBER3XvvveRyuWjfvn3SMg899BDZ7Xby+/0UDAYpNzeXfve730nbGwqFqHPnzk2CnJyef/550uv1NGrUqCbJ47PPPiMA9Oabb0rvbdy4kQDQ6tWriYho/Pjx1L9/fwoGg9IyV155JQ0YMIAEQaBt27aRSqWif/3rX9Ln+/btI51OR//9739l35bp06dTQUEBbd26tcVn//jHP8hsNtO2bduk95577jnS6/VUXl5OiUSCOnfuTFdccYUUxOLxOPXr14+uu+46IiJ6+eWXSa1W09q1a6Uy3n//fQLQpNxMxWIxcrvd9Mgjj0jvXX/99XTWWWdJP1966aVUWlpKgUCgyfb369ePBEGgHTt2kFqtpieeeEL6/ODBg6TX6+nFF18kIqIbbriBOnbsSNXV1dIyd955J51yyikZJ/f/+7//o+LiYho8eDANGzZMen/Tpk0tEtTu3btJo9HQa6+9RkREV199NfXo0YO8Xq+0zE033UTdu3enRCJB+/fvJ51OR3/5y1+kz6urq8lqtUrbe8cdd1BBQQEdPnxYWub++++n3NxcCoVCKW/PwoULqV+/ftS5c+cmiWHjxo3UqVOnJgdMCxYsIJ1OR4IgkCAI1LdvXxo7dqzUpslkkkaMGEGTJk0iIqKVK1cSgCYHB19++SUBkPa1UaNG0Zlnntmk7pMnT27Stq3JiuQRDofJZDLR888/3+T9devWEQDasmXLCapZ6wRBoNLSUmkHeOaZZ8jhcEhHPqLrrruOhg4dSkREvXr1ogULFjT53OPxEABasWIFvfPOO6TVaqmioqLJMgsWLKAuXbrIvg379+8nu91Of/zjH2n27NlNksfs2bNpyJAhLdYZOHAg3XDDDeT3+0mr1TY5GiQiWr16NQGg3bt305/+9Cfq3LkzJZPJJsuMHTuWLr30Ulm3ZcuWLaTX6+nbb79t9fNzzjmHZs2a1eQ9cZ976qmnpP1MPDgQPfzww2Sz2UgQBJo8eXKTo0GihkBQXFxMf/zjH2Xblurq6haJ+5ZbbqHevXsTEVEwGCS9Xt8iAYsJf8eOHbR48WLq0KGDlAhFEyZMoPHjx1MymSS3201///vfm3y+Y8cOAkCfffZZxtuRTCZp+vTpTfare++9l3r06NHiYHDUqFF0+eWXUywWI5vN1uSAg4jou+++IwC0YcMGeuyxx8jtdlMkEmmyzJVXXknnnXceCYJAnTp1ovvvv7/J54cPHyYATXpnbSUmgn79+tHs2bOPuuyNN95IBQUFRERUVlZGAOirr75qssyzzz5LarWaotEozZgxg0aMGNGinD59+tCtt95KNTU1pFKp6O23327y+XvvvUcA6NChQ0esS1Zc8/juu+8QDocxYsSIJu937doVQMM5w5OFIAi45557sHnzZlx77bUAgDVr1mDw4MEtJq3v2rUr9u7di9raWpSVlbXYvpycHOTk5GDv3r1Ys2YNevfujYKCghZlHDhwQNbh3AVBwDXXXIMOHTpg3rx5LT5fs2ZNi7o23p6vvvoKiUQC5557bovPAUjbM3z48BbDgYtlyOmvf/0rzjjjDDz33HOYMWPSiQAAEhRJREFUOHEi7r//fhw+fBhAw7WqL7/8skVdjUYjSkpKpLoWFRWhR48eLeoaCATg9XqxZs2aFmWo1Wp06dJF1u3Jzc3FBRdcgGuvvRZPPPEEbrvtNjz22GOYMWMGAGDDhg2IxWJH/a588cUXOOecc1o8jCa2/c6dO1FTU9OijC5dukhlZEqtVsPj8UgjyQLAF198gREjRrQYylys15YtWxAIBI65bUOHDm3xlLZYxuHDh7Fv374WZRQWFsJoNKa1beKEbs23p7nXX38dS5culeLCF198AZPJhEGDBrWoqyAIOHDgQKv7VePtWbduHYgordiYFQ8JihfynE5nk/fFYNyWiZ3aQ3l5OaZPn47Vq1fjkUcewbhx4wA01L953YGG+ouTNwEtt6/xMkcro/FcBHJ47LHH8Mknn2D9+vXSfCONHa0u1dXV0t/L4XC0+ByAtD3Ng7G4jJx/z9raWixbtgxEJM3quHjxYjz11FPYunUrdDodkslki7o2rsvRtrfx9hytDLmoVCosXLgQ559/PmbPng1BENC1a1eMHj0aQNu+K4FAoNWRWxtvb2tliBOBybU9tbW1OP3006Wfj/U9acu2pVuGSqXK+G9VW1vb4uBOrNe8efPw6KOP4qqrrsJ9990HANI+0zxZtnW/EufbUavVsNlsRyzjSLKi55GbmwsALe5O8ng8TT4/kT7//HOUlpbi4MGDWLt2LW655Rbps9zc3BZ1Bxrqn5ube8TtIyJ4vV5pmSOVYbVaWw3y6Vi/fj1uu+02FBUV4emnn8acOXOwZs0a7NmzBw8//DCCweAxt8ftdre6PY3/XscqQy579uxBMpnEJ598gs8++wwfffQRNm/ejNraWqxYsUKawe1Yf58jfa5Wq6XpZNtje+LxOK677jqcf/752LVrFzZv3gyXy4URI0bA5/O16btyrLoeqQxxIiW5tsfj8SA/P1/6Od3vSSr71ZHKEAShSfulSpzut3ny2L9/P4YOHYr//Oc/eOmll/D8889Lsy+KdaVmd/Wl+rcSJxg7UhlHkhXJQzxKKisra/K+eOunOL/HiSLeznrOOefg22+/xZAhQ5p83rFjR+zYsaPFTvLdd99h0KBB0tzizbdv586dCIVCGDRoEDp27Ig9e/a0OJIQy5BLPB7HxIkTUVpait27d2Pjxo0oLy+Hx+PBK6+8gvLycnTs2LFFXYlIqkvHjh0BtP730mq1KC0tbbWM47E94jzjYjceALp3746ePXti586dUKlUrdalqqoKBw8exMCBA9GxY0dUVVVJX8jGde3Tpw/MZnOrZUSjUXz//fcYOHCgbNvz1ltvYd++fVixYgW6dOmCvn374qWXXoLH48GqVauO+l1Rq9Xo37//Mdu+sLAQWq32iN83uf4+4XC4yfDjx6pXSUkJVCrVEesl/q3KysqO+F1zuVywWCwtyti8eTMEQUh728TbiZsPp37ZZZchEolg06ZNmDZtWpNeRseOHRGNRrFv374WdS0pKUFhYaEUOxoTBAEbN2485nfNaDSiT58+R670Ma/m/EIMGTKErr76aulnQRBoypQpdNppp53AWjV45plnyGAwkMfjafVz8S6Szz//XHrv0KFDZDKZpIuSV111FQ0dOrTJxcIFCxaQ2WymaDRKhw8fJpVKRStWrJA+9/l85Ha7af78+cdpyxo0v2D+/PPPk8FgaHInjngx/JNPPiEior59+9INN9wgfS4IAo0dO5YGDx5MREQffPABAWhy2+yWLVtIpVLRK6+8IlvdDxw40OJulXg8Tm63mx599FEiIrr11lvp1FNPpXg8Li3z8MMPk0ajoZqaGgoEAmQ0Gumf//yn9Hk4HKYuXbrQzJkziYjowQcfpLy8vCZ3vLz00kvSRWq5PPbYY+RwOJrcaFBTU0MAaOnSpURE1L9/f5oxY4b0uSAINGHCBDrjjDOIiOijjz4iALR9+3ZpmW3btpFaraaXXnqJiIhGjx5N48ePb/K7Z86cScXFxbLd3dinTx9auHCh9PNbb71FKpWK9u7dK7337bffEgB6/fXXiYho+PDhdPnllzfZtiuvvJK6d+9ORETr168nAPT1119Ly+zZs4f0er3095syZYp08Vw0d+5ccjqdLW4iaKtoNNqknkQ/XxBftWpVq+vEYjHKycmhxYsXN3mvtLSUpkyZQkRETz31FFksliZ3l4kXw9evX0+CIFD37t3plltukT5PJpM0cuTIFrfXN5c1yePll18mADR79mx677336MorryQA9MILL5zoqtE999xDOTk5dPPNN9O0adNowoQJNHHiROk5D0EQaOTIkVRUVETPPPMMLV++nDp16kQFBQXSTvH111+TWq2myy67jN577z26/fbbCQDdfffd0u+ZNm0auVwueuyxx+j111+nPn36kM1ma3J77/Hw+9//vsmOGAgEqGPHjlRaWkorVqygJ598kmw2W5Pk9+9//5vUajXddttt9N5779GUKVOkO8eIfv6SdO/enZYtW0bPP/885eXlUc+ePaVnDeRy1lln0TnnnENVVVUUi8Xo7rvvJr1eTwcPHiSihruIDAYDjRkzht59911atGgRqdVquv7666Uybr75ZrJarfTQQw/R22+/TYMHDyaDwSDd+lteXk45OTl0zjnn0FtvvUUPPfQQ6f9/e/cfE2UdxwH8/dxx3PFLBYSUu24nJyJJGZbNws2oNlmSXPQHCfPGypi4WDhbs5a6lq3YXJsGy6GmMQ2cFqMtWG7VrKau9cPmWMrgzDiKE0Qx53UH97z7g90zrgMUO7R2n9fGH899H577Pgd3n+e57/f7+cTG0uFwRPRcghcir732Gj0eD3t6elhZWUmDwcCenh6SZGNjIxVF4caNG9nW1sbS0lICYHNzM8nR4JmXl8fMzEweOnSIjY2NTE9Pp91u12YpBYP7unXr2N7ezsrKSgLgzp07//U5XLx4kTt27KDVauWyZcu4d+9ekqMfwDk5OVywYAGbm5u5f/9+pqSk8J577tGmsra0tBAAN2zYwLa2NlZUVIQETlVVmZ+fz7vvvpsHDhxgU1MTzWYzzWYzr169SpL89ttvqSgKy8vL2d7ezpdeeokAuH379ls6n7Nnz7K2tpZ6vZ4Oh0ObsRV8DdevX0+n00mHw8GioiK+8cYbWvDfunUrTSYT33zzTX722WdcsWIF9Xo9T548SXL0AnHu3LlcsmQJW1paWFdXx4SEhJDgV19fT71ez82bN7OtrY2rV68mgLAZWP8UNcFDVVV+/PHHtFgsBECr1RqRBUuRcOzYMT744IMsKChgSUkJ165dy9LSUj733HPaPkNDQ6yurqZerycAFhUVhc3//+abb5ibm0sATElJYW1tbcjVsNfr5euvv06j0UgAfPTRR/n9999P+/nt2rWL5eXlIY/99ttvLC4uJgDGxMTwhRdeoMfj0dpVVWVTUxPnzJlDAMzMzAxZNEmOfog4nU4qikJFUbhmzZqQq85IOX36NLOzsxkbG0uj0cjZs2eH3d388MMPXLp0KQEwKSmJ27Zto9fr1dr9fj/feecdJiQkEAAfeuihsCmrZ8+eZUFBAQHQZDJx06ZNIeuSIqWuro6pqakEQACcN28eP/nkE61dVVUePnyYGRkZBECbzcaDBw+GHKO/v58VFRXU6XRUFIWlpaU8f/58yD7t7e202+0EwDlz5rC+vj4idx3B90teXh7z8vK0q2xydE3UmjVrqCgKdTod165dqwXFoNbWVlqtVgKgxWLhnj17Qvp1+fJlrl+/XnuvORyOsGnWX331FRcuXEgAnD17Nt99991bvuuoq6vjkiVLtPMJXvD19PRw+fLlzM/Pp8PhYHl5OZ1OJwsKCrT1TyMjI3zvvfc4c+ZMAuB9990Xsi6EHL1zCi6KNhgM3LBhAwcGBrR2VVX54YcfMj09nQCYlZXFI0eO3LDfUZdVlyS8Xi/i4uLCZin8H4yMjCAQCExa8OX69euTnl8gEMDw8DBMJtN0dfOm+Xw+6PV6xMSMP/HvZv5efr8fiqJoA4nTIRAI4NSpUxgeHsayZcsmfO28Xi+MRmPYFOIgVVXh8/kQFxc34XP99ddfMBgM01qXYWRkBL///juMRiPS09PHfW1v5rUfHh7WZqJN5Eb/j9PhRv8TN3NukXiv3S7B8/nndP6xIvFeGyvqgocQQoh/LypmWwkhhIgsCR5CCCGmTIKHEEKIKZPgIYQQYsokeAghhJiyqEiMKMRYJPHHH3/gypUrSEpKwty5cyecvhgpQ0NDiIuLi1gOMSHuNLnzEFFj9+7dyMnJQXJyMsxmMxYtWgSr1YrU1FTs3r17Wp/7mWeewdatW2/pd7/44ouw/Favvvqqll1ViDtB7jxE1Ojt7cWvv/6KV155BYsXL0ZKSgouX76Mn376KSTx4XRISkq65ToWzc3NIQnzSOLAgQPYsmVLhHonxNRJ8BBRIzU1FSaTKeyK/emnn57251YUZcJV5+P5888/cf78eZDEl19+icceeww///wzAMDlcqGvrw9msxkul2vaA58Q45HgIaKGz+ebNNUESVRXV6OmpgaJiYlwuVy4//77w1I+XLx4EWfOnEFubm5Y/QW/34+DBw+CJAoLC2E2m7XnDlaJ6+rqwrVr13DvvfdOmIJk48aN2Ldvn7btcrmwd+/ekH0cDgeSk5Ph8XimNTWLEOORMQ8RNa5evYrExET09fXh9OnTaG9vx/79+9HQ0IBAIIDBwUHU19fD6XTCbDYjPz8fOTk56Ojo0I7x9ttvw2az4YknnsD8+fPx6aefam1DQ0N45JFHUFNTgy1btsBut+P48eMARgMTSZSVlSErKwt5eXl4/vnnJ+zrnj174Pf7sWvXLlgsFvh8Pvj9fvj9fjz88MN4+eWX4fP50NfXJ4FD3Bm3lAZSiP8hp9OpZZId+5OWlsZLly5pNRUyMjJ49OhRnjlzhosWLWJZWRnJ0WyuALht2zZ2dnby8ccfp91u1zKyvvjii5w3bx4HBgaoqiqPHDlCl8tFkiwsLKTBYGBGRgZbW1u5fft2xsbG3jDL7JNPPsmqqipt+8qVK9Tr9WGZU4W43SR4iKhRVlbG3NxctrS08MSJE+zu7tZSWweZTCY2NDRo27W1tbRYLCTJkpISLl++XGs7deoUAfDChQscHBykyWQKKbY11sqVK6koCr/77juS5MmTJwkgpEhPUCAQYFdXFzs6Omgymfj++++zs7OTnZ2dbGhooKIo/PHHH9nd3R1S1EmI20nGPETUuH79Oux2OxwOx4T7JCQkaCVBASAtLU0rH9vV1YXCwkKtzWazARj9Oqy7uxskUVxcPO5xSaKoqAhLly4FMJruGwCuXbuGWbNmhezb1taGp556StuuqqoKO16wdHJraytWr1494fkIMV1kzENEDUVRoKrqpPsYjUbtgx0Azp07pw16z5gxA/39/VrbL7/8AkVRYLPZ4PV6Jz1+IBBAWlqatj1jxgwAgMfjCdt31apV6O3tRXFxMVatWgW32w23242enh6kpqZi8+bNcLvd6O3tDQkyQtxOEjxE1LBarXC73SEf8CTh8/m0bZ1Oh9bWVnzwwQd46623sHPnTlRUVAAASktLcejQITQ2NuLzzz9HVVUVnn32WSQmJqKgoADx8fGorKyEy+VCR0cH1q1bh48++gjAaJGohISEkL4AwIULF8L6qSgK0tPT8fXXX6OkpARmsxlmsxmDg4O4dOmS9lhGRsYdL0Ikotgd/tpMiNsmOOAdHx/P5ORkJiYm0mAwcObMmVoJUYvFwlmzZlFRFMbHx7Ompkarie73+7Wa1wC4YsUK9vX1acc/fvw4bTab1v7AAw9oNcoLCwvZ1NQU0p/MzEzu2LFj3L729/dzwYIFIeVCjx07xsWLF99yuVMhIkkqCYqocu7cOZw4cQIDAwMgCaPRiIULF2LlypUAAIvFgk2bNqG6uho6nW7chX0ejwdDQ0PIysoKu/IfGRlBd3c3dDod5s+fP+mdgdvtRkpKyqSlQ4X4r5IBcxFVsrOzkZ2dPWG7qqpQFGXSRIl33XVX2OLAoJiYmEmPP5bFYrmp/YT4L5IxDyHGCAYPIcTkJHgIMUZxcbE2DVYIMTEZ8xBCCDFlcuchhBBiyiR4CCGEmDIJHkIIIaZMgocQQogpk+AhhBBiyiR4CCGEmLK/ATPWGCbGvnsyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.xkcd();\n",
    "plt.xlabel('Epoch #');\n",
    "plt.ylabel('Loss');\n",
    "plt.plot(losses);\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 保存模型 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(cnn.state_dict(), \"fm-cnn3.pth\")\n",
    "# 加载用这个\n",
    "#cnn.load_state_dict(torch.load(\"fm-cnn3.pth\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型评估\n",
    "\n",
    "模型评估就是使用测试集对模型进行的评估，应该是添加到训练中进行了，这里为了方便说明直接在训练完成后评估了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "准确率: 90.0000 %\n"
     ]
    }
   ],
   "source": [
    "cnn.eval()\n",
    "correct = 0\n",
    "total = 0\n",
    "for images, labels in test_loader:\n",
    "    images = images.float().to(DEVICE)\n",
    "    outputs = cnn(images).cpu()\n",
    "    _, predicted = torch.max(outputs.data, 1)\n",
    "    total += labels.size(0)\n",
    "    correct += (predicted == labels).sum()\n",
    "print('准确率: %.4f %%' % (100 * correct / total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型评估的步骤如下：\n",
    "1. 将网络的模式改为eval。\n",
    "2. 将图片输入到网络中得到输出。\n",
    "3. 通过取出one-hot输出的最大值来得到输出的 标签。\n",
    "4. 统计正确的预测值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 进一步优化"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch : 1/20, Iter : 100/234,  Loss: 0.0096\n",
      "Epoch : 1/20, Iter : 200/234,  Loss: 0.0124\n",
      "Epoch : 2/20, Iter : 100/234,  Loss: 0.0031\n",
      "Epoch : 2/20, Iter : 200/234,  Loss: 0.0020\n",
      "Epoch : 3/20, Iter : 100/234,  Loss: 0.0013\n",
      "Epoch : 3/20, Iter : 200/234,  Loss: 0.0041\n",
      "Epoch : 4/20, Iter : 100/234,  Loss: 0.0016\n",
      "Epoch : 4/20, Iter : 200/234,  Loss: 0.0023\n",
      "Epoch : 5/20, Iter : 100/234,  Loss: 0.0010\n",
      "Epoch : 5/20, Iter : 200/234,  Loss: 0.0008\n",
      "Epoch : 6/20, Iter : 100/234,  Loss: 0.0017\n",
      "Epoch : 6/20, Iter : 200/234,  Loss: 0.0010\n",
      "Epoch : 7/20, Iter : 100/234,  Loss: 0.0009\n",
      "Epoch : 7/20, Iter : 200/234,  Loss: 0.0009\n",
      "Epoch : 8/20, Iter : 100/234,  Loss: 0.0005\n",
      "Epoch : 8/20, Iter : 200/234,  Loss: 0.0008\n",
      "Epoch : 9/20, Iter : 100/234,  Loss: 0.0005\n",
      "Epoch : 9/20, Iter : 200/234,  Loss: 0.0006\n",
      "Epoch : 10/20, Iter : 100/234,  Loss: 0.0016\n",
      "Epoch : 10/20, Iter : 200/234,  Loss: 0.0011\n",
      "Epoch : 11/20, Iter : 100/234,  Loss: 0.0003\n",
      "Epoch : 11/20, Iter : 200/234,  Loss: 0.0009\n",
      "Epoch : 12/20, Iter : 100/234,  Loss: 0.0010\n",
      "Epoch : 12/20, Iter : 200/234,  Loss: 0.0002\n",
      "Epoch : 13/20, Iter : 100/234,  Loss: 0.0004\n",
      "Epoch : 13/20, Iter : 200/234,  Loss: 0.0005\n",
      "Epoch : 14/20, Iter : 100/234,  Loss: 0.0003\n",
      "Epoch : 14/20, Iter : 200/234,  Loss: 0.0004\n",
      "Epoch : 15/20, Iter : 100/234,  Loss: 0.0002\n",
      "Epoch : 15/20, Iter : 200/234,  Loss: 0.0005\n",
      "Epoch : 16/20, Iter : 100/234,  Loss: 0.0002\n",
      "Epoch : 16/20, Iter : 200/234,  Loss: 0.0007\n",
      "Epoch : 17/20, Iter : 100/234,  Loss: 0.0003\n",
      "Epoch : 17/20, Iter : 200/234,  Loss: 0.0002\n",
      "Epoch : 18/20, Iter : 100/234,  Loss: 0.0004\n",
      "Epoch : 18/20, Iter : 200/234,  Loss: 0.0001\n",
      "Epoch : 19/20, Iter : 100/234,  Loss: 0.0003\n",
      "Epoch : 19/20, Iter : 200/234,  Loss: 0.0005\n",
      "Epoch : 20/20, Iter : 100/234,  Loss: 0.0002\n",
      "Epoch : 20/20, Iter : 200/234,  Loss: 0.0002\n",
      "Wall time: 2min 21s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#修改学习率和批次\n",
    "cnn.train()\n",
    "LEARNING_RATE=LEARNING_RATE / 10\n",
    "TOTAL_EPOCHS=20\n",
    "optimizer = torch.optim.Adam(cnn.parameters(), lr=0.001)\n",
    "losses = [];\n",
    "for epoch in range(TOTAL_EPOCHS):\n",
    "    for i, (images, labels) in enumerate(train_loader):\n",
    "        images = images.float().to(DEVICE)\n",
    "        labels = labels.to(DEVICE)\n",
    "        #清零\n",
    "        optimizer.zero_grad()\n",
    "        outputs = cnn(images)\n",
    "        #计算损失函数\n",
    "        #损失函数直接放到CPU中，因为还有其他的计算\n",
    "        loss = criterion(outputs, labels).cpu()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        losses.append(loss.data.item());\n",
    "        if (i+1) % 100 == 0:\n",
    "            print ('Epoch : %d/%d, Iter : %d/%d,  Loss: %.4f'%(epoch+1, TOTAL_EPOCHS, i+1, len(train_dataset)//BATCH_SIZE, loss.data.item()))\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可视化一下损失"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEVCAYAAADdFfNTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3hUZdo/8O/0mmnpIXRBAlJUuiAquGJBfMFC7LBrYRdldS3oiou4C/jD1V2wvApYWQv6Sl8RUIqKkSY9oSQQWpJJMr238/tjck5myJTMTAph7s915WI4OXPmmZOZ+z5POc/DYxiGASGEENLC+O1dAEIIIZcmSjCEEEJaBSUYQgghrYISDCGEkFYhbO8CkJbBMAzMZjPq6+thNptht9thNpthNBpRX18Pq9UKt9sNj8cDj8cDr9cLh8MBu90Op9MJj8cDn88Hv98fdlwejweBQAChUAixWAyRSAShUAiRSASRSAS5XA6dTgeVSoWMjAyo1WooFApoNBqo1WpIpVJIpVIoFAqo1WqIRKJ2OkOty+fzwWQywWazwW63w2KxcOfW6XTC5XLBZrPBarXC4XBwPx6PB263Gy6XC16vFz6fj/sJBAIIBAJgx+HweDwA4M576LmVSCQQiURQKpVQq9VQq9VQqVRQqVTc45ycHKjVau44HY3VaoXBYIDdbud+HA4HrFYrrFYrd37Zx+w5dblccLvd8Hq98Hg8YZ9xHo/HfbbFYjFkMhkyMjK4n9Dzp9FooNFouMdarfaS+Dy73W6cP38eRqMRBoMBNTU13OfX5XJxn1W32819ptnPqt/vRyAQwIABA7Bw4cImx6YE02DmzJk4dOgQZDIZNBoNdDodFzBlMhmUSiW0Wi33YdPpdNDpdFAoFBAKW+Y0BgIBOJ1OWK1WWCwWOBwOWCwWWCwW2Gw21NTUoKamBtXV1aivr+d+ZzQaUVVVBZfLFfP4PB6P+yKxXyaFQgGZTAaJRAKBQACBQAAejwcejweGYeD3++F2u+Hz+bjE5PP54PV6uSRlMpkQCASa9R6lUik0Gg0yMzOhVCqhUCig0+mQlZXFfXFzcnKQmZkJhULBfcHZL7ZMJmvxAOnxeFBbWwuDwcAFp/r6etTX13OBymazwWg0wmKxwGw2w2q1ckHOZrOhrq6u2ecAAGQyGWQyGcRiMSQSCaRSKZe82R8+n8/9AMGLCPYzUlNTwyUuh8PBBVOPxxPzdcViMXJycpCdnY2cnBzk5+cjNzcXubm5kMvl0Gg0yMrKglarRVZWFjQaDZRKJVeGVDEMA7fbzV3csEmCvTiqqqpCdXU19291dTUMBgP3t2gOiUQCpVIJmUwGoVAIqVTKJWCxWMx9xgHA7/fD5XJxF14ul4v7/jmdzrivJZfLoVQqkZGRwZ3TzMxM6HQ6yOVyZGdnIysri/usq9VqaLVaLlm1xHllGAYejwcOhwM2mw0WiwW1tbUwGo3c/9n3xF50VlVVoba2Fnq9HrW1tTGPLxAIIJfLIZFIuHgR+lkVCARwOBwRn8ujYcpBM2fOxO7du+FyuWAwGGAymWC1Wptc0UciEokgkUggFoshl8u5q0uJRML9Afh8PgKBAPx+PxcIvF4vF6DYIBGPQCBATk4OcnJyuASo0WiQl5eH/Px8ZGVlcbUItVoNnU4HrVYLlUoFoVDYKlevgUCAu5I0mUyw2+0wmUwwm81wuVxwuVxcjYq9CjUYDNzVfn19PQwGAywWC9xud9z3r1AouATJBhG2RsXn87lEyX55/X4//H4/lyTZMnk8HthstmYFLjb4srWDjIwMyOVyKBQKZGRkcH8ThULBbWO/jOwPG4ikUmmLBewLeb1eWCwWmEwmLrCYzWaYzWbU1NRAr9dDr9ejrq6OC+J6vR5erzfqMXk8Hpfc2SAtEom4zzgbsPl8Png8Hlfz8ng8cDqdXOBjr37jhRw+n4+cnBwUFBQgLy8PWVlZ0Ol0KCgoQGZmJnfeFQoF5HI5V3tWKpVQKpUtVqvw+/1hFxQmk4k7ryaTCUajkYsTVquVO6+1tbUwmUxRg+6F51WhUHDnlY0jbAAXCARcWdjPsNvthtvthtPp5GrNzQnjQqGQixe5ubncue3UqRM6derEXVjk5uZCrVZzcUwkEiUdNyjBxMAwDBwOB5xOJ3cFazabYbFYUFdXB6PRyF2Bsc1PbHWSrZaz1UiGYbjmptAvKfulYGsTcrmcq56zV/AqlQpKpRLZ2dnIzMzssE0czeFwOKDX67lzywbH0IBps9m44MVeubM/bBJnzzkALumwTSFs05JYLIZSqYROp+OuNNlApdVqkZ2dDYVC0aoJ4WIQCAS4JhG2mYStwYWef7ZphL04Yj/j7Llmf9hkI5FIwpIr+/lmP+vs/9nPeWZmJpeoL4XzHQgEUFdXx9W+QputTSYTdyFrt9u5zy974cO2FLC14tDPsEQigUQi4S56lEolpFIpFzvYc6nT6aBUKrkE3Bq1f1ZBQQEmTpyId999N2w7JZgQvXv3xpgxY7BkyZL2LgohhHQYPXr0wMiRI7F8+fKw7R3/MqEFicViGAyG9i4GIYR0KHK5PGKfFSWYEDKZrFkde4QQQhpFi52UYEKIxeK4ncyEEELCRYudlGBCsCO9CCGENF+02EkJJgSfz2/WcD9CCCGNosVOSjAh2CGWyT63vNYGn59qQISQ9BItdlKCCREIBJJKMF5/AH/8z16M/ec2vLn5WCuUjBBCLl7RYiclmBDJ1mC+2Hka3x6qBgBU1NpbuliEEHJRuyhrMDabDatWrcLatWubNXrLbDbj22+/bbKdYRj8+OOPWLFiBc6ePZt0efx+Pzc1QyKO6xunGslUipN+fUII6Yiixc52SzBr1qxB7969MWnSJEycOBF9+vTBjh07ou5fWlqKIUOG4KGHHgrbfvr0aVx33XW49tprce+99+Kyyy7Da6+9llSZ3G43JBJJws8zORrncdLJKcEQQtJLtNjZLgnm6NGjuPvuuzF+/HgYjUbU1tZi8ODBuPvuuyPOBlteXo5hw4ahsrISUqmU2x4IBHDXXXehvr4e+/fvh9vtxqJFizBr1ixs27Yt4XK5XK6w4zeXw9M4IaZCQhNUE0LSS7TY2S4JZvHixejevTuWLFkCtVqNzMxMLFq0CNXV1Vi1alWT/TMzM7Fw4UK8+OKLYQnoxx9/xM6dO/Gf//wHAwYMgEAgwKOPPorrr78eixcvTrhcXq83qZlY3b7GBCOnBEMISTPRYme7JJiNGzfivvvuC2uzy8/PR69evbBv374m+2s0Gjz22GNwu91QKpXc9k2bNqF///4YOHBg2P6jR4/G/v37mxyntLQUK1euxOrVq7FhwwaUlJTgwIEDMJvNAILrgojFiTdxOUNqMEpJ4n04hBDSkUWLnW1+uc0wDMrLy9G9e/cmv8vKykJ1dXXU59bX1yMnJ4f7f3l5OXr06BHxOFVVVU22r1ixAnPmzGmyvXfv3jh69GjSNRhPyL0vUiElGEJIerloajA8Hg8ymSxiX4vT6YzZB1JXV4e8vDzu/4keJ9riYTKZjHse+zgRXn/jHaxSMSUYQkh6iRY726XDIDc3FzU1NWHbGIbBmTNn8MADD0R9ntlsRrdu3bj/5+TkRGxSO336NPr06dNke1FRESZOnMgtA8wuKVpQUMAtuqTRaBJ+P25vY+JSSTv+Gt2EENJcsWJnuySYYcOGYfPmzZg1axa37cCBA9Dr9Rg2bFjU57FrxYceZ+HChaitrUV2djaAYKLavHkzbrnllibPLy4uRnFxccRjWywWMAwDtVqd8PtxhSSYfHXio9AIIaSjYpdsjhQ726WTv7i4GFu2bMGaNWsABJu+ZsyYgcLCQlx11VURn8MwDHQ6HWpra7lmsZtuugkqlQovvvgi3G43AoEAXnnlFRw9ehT/8z//k1CZTCYTACSVYLyBxiYypZRGkRFC0kes2Nku0fC2227D9OnTMXHiRPTt25frkF+xYgU3EmHEiBHo168fli5dit9++w2jRo2Cw+EAEBxVVl9fD7lcjo8++ggPPfQQ1q9fD5lMhpMnT+K5557D6NGjEypTXV0dgOCQ6ER5fI2d/AoxJRhCSPqIFTvbJRryeDy89dZbKC4uxubNm6HVavHggw+GteHp9XooFAoAQL9+/bB+/Xr4fD4wDAOVSsV1KE2cOBFlZWX49NNP4XK5cMcdd6B///4Jl8loNAJILsGYnY138gv4yc3GTAghHVGs2Nmul9vXXHMNrrnmmoi/Ky8v5x6LxWJcd911UY+Tl5eHZ599NqWysFlYp9MlfQwVNY8RQtJMrNhJsyk3YNsRtVpt0segaWIIIekmVuykBNOA7d9hm+WSIRPRPTCEkPQSK3ZSgmlQU1MDkUgElUqV9DEyqImMEJJmYsVOSjANampqkJOTAz4/+VOioan6CSFpJlbspATToKqqKmwammRQDYYQkm5ixU5KMA30ej3y8/NTOgY/ieWWCSGkI4sVOynBNKitrUVWVlZKx2Di70IIIZeUWLGTEgyC09Do9fqwpQCSQfUXQkg6iRc7KcEgOEuzx+NJOcEI6S5+QkgaiRc7KcEg2IYIBJcRSIWfoUYyQkj6iBc7KcEgOFU/kNxMyqFCJ70khJBLXbzYSQkGwWoekHqCcXojr5hJCCGXonixkxIMGrNwRkZGSsfx+amJjBCSPuLFTkowaDxJqUwTAwAMDVQmhKSReLGTEgwaq3mR1pROBPXxE0LSSbzYSQkGjScp1RqMmzr5CSFpJF7spAQDwGazQSwWQyQSpXQcS8jKloQQcqmLFzspwQDwer0pJxcAsLgowRBC0ke82EkJBoDb7YZUKk35OE4PDVMmhKSPeLGTEgwAu90OuVye8nEsLl8LlIYQQjqGeLGTEgwAl8uVUg1GIW5cKtkfoKFkhJD0EC92UoJB8CTJZLKkny8UNJ5GK/XDEELSRLzYSQkGgMPhSCnBCEJmUa6xuFuiSIQQctGLFzspwSD1UWShs/RXmZ0tUCJCCLn40SiyZuLzkz8VoTUYK3X0E0LSSKzY2S4Jxmq1Yv/+/airq2vW/jabDfv370dtbW3UfRiGAZPkXC3JPo8lFjaeRhqqTAhJF/FiZ5smGIZhsHjxYnTp0gWDBg1CYWEhZs2aBb8/elB+77330LVrV27/v/zlL/D5GmsJdrsdM2fORE5ODpRKJaZOnRr2+7YgCsngdLMlIYQEtWmCWbJkCZ566ik89dRTOH78ON577z28/fbbePPNNyPu/8knn2D69On44x//iOPHj+ODDz7AsmXLsGDBAm6fJ598Et9++y2WLFmC//u//0NJSQn+9Kc/JVQuHo+HQCD5ecRCazAuWhOGEJIm4sVOYVsVJBAIYM6cOZg5cyZefvllAMBll12G48eP480338Sf//xnCIWNxWEYBn/7298wffp0vPrqq9z+J0+exKJFi/Dss8+Cz+fj008/xQcffIA77rgDAHDw4EG8/fbbCZWNz+enlGAUksZyu7w04SUhJD3Ei51tVoM5ePAgqqqq8Nhjj4VtHz9+PM6fP48zZ86EbT927BhOnToVcf/a2lpUVFSAz+dDIpFg9+7dYBgGXq8XP/74Y9TV1UpLS7Fy5UqsXr0aGzZsQElJCRiGSTnByESNN1ra3NTJTwhJD/FiZ5vVYA4dOgShUIiePXuGbc/NzQUAnDt3Dt27dw/bHwAuv/zyqPsXFRVh/vz5eOKJJ7Bt2zbo9XowDINvvvkmYhlWrFiBOXPmhG1zOBwQCoUp9duoZI2nsc5G98EQQtJDvNjZZjUYoVCIQCAAHo8Xtp0tXGjzWOj/4+0vlUohEAigUqnQrVs3VFdXY+3atRFHN0QaTODz+VJOMGqZmHtM85ERQtJFvNjZZjWYnJwcBAIBmEwm6HQ6bntVVRUAhNVe2P0BoK6uDgUFBRH3r6urw9NPP42///3veP755wEAb7zxBp555hnceuutGDlyZNgxi4qKMHHiRPj9frjdbhiNRgQCgZQTTGgTmYOayAghaeKiqcFcccUV4PP5+PHHH8O2b9u2DV27duWavlhFRUUQCoUR98/NzUWXLl2wefNmOBwOPPPMM+DxeODxeJgxYwZ4PB527drVpAzFxcVYtWoV1q5di40bN2LXrl1Qq9UQiUTwepMfXiwPmezS5aNRZISQ9BAvdrZZgsnOzsaYMWPw73//G05ncDqVo0eP4n//938xfvz4JvtrNBrceOONWLx4MRwOBwCgvLwcb731FsaPHw8ej8c1u+n1eu55VVVVYBgm6hrRkUilUrhcrqTfm0wcWoOhBEMISQ/xYmeb3gfz+uuv48CBA+jXrx/uvPNODBkyBBKJhBu2bDKZMHToUJw8eRIA8Nprr+H48eMoKirCXXfdhauvvhoMw2Du3LkAgNtuuw2FhYW49tprMW/ePLz66qv43e9+h65du+Kuu+5qdrkkEgnc7uQ75yUh98HQKDJCSLqIFzvbNMFcddVVKCsrw1133QWhUIjZs2ejtLSU62M5dOgQdu3ahe+++w4A0L9/f5SVleG+++6DQCDArFmzcPToUXTp0gVAMHuWlJRg8uTJ2LJlC3bs2IEpU6Zg9+7dCS0gJhaL4fF4kn5foffBuH10HwwhJD3Ei51t1snPysrKwmuvvRbxd6NGjYLb7YZY3DgqS6vVYt68eVGPV1BQEPV4zSWXy7lmu2ToFI3l9VCCIYSkiXix86KbTTk0ubQV9iQle7NlToaEe+z0+mlVS0JIWogXOy+6BNMe2Oa0ZDv689XhC+54/VSLIYRc+uLFTkowADIyMgAElxFIhlYRvuCOj2owhJA0EC92UoIBoFQqAQTXnUmGRCgIG0lG/TCEkHQQL3ZSgkFwNBqAlDr61bLGWowvhYkzCSGko4gXOynBAJDJgn0oqSQYpTR0qYGUi0QIIRe9eLGTEgxaJsGE1mB4MfYjhJBLBSWYZlAoFACCyy8nK3So8oUzQBNCyKUoXuykBANApVIBSH4UGQBo5Y3371B+IYSkg3ixkxIMWr4GIxLQaSWEXPqoBtMM7FC7VBKMJqQGE7o+DCGEXKrixU5KMAguDcDn88Om/U+UMmTCS5GA2sgIIZe+eLGTEgyCq7JlZWWllGBCh45RJz8hJB3Ei52UYBoolcqUOvlB974QQtJQrNhJCaaBQqFIqQ9GSM1ihJA0FCt2UoJpoFAouKWZkyGljn1CSBqKFTspwTTIyMhIqYmMTxUYQkgaihU7KcE0UKvVMJvNST+fTx37hJA0FCt2UoJpoFKpUkownpBFxhia7ZIQkiZixU5KMA20Wi1MJlPSzzc6vNxjr58SDCEkPcSKnZRgGiiVSjgcjqhrS8djdTUmGD+taEkISROxYiclmAbswjnR1paOx+nxc48D1ERGCEkTsWInJZgGqc5HZgppIiOEkHQRK3ZSgmmQmZkJAKitrU3q+XaPryWLQwghHUKs2EkJpgF7koxGY1LPN9o93GMaskwISRexYme7JRi3253QcF632x23A97r9SY9RJit5tlstqSeX2drTDA0bQwhJF3Eip1tnmB27tyJkSNHQiqVIjMzE6+//jr8fn/U/ffu3Ytrr70WUqkUOp0O8+fPh8/XtDnqyJEj6NatG77++uukypWRkQEg+VUtz5sa16QW0m39hJA0ESt2CptsaUV79+7F6NGjMWbMGKxZswZHjhzBnDlz4HK58NJLLzXZ/+DBgxg1ahSGDx+O1atX49ixY5g7dy7sdjv+/ve/c/tVVFRg7NixKCoqwoQJE5Iqm06nAwDU1dUl9fz6kCYymq6fEJIuYsXONk0wL7zwAoYPH44NGzaAz+djwoQJEAgEmDdvHp566ilu+U3WSy+9hAEDBmDTpk0QCIKTSUqlUrz44ot49tlnoVar4fP5cMstt6B///5YtWoVN2QuUdnZ2QCS6+T3hdzFT6tZEkLSSazY2WZNZDabDVu2bMHMmTPB5ze+7KRJk2A0GvHbb7+F7e9yubBp0yY8+eSTXHJh97dardi1axcA4KuvvoLdbsfXX38NuVweswylpaVYuXIlVq9ejQ0bNqCkpAQHDhyAxWKBWCyGUqmEwWBI+L2F9r9IRDRughCSPmLFzjarwRw6dAherxdXXnll2PaCggIAwOnTp8O2l5WVwel0Ntk/NzcXfD4fp0+fBsMwWLBgAXJycjB69GicO3cOI0eOxEsvvYShQ4c2KcOKFSswZ86cJtt79+6No0ePQqlUJtXJX2t1c4+lQqrBEELSS7TY2WaX2+wIMLFYHLad7a+4sN+iOfsfOHAABw4cgMFgwIQJE/D444+jrKwM48aNi5hNow0m8Hg83GuxjxNhcDQ+Ry6hBEMISS/RYmdCNRiv1wuXy8WNGkhETk4OgGBHUKdOnbjtbLsdW5OJtH/Pnj257QaDAYFAAAUFBVyt59ChQ1z/zcyZM9GpUyesWbMGDz/8cNgxi4qKMHHiRPj9frjdbhiNRthsNu7ESKXSpKaKcYbcZCkXU4IhhKSXaLGz2TWY1157DQqFAoMGDQIAnDt3DsXFxejWrRuee+65uPeoFBYWQiaTYc+ePWHbf/nlF/D5fFx99dVh23Nzc6FSqSLuDwBDhgzhajeh975kZ2cjOzsbNTU1TcpQXFyMVatWYe3atdi4cSN27dqF0tJSlJeXA0g+wdjcjTUjCTWREULSTMoJRqFQwOv1YtWqVQCABx98EFu2bMFdd92FhQsXYvXq1XELcPPNN2Pp0qXcfSwOhwP/+te/MGTIEO5mHZZIJMJtt92GDz74AF5vcJ4vl8uFN954AwMHDoROp0OvXr0AADt27OCed/78eVRXV6N///7NfWucZJvIQmswEiF18hNC0ku02NnsaDhp0iQAwIkTJ1BRUYEffvgB77//PhYuXIi+ffvi559/jnuMl19+GQcOHMDQoUMxe/ZsjBgxAiUlJZg/fz6AYE1k5syZqKysBBAcpnzs2DEMHjwYs2fPxsiRI7Ft2zYsWLAAANCjRw/ccMMNePLJJ/HDDz9gx44dmDJlCrp164Ybb7yxuW+NIxQKI97EGY/FRU1khJD0FS12NjvBFBQUYPjw4Xj11Vfx9ddfQygU4vrrrwcQrG243e44RwAGDhyIw4cPo2/fvli3bh2Kioqwb98+7jhnzpzBokWL8MILLwAI9pkcOXIEgwYNwrp169CzZ0/s3bsX48eP5475ySefYNCgQRg3bhxGjx6NrKwsbNy4ESKRqLlvjSMQCGLOKhCN3d14YpWSNr21iBBC2l202JlQNJw/fz7uuOMOPP/885g0aRIyMjJQWVmJI0eO4Mknn2zWMbp27Yrly5dH/F2XLl2wf/9+9OnTh9tWWFiIjz/+OOrxOnXqhC+++AKffPIJeDxeUomFJRAIklpwzOGhPhhCSPqKFjsTSjDXXXcdjh49itLSUgwbNgxAsMnsxhtvxL333tsiBR0wYEBSz7twOHNbCl1sTEo3WhJCCIAEEozH40FZWRkCgQCuu+46AMF7VQYPHox169ZdEvNvBQIBCIWJN3FZQpZL1sjbL9ERQkh7iBY7m325/eKLL2LgwIGYMmUKAGDPnj3o378/NBoNxo8fD6fTGecIFz+/3x82LU1zOb2NNRi1LPkmOkII6Yiixc5mJxj2/petW7eCYRg89NBDEIvF+PTTT/HDDz/giy++aLnStpNkE8zpegf3mEaREULSTcoJZsKECRAKhVi/fj0OHjyIw4cPY9GiRbj//vtxxRVXNJmssiMKBAJhE3E2V0Vd41rUMkowhJA0Ey12NrvDQa1W46abbsKcOXNwyy23QC6XcxNKut3upPouLjZerzelUWgATddPCEk/0WJnQpfrb7zxBtRqNd5//31Mnz4dEokEe/bsQVlZGUaPHt1ihW0vLZFg6D4YQki6iRY7E4qGvXv3xv79+2GxWKDVaoMHEArxwgsvYOLEiS1T0nbk8/lSTjB56uQWPCOEkI4qWuxM+HKbz+ejqqoKJSUl0Ol06N+/P/7xj3+0SCHbm9PpTHhFTH+ACft/jooSDCEkvUSLnQk1kVVUVOCqq65Cv379MGHCBIwcORJdu3bFkiVLWqyg7cnpdEImkyX0HJsrfP4dBXXyE0LSTLTYmVCC+fOf/wyTyYTvvvsOXq8XVqsVTz/9NB599FHs3LmzxQrbXjweT8IzAtg84QnGF2AQuKBWQwghl7JosTOhBHPkyBFMnjwZv/vd78Dj8SCXy/HCCy+gV69e+Oyzz1qssO2BYRjY7fYmywbE4/KGT/B21dxNeHrFvpYsGiGEXLRixc6EEszYsWPx2Wef4dy5c9y26upq1NTUhK1S2RE5nU74/f6EV+t0uMMTjNXtww9l+pYsGiGEXLRixc6EEszLL78MiUSCnj174tZbb8U999yDK664AjqdDo888kiLFbg9WCwWAIBKpUrseSHzkLFoPjJCSLqIFTsTSjCdOnXCoUOHsHDhQqhUKthsNjz//PPYvXs3NBpNy5S2nZhMJgBI+H1YXU0X2bmw2YwQQi5VsWJnwsOUFQoFnnjiCTzxxBMAgk1kJSUluPXWW1MsZvsym80AgjMWJKKiztZkm9ef+JoyhBDSEcWKnSkvXvLtt99iwoQJOH/+fKqHaldsNS/hBFNrb7LN7aMEQwhJD7FiZ8oJZvTo0WAYBps2bUr1UO3Kbg8mCoVCkdDzLM6mfTBUgyGEpItYsTPlBNOzZ0907twZu3fvTvVQ7aq+vh4AuClwmssVobbi9dO9MISQ9BArdqacYHg8Hvr164djx46leqh2pdcHhxbn5uYm9DxvlOYwP0MJhhBy6YsVO2N28jscDqxfvx4Mw0AgEEAgEEAikUAul0OhUECpVCI7Oxs5OTk4cOBA65S+jZhMJkgkkoSniglESSSx8sunJZUAw+D+4V0viaWmCSHpK1bsjJlg9uzZg3vuuQdMM67Gx4wZk3wJLwIWiyXhe2AAINqZYaL8JhBgMHvVIQDAWaMTL9xSlPBrEkLIxSJW7IyZYEaPHg2v1wufzwe/3w+/3w+32w2HwwGHwwGbzQaz2Qy73Y7hw4e3SuHbSl1dHXQ6XcLPi5Z8eYhcM+HzG7e/t70Cz950OeiuIUoAACAASURBVISClFsqCSGkXcSKnXHvg2GbxliJTqXSURgMBmRmZib8vGiVO36Mlq8spRh1Ng8AwOn1I4MSDCGkg4oVOymyNbDb7QkPUQYAoSByJhHEyDChU8mYIwxzJoSQjiJW7KQE08BmsyU8k3IssTrvdSEJxuGhaWUIIR1XrNjZ5gnm3LlzmDp1Krp164bhw4dj7dq1Mfevrq7GI488gu7du2Po0KH45ptvovZ7fP/99+jWrRuWL1+ecLnq6+uT6oMpqTA0e1+23CpZ49KiTkowhJAOLFbsbNMEc/LkSQwYMAA7duzAjBkz0LdvX9xxxx1YtmxZxP3Pnj2LAQMG4IcffsAf//hHDBw4EHfffTfeeeedJvvq9XoUFxejsrISFRUVCZfNZDIllWCaq8rsxOC/b8ad7+5AaZWF2x5pNmZCCOkoYsXOhCe7TMWsWbNQUFCAnTt3cmOmc3NzMWfOHDz44IMQiURh+//1r3+FVqvF3r17uTa+wsJCzJ07F7///e+5NaAZhsFjjz2Gzp07JzWrs9frhcvlatUBDMdrbKi3e1Bv94RtpyYyQkhHFS92tlkNxu12Y926dXjmmWfCbsh54IEHcPbsWezZsydsf5/PhzVr1uDpp58O60B64IEHoNfrUVJSwm379NNPsXbtWixduhRCYfScWVpaipUrV2L16tXYsGEDSkpKUFlZmfRMyomIdr8MTe1PCOmo4sXONqvBHDx4EA6HAyNGjAjb3rVrVwBAZWVl2L00ZWVlMJlMTfbv3Lkztz8AnD59Gk888QSeeeYZXHnllTHLsGLFCsyZMyds24MPPoi5c+cCSHyiy5bgoZmXCSEdVLxJgtusBuN0OgGgyWgDsTg4osrn8zVrf6FQCKFQCJ/PB5/Ph6lTp6KwsLBJ4ojE729aW5BKpXC5XNzj1hJtTJmHZl4mhHRQ8WJnm9VgsrOzAQRvyikoKOC2GwyGsN9H2r9Hjx7cdovFAp/Ph+zsbMydOxc//PADBgwYgIceeghisRjnzp3DmjVroFKpMHPmzLDhwkVFRZg4cSI3I4HRaETXrl3bJMHwowxbplFkhJCO6qJJMIWFhRAKhTh8+DCuuOIKbjs7zf/gwYPD9s/Pz4dEIsHhw4fDfsfuP2TIEDgcDvz+97+H2+2G1+uF3W6H3++HwWDA3r17m5ShuLgYxcXFTbZv374dQOv2wfCj1BW9fpp1mRDSMV00fTBKpRI33HADli9fjnvuuQcAEAgEsHTpUvTt27fJMDeJRIKbbroJy5cvx4MPPggejweGYbBkyRL06NED+fn5mDJlCqZMmRL2vL59+2LKlCl4+eWXm122WGtKt5RoNRgfNZERQjqoeLGzTYcpP/fccxg3bhzuvvtu3H777fjqq6+wZs0afP7559w+n3/+OW655Rao1Wo8++yzuPbaazFp0iRMnjwZK1euxDfffIMPP/ww6msEAokH7GRXs4xHb3EhUymBgM+DMMrUMbRuDCGko7poOvkBYOzYsfjpp59QUVGBRx99FKdPn8bKlSu5Wsjp06dx77334rHHHgMAjBo1Cr/88gvOnz+PRx99FMePH8eKFSvw8MMPR32NUaNGoVevXgmVi63mtWQN5sOfT2LovO/x8urg1PwSoSDifpRfCCEdVbzY2aY1GAC45pproi6v3KVLFyxbtgy33XYbt23YsGH49ddfm338pUuXJlwmq9UKoGVnin5l7REAwNajtQAAuSRygqH7YAghHVW82NnmCSaeadOmtflrWiwW8Pl8yOXyFj92J03wplKZKHKCoaliCCEdVbzYSbMpIzgUWqPRgB9tqFcK3L5gDUUaJcHQMGVCSEcVL3ZSggHgcDhapfYCADZ38AZSiTDyqXZ5aRQZIaRjihc7KcEgOGHbhRNtNpdCHLlmwjptcAAA5GJBxEXIfEmMeiOEkItBvNhJCQapJZjQ1SkjHrvhRkoejxcxGdGNloSQjooSTDP4fL6YszDHkpUhafa+GdKmfwh/gBIMIaRjihc7KcEgtRpMliJ2DQYAnvlqP4BgM9mFaLJLQkhHRTWYZvB4PNyszonqrIs/OODrPWfh8voRiHBXpZum6yeEdFDxYiclGKTWRCaOMjrsQna3L2J/iyjKFDKEEHKxoyayZvD7/RAIYo8Gi4YdhhxPtKWRo02CSQghF7t4sZMSDACGYZK+yfK8ydms/bz+AHXoE0IuKfFiJyWYBrwkaxJ6i7tZ+5VUGKK8blIvSwghF4VYsfOim4usvTBJTmtcZW5eDebFlQcjbhcJKMcTQjquWLGToluDZBOMxdW8PphoVDLK8YSQjosSTBwCgQB+f3KTTqbaryIXU4IhhHRM8WInJRgAQqEw6QSTKrUsuRs8CSGkvcWLnZRgAIjFYrjdzeusb2mqCNPHEEJIRxAvdlKCASCTyeB0Nq+z/kLNvdEyGq2CEgwhpGOKFzspwQBQKBSw2+1JPdeT4lQvNIqMENJRxYudFN0AyOXypGswqaIEQwjpqOLFTopuAEQiETweT7u8drSVLgkh5GIXL3ZSdEOwo6q9EszPJ+pQXmtrl9cmhJBUxIudlGDQeJKSvdkyFV/sOsOtF0MIIR1JvNhJCQaARCIBwzDw+VK7Kz9ZB86a2+V1CSEkFfFiJyUYABkZGQAAi8XSLq+vldNQZUJIxxMvdlKCAZCZmQkAMBqN7fL6dnf7zCJACCGpiBc72zzBOJ1OLFiwADfddBMeeughHDlyJOb+brcbr7/+OsaPH48HHngABw4caLJPTU0NFixYgOnTp+Odd95J+J4WrVYLADAYIk+p39qcXn+79P8QQkgq4sXONk0wBoMBV155JRYsWIAuXbrgxIkT6N+/P/773/9G3N9sNmPw4MGYO3cuCgsLUVlZiUGDBmHlypXcPhs3bkSfPn2wePFinDhxAk8//TRGjhyZ0KgwtVrNvV57cad4wyYhhLS1eLGzTafy/etf/wq73Y4jR46goKAADMPgD3/4A5555hncfPPNTRaumTNnDurq6nD48GF07twZDMNgxowZePbZZ3H77bdDIBBg+vTpuPfee/HGG29AIpFg+/btGDNmDL7//nvcfPPNzSqXQqEAgKTv5m8JHn8AUlFyyzYTQkh7iBc72yzB+P1+fPnll5gzZw4KCgoABFdCmz59Oj744APs27cPV155Jbc/wzD44osv8NRTT6Fz585h+7/zzjvYuXMnRowYgWPHjoWtCc02NSmVyiZlKC0tRVlZGfh8PiQSCTQaDQoLCy+KGoyXajCEkA7moqnBHD58GEajEWPHjg3b3rt3bwDAyZMnwxLM8ePHUV1d3WT/Xr16cfuPGDEiLLnU1NTgySefRI8ePTB8+PAmZVixYgXmzJkTtm3ixIn46KOPAAB1dXVJv79Uef3UB0MI6VjYTv5osbPN+mBMJlNYgVjRqljR9heLxRCJRE3237hxIwYOHAiDwYDVq1dDJGo69DfSugV6vR5qtRpSqRRVVVUJvquW4wtQDYYQ0rHEi51tVoMJHc6Wl5fHbWcTCTsaIdL+3bp147Y7HA54vV5u/0AggNmzZ2PevHm4//77sWjRoibHYhUVFWHixInw+/1wu90wGo2QyWTg8XjIz89HdXV1i73fRFF+IYR0NPFiZ5slGLbfpby8HEVFRdx2dtjx1VdfHbZ/Xl4e+Hw+ysvLw5rOLtz/rbfewvz58/HZZ5+huLg4ZhmKi4uj7qPVarlk1x4CNEyZENIBxYqdbdZEptVqMWLECHz11Vdh27/88kt06dIF+fn5YdsVCgXGjBkTcf+cnByuVrNs2TJMnTo1bnKJR6VSJdXJz+fF36c5eC10HEIIaUuxYmebDlOeMWMG7r//fnTp0gW33347vvzyS7z33nt44403uH2OHTuGHj16QCgUYsaMGZg8eTK6d++OyZMnY+XKlfj3v/+NefPmcUOajx07hp49e2LWrFkwmUxwOp3Iy8vDM888g+zs7GaXTaVSobKyMuH3JBEK4PSmfic+nzIMIaQDihU72/RGy+LiYnzwwQd45513MHToUCxbtgwLFizAzJkzAQDnz5/H5ZdfjkceeQQAMGnSJCxfvhwffPABhg4dirfffhuvvvoqnnvuOe6Y48ePx759+7Bt2zaUl5fDbDZjw4YN2LNnT0Jly8zMhF6vT/g9qWQtk6PZ/EJ39BNCOpJYsbNNazA8Hg8PP/wwiouLodfrkZ2dDalUyv0+NzcXjz76KP70pz9x2+677z7cddddqKmpabI/gLC7+lORl5cHvV4PhmGa3PAZS6ZCghqLO+XXFwn4OHLegvuX/Ypnb7ocxUO7pHxMQghpbbFiZ7tMdimRSNC5c+cmyUIgEOC9997DgAEDwraLxeKI+7ek3Nxc+P1+1NfXJ/Y8laRFXl8i5GPD4WoY7B688M1BODzts3QAIYQkIlbspNmUG+Tm5gIAamtrE3pelrJlEoxIwIdU1PjnMNgjz6X2aUklNh+paZHXJISQVMWKnW3aRHYxY6eWsdkSW75YqxC3yOufrLPD7PA2lkfS9E+jt7owe9UhAMCpBbe2yOsSQkgqYsVOqsE0UKlUABJfdEzTQouFTX53B8qqrdz/M6RNjyvkN/65vH66M5MQ0v5ixU5KMA2STTA6ecvUYNy+APZUNi7aI4hwg41Y2PjnctAiZYSQiwAlmGaQy+UAEp+yXyZuuSn2be7YHfvSkARjcXlj7EkIIW0jVuykBNOAzcJWqzXOnuFaYw0XiTDyn0UoaNyut7pa/HUJISRRsWInJZgGGRkZABJPMLJWSDBuXwDLfjoZc5/jNYkNRiCEkNYQK3ZSgmkgk8kABGdrToRC0jqrUK7Zfx5WlxfLfjqJyvqmVc8TekowhJD2Fyt2UoJpwOfzIZVKE+6DkYtbZ6T3/jMmzF51CK+uO4IxC7fi24Ph6y0cPm/B1qN6+AM0tQwhpP3Eip2UYELI5XI4nc6EnhOtv6QlrNp3nns8/T978dYPx7n//1JRj4c/3IV1B85HeiohhLSZaLGTEkwIpVKZ8I2Wklbog4nm9Y3HmmyjvhhCSHuLFjspwYRQKBQJJxhpK9ZgmqNHtqJdX58QQqLFTkowIUQiEbzexO4vacn7YJLRWn1ApGPz+gPYdKQGJkfkOe0IaUnRYiclmBBisRgeT2JfyNYYppyIx5dHX/emxuLCqbrEBi20lHe3luP3H+3CoXOJrxJKUlNZb8fwed/jkU924/WNR9u7OCQNRIudlGBCJFODSWTtmNZkdnpxQm9DIMDA6vLiH+uPYNi87/G7N7fHnSEgUT5/AIu+P45/xgheS36swPdlety2+Ce8u7W82cd2eHxYve8c7C1c5nTy6rpS1DfMxl1voxoMaX3RYie1r4QQCATw+zveHF+V9XaMWbgVAJCnksLPMKi1BhdB8/gDcHh8EWdnjsbl9WPuuiO4/vIc3Ng3N+x3Pn8AUz/ahR+P1wEAHrm2B1QRJuY0Oxs/bKt+O4fp1/WM+7oMw2DcP7fhvNmFv03oi6nXdG92mZvD5w9g0Q8nsPNkPd697+oWmwn7YlNtaRzNo5a1zGSshMQSLXZSDSYEn89Pasni//xhWCuUpvme/eoA97ja4uKSC0sibGzGqza74IszE/O8/5bis19P45FPdjf5XZXZxSUXAHB7Ix9LEFKzEwmbV8s7WmPFeXNwCpzjLXwjKcMweOzTPVj0/XGUVBhw4BJuunOF/E00LTQZKyGxRIudlGBCBAKBpJq8rrksC3Mn9muFEjXPzlOGmL+XNwxEOKG3Yfj873HXe7/E3H/r0eiLrqkuuCJ2eZtetfj8AXhCktihcxacro8/QwIPjee+pfu26u0efF/WuG64WHDpfvQ9vsZz3959hCQ9RIudl+63LAl+vx8CQXJfSPtFPH3+piM1OGt04GjDejO/nTZh/L+246kv90XcP3Q1zQtrQyqpEMKQpQScERKMy9e0VnPtwi1xy6mUNjbjRUpcqbiwTyeRJsOOJjTByNt5lCNJD9FiJyWYED6fD0JhcoHnvCmxGQDa0h//sxejXtuCP322l9tWVm3Fyt/Owenxg2EYnDc5wTAMvtp9JmxQwF9XHuQeBwIMeDxeWLt+pAEE0ZrgvP4AGIaJ2gypCkswjcf4+UQdZq86hIra5JvNfBdMqROazC41/pDzq7iEEym5eESLnfTpC+F2uyGRSJJ6bmindkfi8Qfwwc8nsfC7o+iRpUDFBcOa9581AQCsLi/6z9nY5PlOT4Qmsijzo+2pNOKLnaex5WgtHh7ZDdNGdQ9LVqG1CmvIejfvba/A9mO1+LSkEn+5sTeeGNsrsTcJwOcPL1NrTVJ6MQgEQhPMpfs+ycUjWuykGkwIl8sFqVSa1HNDm5U6klqrC3sbVtK8MLkAQI0l2ET25a4zEZ9/4WSbRrsHT37+W8R9p7xfglX7zsPs9OLf3x/HjW9s45quGIbBhkPV3L5V5sb1bkKby/79/fGkBmL4Ao01Ih6v5VYivRiFJvj/t+Foiw9TJ+RC0WInJZgQDoeDW50tUdYOusLk7FWHwzq/I/l4xyn8fX1pxN/5GQbHa6xcorh36a/YUV7frNfWW93QN/TxXPnqJkz/T2MT3uHzjaO8QodB+wIM7B5/wn00O040lkkrF4ct3taSfj5Rhzvf3YGno/Rvperbg1W4b2kJ/nuwKqymEioQkoDPmZwY989tSSXljqLW6u6w379LRbTYSQkmRCoJZuKgTi1cmrbxS0X8ZPC3NYej/m7qh7tw45vb8fSKfSirtqC0qum63LFsKdPjznd3wOQIDxABJhgcGYaBSBA+OuWKv32HcW9si7hUgdcfQElFPdy+xgRkc/vwj/82Jshotc1Ub+78+7ojuG/pr9hdacQ3v50DEKyZ7T1tDOt4T8VXe87i5xP1+ON/9uJ/3vk5YmC9MJdUW1wRa6cdnccXwD83HsWQf2zG/Ut/be/ipDVKMM3g8XggFifXdDL1mm4tW5gO5rvDNRj/rx8Tft5HO05hd0MT3YWuWfAD/rJiP74NaTpjnTU6m9RivP4A/uednzHl/RJ8+kslgOCAgwNnTHHL8c7WE+j3t++w+UgNgGCz3NIfK/C7N7fh4Nno98xsOFSNF745gHMmJ7YcDa8Jrtl/HmsPVGHSOzvw5y8jNxte6NOSSjz/9QFuxN+FvCEDKPafNeNYTdP9ItVW6i4YDQg0f6Re6MCMWIM02triH45j8Q8nAATPBWk/0WJnm3fyMwyDVatWYcuWLcjLy8P06dOh1Wpj7r9u3Tps3rwZ2dnZePzxx5GVlRW2j8FgwLvvvouamhqMHTsWt99+e1L3s6TSyc/j8fDWvVdixmfNCyQk6LQh9v0xbE0gkl2nDLi6qxa/nTahrNqCPLUMh84Fa1AHz5kRCDC47K/fRnxut1nrI25/adUhKCRC/PnL37j+pwlv/YRTC24FEPw8+gIMRA1NbO9uPYH9Z834fGfTPqpPfznFTUb634NNkyTrvwer0DNbicvzMvDOlhOoMrvw5e4zmHRVJ7x+50DwQ4aFXzhYQcBveo3ojVCzM10wCOV/t5Vj4XdH8afreuLP43qDz+fB4wvev8QOtth+rBZnjA58d7gGeosLHzw8BNct3Irfj+6O58f3ifheXF4/pG107039BTXRk3V2dM9SYOmPFfhy1xn8flR3DOysQVG+qk3K4/b58fmvp7H0p5MYV5SLObe3371x6w6cxy/l9XhoZDf0zs1o9deLFjt5TBtejrhcLkycOBGbN2/GNddcg/LycjidTmzcuBGDBw9usr/H48HkyZOxfv16XHPNNTh16hQsFgs2bNiAESNGAAC2bt2KyZMnQyKRoFevXvj5559xyy234JtvvkloyLHP54NIJMIrr7yCl19+Oen3yAauXjlKfDh1CEa9Fv/+D3Lx+3jaUEiFfHx7qBrf7D2LTU+PgVomQp/ZG2I+r3uWAicbmqd+P6o7LE4vHhvTA4VaOd7ecgIn6+xYdyC4WmnJC2MxfP73TY4x73/64+7BhRAK+Bj/r+0oC6ndrH9yFPoVqAEER/TJxIKIyfNf9wzCHVc2NuMOn/c9qi2NAyn+eF1PfFoSrPX99ZYi9MxRYsZne7kke6HSueObzCT+8Y5T+Nuaw7iikwoLJg3AFZ3UMc8Ny+3z49uD1bh1QD5EAj7WH6jC5XlKXJYTDIx1Njd8fgZ56sZO5JKKekxfvgdGR/y+l50vjkWOqmkH9IZDVfjw51OYP6k/emQrm1XWaPwBBrf8+0ccbahR5qok+PXFcSkdMxH1Njf2VBoxtigXAj4PN725nSvL5KsK8drk/q3W7xgrdrZpgpk9ezYWL16MzZs3Y/DgwXC5XJg8eTLMZjN++umnJvv/4x//wIIFC7Bp0yYMHz4cHo8HU6ZMQWVlJXbv3g2Hw4FevXph5MiR+Pjjj6FQKLB//34MHToUH3/8MaZMmdLsspnNZmg0Grz++uv4y1/+kvR7PGt0wOdn0C0ruE7L1qN6PPzhLswc2ws/n6iL2hxESDL65GWEJZxYFt45AAMKNdh6VI/535al/Non59/CtRSs3ncOM78IH9hQPLQzhnXPxKl6O/61Obga66398/HanQMgFwm4mtkNr2/l+oieH98Hr20Ilu3UgltRa3XjtsU/osbixronRuGKTmqYnV4MfKXpkPlYjv39Zji9fhw5b8FTX+5DdoYEB0OmCyqfdwt+raiHSibCnkojRvXKQs+QpOPzB7D9eC2Gdc+MeG9RpGH81/bOxifThoJhGO48uX1+CBumVWEDvj/AoNriQieNrFnvZfcpA0QCPk7V27HhUDX+MLoHJr+7A0DwxlpHhFsHpgzpjAWTB+CE3opdp4y4Z3DnsJpxKmLFzjZLMAzDoLCwEI899lhYltu6dSuuv/56lJaWok+f8Gp3z549cc8992DevHnctpKSEowYMQK//fYbKioqcM899+Ds2bPIzW2clPHOO++E2WzGpk2bml2+kydPokePHli2bBmmTZuWwjttyucPQCjg47vD1Xjs0+jT6xNCLn53DCoIW868LfTNV+G58ZfD7PQ2SeSpunVAPs4ZnfjT9Zc1mdy2OWLFzjbrgzl27BjOnz+P22+/PWz7gAEDAADHjx8PSzCVlZWoqKiIuf/27dsxbNiwsOTC7rNs2bImZSgtLUVZWRn4fD4kEgk0Gg3kcjkuu+wyGAzB+bwu7N9pCeyVyk398vDFo8Ox/kAV/jC6O/625jA379ekqzqhq06BQ+fNeOPugXh8+R783DC0dkChGs+P74ORPTPh8gbw2oYyfLTjFPLVUhjsHrgTGKE0qLMG+5rR6U0IiaytkwsAHKmy4OEPd7XKsdc3NNE+8slu/PLCDchXN68mxYoVO9sswdTUBEfn5OXlhW1Xq4PttEajsVn7y+VyiEQiGI1G1NTUID8/v8lraTSaJscDgBUrVmDOnDlNts+ZMwfXXnstAEClat0OweE9MjG8RyYA4KOpQ3G63oFvfjuLaaO6h93v8fJt/fB/e8/iT9ddBrW8cbtMLMCc2/uFdSCaHV6UVlvQI0vBtTWbnV5sP1aLK7toIBUJoJWLIWioEpudXqz67VzY8OPX7xqIod106KSVYcOhaogEPNzYNxf+AINNR2qQo5JCKRFCKuKj2uzCmv3noZKJsO+0CQ6vHy/dWoRCrQyLvj+Oz3eeweanx2DcG9sAAF89PgKvrjuC3rkZeOzaHvjrqkPYebJxgs4MiRBXd9OGTbLZv5M6rAkDAD54eDCmfdR0hufmuG1APtfXQQiJLNLMHPFYLMGBNZFiZ5slGI1GwxUmNGlYrcH244yMjKj7h3K5XPB6vcjIyIBarcbp06ebvJbFYon4ZqOt9aLVamE2B4MZm/DaSpdMOf48rneT7ZfnZeDFW4qadQy1XMQlLW6bTIQJAwsi7y8T4aGR3fDgiK4or7Wja6acGxUFBKvMLKGAh5v7hyfxrpkKDLvg9VjzJw3A/EnBWiY78goA1swYxT1e8diIZr0vlsnhgdfPIDtDEnbMUHa3D15/oMn09IEAw7U1v3Vv5OM7PL6oS0+Htp8DwXsvDpw1QSYWoFAjh1IqhIDP4256ZF/rwudtOarHf0oqMX/SAEhEfO5iwunxw+LyYuPhany28wyu6ZmJJ27oBZVMCL3VjR3lddhTacRfbrwcQgEPtVY3ujf07+0or0edzY2ZX+zD9Ot6ho3s8vkD+Lm8HgML1ThrdKJvvgq/nTFxbfUAsOHPo9EnTwW3z483Nh7De9srcHluBm4fVIDpY3qCAcAD8M9NR/H2luCicTOuvwyPjgmuAXTwrBkT3mrsO31tcn/ccWUn1Ns8WLP/PBaE9PO8de+VuKlfHuptHnh8Ady66EdYG+47GtxVi1k398EVndQQ8HlY8G0Zlv10MuzvUPbqeAQYBnKxEGcMDuit7rD38tS43rhveBdkKsT4YtcZvPBNcA694qFd8IfRwYs3BgyG/qNxEMVdVxfipVv7IsAwqKizYfK74bOM//riWNRa3chVSWF2ejDuje1hvy/UyvDhw0OQkyFFnd2Nsf/cxv3u/905AHdeVQin14/7lv7apNXg51k3oJMmeDEXaVXahXcOwLW9s6GVi9H7paYjId9/4GrYPT7kZkhhc/vwaISm919euAEGuwe3Lgrv3x7aXYclDw5ucpH51r1XJjXYIVbsbLM+GL1ej9zcXGzevBljx47ltv/6668YPnw4ysvL0aNHD24723G0bt063HprY1DZv38/Bg0ahMOHD+Pzzz/H119/jdLS8LvMH3jgARgMBqxfHz6a5vPPP8eXX34Jv98Pt9sNo9EIm82G2bNnw+12Y9q0aTh58iS6devWOieBEEJaSXAy2rZfZffDDz+MGjvb7EbLnJwc9O/fH2vWrAnbvnr1amRlZaF79/DVC9VqNYYMGRJx/4yMDPTpPVwt+AAAHA5JREFU0wfjxo1DWVkZjh8/zv3e6XTiu+++w9ChQ5uUobi4GKtWrcLatWuxceNG7Nq1C6Wlpbj33nthswVn6lUqUxuuSAgh7YHP57XLEu6xYmeb3sk/bdo0vP/++1i+fDksFguWLFmCf/7zn5g2bRp3YkKbsaZNm4YPP/wQH374ISwWCz7++GPMnz8fU6dOBZ/Px6hRo9CrVy88/PDDKCsrw7lz5zBp0iSYTCbcf//9CZXN6QxOty+TJdbBRQgh6Sxm7GTakNfrZWbNmsUIhUIGAMPn85mpU6cyTqeTYRiGqaurYwAwzz//PMMwDOP3+5nZs2czIpGIAcDweDzm/vvvZ+x2O3fMI0eOMEOHDmUAMACYgoIC5ptvvkm4bC+++CIjEAiYQCDQMm+WEELSQKzY2aZTxQiFQsyfPx+PP/44jh8/jp49e4Y1jSkUCgwdOpQb0cXn8zF37lw88sgjOHr0KLp3746ePXuGHbOoqAg7duzAzp074Xa7MWzYsKRqIVarFRkZGe1SxSSEkI4qVuxs0zv5L2YPPfQQtm/fjpMnT8bfmRBCCADA6w1O1yMSiZr8jhJMiGjrSncEDMPAbDajvr4eZrMZdrsdZrMZRqMR9fX1sFqtcLvd8Hg88Hg88Hq9cDgcsNvtcDqd8Hg88Pl8TYZy83g8CAQCCIVCiMViiEQiCIVCiEQiiEQiyOVy6HQ6qFQqbui4QqGARqOBWq2GVCqFVCqFQqGAWq2O+CG8FPh8PphMJthsNtjtdlgsFu7cOp1OuFwu2Gw2WK1WOBwO7sfj8cDtdnPD730+H/cTCAQQCAS42YvZK0T2vIeeW4lEApFIBKVSCbVaDbVaDZVKBZVKxT3OycmBWq3usLV0q9UKg8EAu93O/TgcDlitVlitVu78so/Zc+pyueB2u+H1euHxeMI+4zwej/tsi8ViyGQyZGRkcD+h50+j0UCj0XCPtVrtJfF5drvdOH/+PIxGIwwGA2pqarjPr8vl4j6rbreb+0yzn1W/349AIIABAwZg4cKFTY5NSyY3mDlzJg4dOgSZTAaNRgOdTscFTJlMBqVSCa1Wy33YdDoddDodFApFQpNqxhIIBOB0OmG1WmGxWOBwOGCxWGCxWGCz2VBTU4OamhpUV1ejvr6e+53RaERVVRVcLlfM4/N4PO6LxH6ZFAoFZDIZJBIJBAIBBAIBeLzgaBSGYbgh3T6fj0tMPp8PXq+XS1ImkwmBQPNmE5BKpdBoNMjMzIRSqYRCoYBOp0NWVhb3xc3JyUFmZiYUCgX3BWe/2DKZrMUDpMfjQW1tLQwGAxec6uvrUV9fzwUqm80Go9EIi8UCs9kMq9XKBTmbzYa6urpmnwMg2CEqk8kgFoshkUgglUq55M3+8Pl87gcIXkSwn5GamhoucTkcDi6YejyxV1YVi8XIyclBdnY2cnJykJ+fj9zcXOTm5kIul0Oj0SArKwtarRZZWVnQaDRQKpVcGVLFMAzcbjd3ccMmCfbiqKqqCtXV1dy/1dXVMBgM3N+iOSQSCZRKJWQyGYRCIaRSKZeAxWIx9xkHgheVLpeLu/ByuVzc94/tvI5FLpdDqVQiIyODO6eZmZnQ6XSQy+XIzs5GVlYW91lXq9XQarVcsmqJ88owDDweDxwOB2w2GywWC2pra7nbMCwWC/ee2IvOqqoq1NbWQq/Xo7a2NubxBQIB5HI5JBIJFy9CP6sCgQAOR+RZ0akG02DmzJnYvXs3XC4XDAYDTCYTrFZr1JszQ4lEIkgkEojFYsjlcu7qUiKRcH8APp+PQCAAv9/PBQKv18sFKDZIxCMQCJCTk4OcnBwuAWo0GuTl5SE/Px9ZWVlcLUKtVkOn00Gr1UKlUkEoFLbK1WsgEOCuJE0mE+x2O0wmE8xmM1wuF1wuF1ejYq9CDQYDd7VfX18Pg8EAi8UCtzvy7L2h71+hUHAJkg0ibI2Kz+dziZL98vr9fvj9fi5JsmXyeDyw2WzNClxs8GVrBxkZGZDL5VAoFMjIyOD+JgqFgtvGfhnZHzYQSaXSFgvYF/J6vbBYLDCZTFxgMZvNMJvNqKmpgV6vh16vR11dHRfE9Xo918wRCY/H45I7G6RFIhH3GWcDNp/PB4/H42peHo8HTqeTC3zs1W+8kMPn85GTk4OCggLk5eUhKysLOp0OBQUFyMzM5M67QqGAXC7nas9KpRJKpbLFahV+vz/sgsJkMnHn1WQywWg0cnHCarVy57W2thYmkylq0L3wvCoUCu68snGEDeBsi0roZ9jtdsPtdsPpdHK15uaEcaFQyMWL3Nxc7tx26tQJnTp14i4scnNzoVaruTgmEomSjhuUYGJgGAYOhwNOp5O7gjWbzbBYLKirq4PRaOSuwNjmJ7Y6yVbL2Wok03Bnt0AgCPuSsl8KtjYhl8u56jl7Ba9SqaBUKpGdnY3MzMyLronD5/PhjjvugEqlglarxdtvv530sRwOB/R6PXdu2eAYGjBtNhsXvNgrd/aHTeLsOQfAJR22KYRtWhKLxVAqldDpdNyVJhuotFotsrOzoVAompUQAoEAVq1axTWBTZ8+Pelz0NYCgQDXJMI2k7A1uNDzzzaNsBdH7GecPdfsD5tsJBJJWHJlP9/sZ539P/s5z8zM5BJ1ayXg1sRcMHtDIBBAXV0dV/sKbbY2mUzchazdbuc+v+yFD9tSwNaKQz/DEokEEomEu+hRKpWQSqVc7GDPpU6ng1Kp5BJwa9T+46EEkyKv14s9e/ZwzRxFRc2b3uVSYjQaodPpAARHAja3KeNSku7n4MLgmo6MRiN69OiB7Oxs5OXlYfv27fGfdImjPpgUlZaWcouf9enTp8m0NengzJnG1Rw7d+7cjiVpP+l+Dg4ePIiBAwcCSO/vAduM1lEHC7W0jlcPvciEtrOm6zQzdA7oHKT7+wfoHERCCSZFJlPjLKlarbYdS9J+6BzQOUj39w/QOYiEEkyKQkc9SSSSdixJ+6FzQOcg3d8/QOcgEkowKQq99yFd213pHNA5SPf3D9A5iIQ6+VPUp08fvPLKK/D7/WFLPqcTOgd0DtL9/QN0DiKhYcqEEEJaBTWREUIIaRWUYAghhLQKSjAp+vXXXzHh/7d37kFRVm8c/y6wCuvqossG6LhyGQgZNRUoJYsuErp4S60GvEwyqUxpVCaRl6bSUiq1yciQclLLS+o4lQqatwzG0kZEIVHZhKBUVgFXrsLu9/cHs++v10UyYAH1fGYcZ8953ofzPGd5H857nvc8Y8ciJCQE8+fPR1lZWUcPqdX89ddfGDt2LNLT02Xt169fx8KFCxEWFobRo0fj8OHDdtf+8ssvGDNmzC39UVtbi2XLluHBBx/Ek08+iV27dt3WOUrtybFjxxAfH4+YmBh89tlndmfEmUwmzJ07FyEhIZg4cSJOnDhhp2Pv3r0YOXIkwsLCsHTpUlRVVcn6zWYzkpKSEBoaCoPBgJ9//tmhNv0XqqurkZqailmzZuGdd96B0Wi0kykoKMDUqVMREhKCGTNmoLCwUNZvtVqxfv16jBgxAuHh4UhLS0NDQ4NM5q+//sLMmTMREhKCmJiYTvlyZlVVFZ599lnExcXZ9f2XOY6Ojm5yjrOyshAdHY3Q0FAkJSXJUp3vChxT4+zeIDU1lQAYFRXFpKQk9uvXj76+viwvL+/oobWYo0eP0svLiwC4efNmqd1kMtHPz486nY6JiYmcOHEiAXDdunWSzOeff27nDz8/P1ZUVJAkKysrOXjwYPbo0YOvvvoqY2Nj6eTkxOXLl7e7nU1htVr57rvvUqFQMDw8nOPHj6dSqWR0dLQkU1BQQK1WSx8fH7755puMjIykQqFgRkaGJLN48WIC4OTJk/n6669Tp9MxNDSUN27cIEleunSJer2enp6eTExM5Pjx4wmAGzdubHebb6auro5BQUFUq9WMiopiYGAglUolf/31V0nm8OHD7NKlC4cOHcoFCxZwyJAhVKvVPHPmDMlGP06aNIkuLi6cOXMmX3zxRapUKsbGxko6cnJyqFarGRQUxAULFnDEiBFUKpU8evRou9vcHPHx8QTAwMBAWXtbzPHq1asJgAaDgW+88Qb79u3LgIAAms3mdrXRkYgA00JMJhNVKhUXLVoklQqtqKigh4cHk5OTO3h0Leepp57ihAkT7H4ZXnzxRfbt25elpaVSW1JSEnv37s26ujqWlpZSpVJx8eLFMn9otVp+8MEHJMl33nmH7u7uLCwslHSsWrWK3bt3l4JQR2Iymdi7d29u2LBBalu7di0BsKSkhCQ5btw4Dh48mJWVlZLM1KlTOXToUFqtVv7+++9UKBRMTU2V+v/8808qlUpu2rSJJDlz5kz6+PjwypUrksy8efOo1+tZX1/vaDObxWKxcOXKlfz7779JNgaL0aNHc/r06VJ/QEAAJ06cKI21oaGBYWFhnDp1Kkly586dVCgU/OmnnyS9Bw4cIACePHmSJBkeHs6IiAjW1tZKP8dgMDAyMrLdbP03MjIyqFAoaDAYZAHGNsdr166V2oqKiqhUKqU/yl544YVm5/jixYt0dXXlkiVLpP6ysjJqNBp+/PHH7WBd+yACTAvZsGED1Wo1r1+/LmuPj49nSEhIB42q9VgsFtbU1BAAv/nmG6ntvvvu44oVK2Sy58+fJwAePnyYX331Fbt3727nj9mzZzM0NJQkOWDAACYmJsr6KyoqqFAouHXrVgda1XKSk5Pp5OTE69ev02w2U6lU8ttvv5XJHDp0iABYUFDApUuX0sfHhxaLRSZjMBj49NNPs6Ghgb169eLq1atl/b///jsBMCsry+E2/ResVivHjRvHadOmkSRPnDhBADx9+rRMLiUlha6urrRYLIyNjeXo0aPt9Pj4+HDhwoUsKSkhAB44cEAms337dgLoFE8AysrK2Lt3b7700ktcunSpLMDczhz37Nmz2TlOS0uju7s7q6urZTIzZsxgeHi44wxrZ8QeTAvJzMxESEiI3ZlDfn5+ds+j7yScnJykfRPb6cAFBQUoLS1FRESETNbHxwcKhQKFhYXIyspq1h8VFRXIzc2106HRaKDVajulz7KysvDee+/hueeeg1qtxvHjx1FfX49HH31UJufn5wcAKCwsRGZmJh555BG74+ZtfsjPz0dZWZmdDl9fX0lHZyA3Nxfz5s3DAw88gB9//BHz588H0Pi979WrFwYMGCCT9/PzQ21tLUpLS5GZmWlnn0KhkHyQlZUFhUKBESNG2OkAgKKiIgdadnvMmTMHTk5OeP/99+36bmeOy8vLm53jzMxMPPjgg3Bzc2tSx92CCDAtpLKyEu7u7nbtKpXqtgqHdWauXr0KAPD09AQA6eh5jUYjk7PVV7FVfGzOH7YN0Jt1/FOms2CxWPD+++8jIiICDz30EFJTUwH83w8326lSqQBA8kNzNt5Kh61wV2fxQ3p6OtatW4fTp0/DYDBIN/9bzbPtRnk734XKykqoVCp06dLlljo6kq1bt2LTpk348ssv0aNHD7v+27EPaH6O7+b7xz8RAaaFaLValJeX27WXlZVBq9V2wIjaDtsKxhZgbPbcbK+t0JpWq/1Xf9hWQzfLkOxUPisvL0dUVBSWLl2KFStWICMjA927dwdwaz/Y/KXVauHh4dGsH26lw2w2w2KxdBo/zJ8/H1evXsWOHTuwa9cuvPzyywDQrH1A46r3374LHh4eqKqqsivv/E8/dhTnzp1DXFwctFotdu7ciYSEBKSnp8NkMmH58uWSDa2d47v5/vFPRIBpIXq9HufOnbOrw56dnY3Q0NAOGlXbYKtFbnvc5enpiS5duuDs2bMyuezsbABAaGgo9Ho9zp49e0t/uLm5QafT2ekwGo2orKzsND6Lj4/H6dOncfz4cSQkJMgeg+j1egBAfn6+7Jrs7Gw4OzvjgQcegF6vt+u3yYSGhsLb2xvOzs5N6gDQafwAND4unThxIuLj46WUdb1ej/Lycly+fFkmm52djYCAAPTo0aNJHzQ0NODUqVPSdwVAk98njUYjrZY6gtraWowZMwbDhg2D0WhETk4O/vzzT1RVVWHbtm0wGo1tMsc2HbwpRf9uuH/I6OhNoDsV24bdwYMHpbaLFy+yW7du/PDDDztwZK3nyJEjBMCLFy9KbQaDQZauSzZu4Ht7e9NqtTIvL48AeOjQIan/77//pkql4kcffUSycQMzLCxMyjIjybfeeotubm6sqalxrFG3wbVr16hUKmXZQTczcOBAzp49W/pstVoZHR0tJTLs3buXAHj+/HlJ5vTp07JEhpEjR3LChAkyvTNmzGDfvn3b0pwWcerUKaakpMjaFixYQH9/f5JkdXU1u3Xrxk8++UTqr6urY2BgoJRptnLlSvbs2VOW8LFt2zYpOcBisbBPnz5cuHCh1N/Q0MDhw4czKirKkea1iJs3+TMyMlo9x9nZ2XZJHcXFxXR1dZX59k5HBJhWMGrUKHp5eTEtLY3btm2jr68vdTqdLDXxTuPQoUNMSEggAM6ZM4c5OTkkyX379hEA4+LimJ6eLr0fsGrVKunap556il5eXvziiy9k/rh69SrJxgwkZ2dnTpo0iXv27GFiYiIBMCkpqUNsvZn8/HwC4JQpU/j888/z6aefZnR0NOfNm8eqqiqS5Lp16+jk5MR58+Zxz549nDx5MgFw27ZtJMkbN25w0KBB9Pf356ZNm7h+/XrqdDoGBgayrq6OJLl7924C4KxZs5iens4XXniBAPjpp592mO02bAHyrbfeYmZmJlNSUujm5sZly5ZJMq+//jpVKhWXL1/OXbt2cfjw4VQqlczOziZJlpaW0sPDg8OGDeN3333HVatW0dXVlQaDQdLx4YcfUqlUcvHixdy9ezdHjRpFhULB/fv3t7vN/8aSJUtkAaYt5thqtfLxxx9n7969uW7dOm7dupV6vZ5eXl6dImW/rRABphVcv36dr776Kl1cXAiAo0ePZm5ubkcPq1XEx8dz6NChHDJkCIcMGcItW7ZIfRkZGfT39ycAenp68tNPP5WtRsxmM1955RXJHwaDgXl5eTL9WVlZHDhwIAGwZ8+eXLZsmfRyWkdTWVnJyMhIDhs2jOPGjWNMTAyff/55PvLIIywqKiLZeGPYvHkzvb29CYC+vr5SOreN0tJSTp8+nQqFggqFgjExMbJ3f8jGG5Cvry8B0Nvbm2vWrJH5siNZvXo1dTodAbB79+5MTEyUvZ9TX1/Pjz76iGq1mgAYEhIiW8mTjSnskZGRBMCuXbsyISFBln5ssViYlpbGXr16EQD79+/P77//vt1s/C9s2bLF7v2ctpjja9eucc6cOXR2diYAjh07VnpZ9W5BnKbcBjQ0NMBisdwzRYaqq6vh5uYGhULRZP/t+OPfdHR2SKKmpqZZG27cuAGFQgGlUnlLPZ3VD1arFVeuXIFGo7nlPFqtVtTV1dml2v6T2tpaKJXKW9ZHsfnRlol3p9EWc3w33z9EgBEIBAKBQxBZZAKBQCBwCCLACAQCgcAhiAAjEAgEAocgAoxAIBAIHIIIMAKBQCBwCC4dPQCB4G6grq4OxcXFsFgs8PDwaJfzpK5cuQIPDw+H/xyBoKWIFYxA0Aoee+wxDBw4EN26dUNAQACCgoLg4eGB+++/36HHrhcXF0On07W4zPCsWbOQnJwsfa6trUVQUBDOnTvXVkMUCMQKRiBoDUajETqdDqtWrUJgYCBcXFxQXFyMvLw86RRmR2A7Rr6oqAj9+/f/T9feuHEDX3/9NVJSUqS2n376CQUFBfDy8mrTcQrubUSAEQhagVarRUREBObOnduuP9f2VvjNRa+ao7i4GGVlZcjLy0NNTQ10Oh1ycnIAAJs3b0ZwcDAuXLiAfv36NVmrRCD4r4gAIxC0grq6umaP+Dhz5gzWrl2LlStXIjc3FyQxaNAgO7lTp06hvLwcDz/8MFxc5L+WRqMRu3btgq+vL0aNGoUuXbqgrq4OQGOBqvr6emRnZ6NPnz7o06dPk+OoqalB//79pcJvADB27Fg7ucGDB2Pu3Ln45JNPbst+gaA5RIARCFqB2WxG165dceHCBVy8eBGXLl3C5cuXERwcjIiICOzZswdpaWk4dOiQtFp45plnsHnzZjg7O8NkMiE2Nhb79+8H0HiDP3DggFSgLSMjA5MnT4a7uzsqKirg6+uLkydPSnVE8vPzMX36dFy4cAHOzs5IT09HZGSk3Tjd3Nxw7do1WK1WDB8+HFFRUXj77bcBNJbEDg4OxqlTpxAQEGBXaVIgaDEdd86mQHBnU1tbS6VSSQB2/+Li4kiSqampBMBx48bx2LFjUl2Uffv2kSRjY2Pp7e3N9PR0/vzzz+zatSuXLFlCkqyqquJ9993HhIQEWq1WVlRU8PPPP6fVauWlS5cIgEqlkhMmTOBvv/3GoUOHcubMmc2O2Xbdr7/+KrWlpKTQy8ur05zmLLh7ECsYgaCFODk5ob6+Hi+99BLGjBkDT09PeHp6QqfTSafrajQaAMD27duhVCoRFhaG/v3748iRIwgNDcWWLVuwdu1ajBo1CgAwY8YMHDx4EIsWLcKGDRtgtVqxfPlyKBQKaDQazJ49GwCkFcyAAQPw7bffSrovXbrU5FjNZjMuX76MH374ARqNBhqNBufPnwcA/PDDDxg8eDAKCgrg7u4OnU7nUL8J7h1EgBEIWkh1dTUAYOTIkVKAuJlu3boBaNwDsQUdnU6HsrIyFBYWwmq1Ijw8XJL38fHB8ePHAQBHjx7F+PHj4erqaqfXFmAWLVok6W1oaEBlZWWT44iLi8OOHTukz0FBQXYygYGB6NmzJ65evdrpygcI7kzEezACQQux3YStVustZWwJAA0NDdL/BQUF6NOnj5RqbDKZJPkzZ87A398fQGNQ4i2qaVgsFgCQrTZ69OiBy5cvNym/ceNGnD17Fi4uLli/fj1KSkpQUlKC9evXAwBOnDiBkpIS/PHHHyK4CNoMsYIRCFqIWq2Gu7s7iouLZe319fVwcnKCs7OzlEacnJyM4OBgfP/99zCZTIiJiYGvry9CQkLw2muvYcWKFcjJycHGjRuRnp4OAJg8eTKmT5+OJ598Ek888QSOHTuGNWvWYMOGDaipqQHw/xUSAOj1ehQVFYGkXZBwc3NDXl4eXFxcMGnSJOm6kydPIjg4GEOGDHGYnwT3MB27BSQQ3NlMmTKFAOju7k53d3eqVCo6OTlxzpw5JMn9+/cTALVaLQEwMDCQO3fulK7Pzc1lYGCgVFr47bffljbbrVYrFy9eLCUSKJVKzpo1i/X19czPz6e/vz/NZrOk68iRIwRAk8nU5FjXrFnD+Ph4Wdu0adOYnJzc1m4RCEiKkskCQauwWCw4cOAAjEYjrl27BhcXF6jVakRGRsLf3x/79+9HZGQkysrKoFarmyyta7FYYDQaodVqmzzDzGw2w2g0ol+/flL6clOQREFBAQICAtrURoGgpYgAIxA4kH379iEqKgrl5eXi7XjBPYfY5BcIHIgtAUBsnAvuRUSAEQgcSEBAACZMmCDbjBcI7hXEIzKBQCAQOASxghEIBAKBQxABRiAQCAQOQQQYgUAgEDgEEWAEAoFA4BBEgBEIBAKBQxABRiAQCAQO4X8krNr+/uC5wwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.xkcd();\n",
    "plt.xlabel('Epoch #');\n",
    "plt.ylabel('Loss');\n",
    "plt.plot(losses);\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 再次进行评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "准确率: 91.0000 %\n"
     ]
    }
   ],
   "source": [
    "cnn.eval()\n",
    "correct = 0\n",
    "total = 0\n",
    "for images, labels in test_loader:\n",
    "    images = images.float().to(DEVICE)\n",
    "    outputs = cnn(images).cpu()\n",
    "    _, predicted = torch.max(outputs.data, 1)\n",
    "    total += labels.size(0)\n",
    "    correct += (predicted == labels).sum()\n",
    "print('准确率: %.4f %%' % (100 * correct / total))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch : 1/10, Iter : 100/234,  Loss: 0.0002\n",
      "Epoch : 1/10, Iter : 200/234,  Loss: 0.0001\n",
      "Epoch : 2/10, Iter : 100/234,  Loss: 0.0001\n",
      "Epoch : 2/10, Iter : 200/234,  Loss: 0.0005\n",
      "Epoch : 3/10, Iter : 100/234,  Loss: 0.0002\n",
      "Epoch : 3/10, Iter : 200/234,  Loss: 0.0001\n",
      "Epoch : 4/10, Iter : 100/234,  Loss: 0.0003\n",
      "Epoch : 4/10, Iter : 200/234,  Loss: 0.0001\n",
      "Epoch : 5/10, Iter : 100/234,  Loss: 0.0002\n",
      "Epoch : 5/10, Iter : 200/234,  Loss: 0.0003\n",
      "Epoch : 6/10, Iter : 100/234,  Loss: 0.0002\n",
      "Epoch : 6/10, Iter : 200/234,  Loss: 0.0002\n",
      "Epoch : 7/10, Iter : 100/234,  Loss: 0.0001\n",
      "Epoch : 7/10, Iter : 200/234,  Loss: 0.0002\n",
      "Epoch : 8/10, Iter : 100/234,  Loss: 0.0008\n",
      "Epoch : 8/10, Iter : 200/234,  Loss: 0.0008\n",
      "Epoch : 9/10, Iter : 100/234,  Loss: 0.0005\n",
      "Epoch : 9/10, Iter : 200/234,  Loss: 0.0002\n",
      "Epoch : 10/10, Iter : 100/234,  Loss: 0.0006\n",
      "Epoch : 10/10, Iter : 200/234,  Loss: 0.0002\n",
      "Wall time: 1min 9s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#修改学习率和批次\n",
    "cnn.train()\n",
    "LEARNING_RATE=LEARNING_RATE / 10\n",
    "TOTAL_EPOCHS=10\n",
    "optimizer = torch.optim.Adam(cnn.parameters(), lr=0.001)\n",
    "losses = [];\n",
    "for epoch in range(TOTAL_EPOCHS):\n",
    "    for i, (images, labels) in enumerate(train_loader):\n",
    "        images = images.float().to(DEVICE)\n",
    "        labels = labels.to(DEVICE)\n",
    "        #清零\n",
    "        optimizer.zero_grad()\n",
    "        outputs = cnn(images)\n",
    "        #计算损失函数\n",
    "        #损失函数直接放到CPU中，因为还有其他的计算\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        losses.append(loss.cpu().data.item());\n",
    "        if (i+1) % 100 == 0:\n",
    "            print ('Epoch : %d/%d, Iter : %d/%d,  Loss: %.4f'%(epoch+1, TOTAL_EPOCHS, i+1, len(train_dataset)//BATCH_SIZE, loss.data.item()))\n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEXCAYAAAAX7LteAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOydeXgT5fbHv9nXJmlLNxBB2axQEAQRkU1QcOGiXFFRdnDBiyKK60+9KKJXUNGrV1C4gODFS1Uum6gFWRQUrIjI0rKWvXubpmn2ZH5/1JlOtknSpk2C5/M8fWhn3sy8M0zOmXPes4gYhmFAEARBEHGIONYTIAiCIIhgkJIiCIIg4hZSUgRBEETcQkqKIAiCiFtISREEQRBxCykpgiAIIm6RxvLkeXl5WLlyJSQSCR5++GHccMMNguO3b9+OZcuWQSQSYdq0aRg4cKDXfrvdjldeeQU33HAD7rjjDgDAr7/+ii+//BIymQx2ux0ulwtisRhqtRp///vfuc+Wlpbi7bffRlFREfr164dHH30USqUy+hcdxzAMg5qaGlRWVqKmpgZ1dXWoqalBdXU1KisrUVtbC7vdDofDAYfDAafTCYvFgrq6OlitVjgcDrhcLrjdbq/jikQiSCQSSKVSyOVyyGQySKVSyGQyyGQyqNVqpKSkQKfTISkpCXq9HhqNBgaDAXq9HkqlEkqlEhqNBnq9HjKZLEZ3qHlxuVwwGo0wm82oq6uDyWTi7q3VaoXNZoPZbEZtbS0sFgv343A4YLfbYbPZ4HQ64XK5uB+PxwOPxwM200QkEgEAd9/591ahUEAmk0Gr1UKv10Ov10On00Gn03G/p6enQ6/Xc8dJNGpra1FVVYW6ujrux2KxoLa2FrW1tdz9ZX9n76nNZoPdbofT6YTD4fB6xkUiEfdsy+VyqFQqJCUlcT/8+2cwGGAwGLjfk5OTL4nn2W634+LFi6iurkZVVRVKS0u559dms3HPqt1u555p9ll1u93weDzo3r07FixY4HfsmCgphmHw8MMPY8mSJRg+fDicTif69++P119/Hc8//3zA8TNnzsT777+PYcOGAQAGDRqEl19+Ga+88goAoLi4GH/961/x008/YeHChdxnT506hddffx1XXXUVtFotJBIJPB4PbrrpJm7MDz/8gNtvvx2tWrVCnz598Nprr2Hp0qX48ccfYTAYwr6umTNn4tChQ1CpVDAYDEhJSeGErkqlglarRXJyMvfApqSkICUlBRqNBlJpdP4rPB4PrFYramtrYTKZYLFYYDKZYDKZYDabUVpaitLSUpSUlKCyspLbV11djeLiYthsNsHji0Qi7svIfiE1Gg1UKhUUCgUkEgkkEglEIhFEIhEYhoHb7eZeEFjl5nK54HQ6OUVnNBrh8XjCukalUgmDwYDU1FRotVpoNBqkpKSgVatW3Jc/PT0dqamp0Gg0nJBghYNKpYq6kHU4HCgvL0dVVRUn4CorK1FZWckJO7PZjOrqaphMJtTU1KC2tpYTlGazGRUVFWHfAwBQqVRQqVSQy+VQKBRQKpXcCwD7IxaLuR+g/rvEPiOlpaWc8rNYLJxAdjgcgueVy+VIT09HWloa0tPTkZWVhYyMDGRkZECtVsNgMKBVq1ZITk5Gq1atYDAYoNVquTk0FYZhYLfbuRckVtGwL1jFxcUoKSnh/i0pKUFVVRX3fxEOCoUCWq0WKpUKUqkUSqWSU+JyuZx7xgHA7XbDZrNxL282m437/lmt1pDnUqvV0Gq1SEpK4u5pamoqUlJSoFarkZaWhlatWnHPul6vR3JyMqfwonFfGYaBw+GAxWKB2WyGyWRCeXk5qqurub/Za2JfXIuLi1FeXo6ysjKUl5cLHl8ikUCtVkOhUHDygv+sSiQSWCyWgJ+NiZLasGEDli5dig0bNmDkyJEAgEWLFuGJJ57AlClTkJGR4TU+Ly8P77//Pj7//HPcfffdAIDly5fjoYcewrRp09C2bVs8+eSTqKyshFqt9vos+/ePP/6I5ORkv7k4nU5MmDABQ4cOxerVq6FSqVBRUYFu3brhgw8+wIsvvhjRtdlsNhiNRhw+fBhGoxG1tbV+lkUgZDIZFAoF5HI51Go195arUCi4/0SxWAyPxwO3280JE6fTyQk5VtCEQiKRID09Henp6UhKSkJWVhays7ORmZmJrKwstGrVirNm9Ho9UlJSkJycDJ1OB6lU2ixv0R6Ph3ujNRqNqKurg9FoRE1NDWw2G2w2G2fZsW/DVVVVnNVx8OBBVFVVwWQywW63h7x+jUbDKVlWELGWnVgs5pQtKwDcbjfcbjenaNk5ORwOmM3msIQfK8BZKyUpKQkZGRnQaDRISkri/k80Gg23jf1Csz+sMFMqlVET+r44nU6YTCYYjUZOONXU1KCmpgalpaUoKytDWVkZKioqUFxcjEOHDqGsrAxOpzPoMUUiEfeCwAp6mUzGPeOs0BeLxRCJRJwF6HA4YLVaOeHJvoWHqkEgFouRnp6O1q1bIysrCzk5OUhJSUHr1q2RmpoKtVrN3We1Ws1Z8VqtFlqtNmrWjdvt9nopMRqN3H01Go2orq7m5ERtbS3Kyspw5swZ/PLLLzAajUEFt+991Wg03H1l5QirBCQSCTcX9hm22+2w2+2wWq2c9R5OXQepVMq98GVkZKBLly7o378/2rRpgzZt2nAvJxkZGdDr9Zwck8lkjZYbolhUnBgzZgysVis2bdrEbbPZbMjIyMBrr72Gxx57zGv8+PHjUVxcjK1bt3LbnE4nsrKy8Oyzz+Lpp5+G0WiESqVCRkYG/v73v2PWrFkAgE8++QQzZszAli1bsH79epSVlWHo0KG49957IZFIsHPnTgwePBgnTpxAhw4duOPPnj0bGzduxNGjR73mUlBQgMLCQojFYigUCu4tsWvXrgH/ExiGgcVigdVq5d6ka2pqYDKZUFFRgerqau5NkHWlsaYx62JgTWKGYTjXGf+Lzn6xWKtGrVZzrgbWktDpdNBqtUhLS0NqamrCumvCwWKxoKysjLu3rIDlC12z2cwJQNaCYH/YFwH2ngPgFBfr1mHdZHK5HFqtFikpKdwbLyvskpOTkZaWBo1G06xKJR7weDyce4d1+bCWJP/+s24e9gWLfcbZe83+sApLoVB4KWj2+WafdfZv9jlPTU3llP2lcL89Hg8qKio4K5DvgjcajaiqquJe6tjnl315Yj0WrHXOf4YVCgUUCgX34qTVaqFUKjnZwd7LlJQUaLVaTok3hxeCpXXr1hg1ahQWLVrktT0mltT27du91oOAehfOFVdcgZMnTwYcP3PmTK9tMpkMHTt25MYbDAburYXvoisuLkZdXR369euH3r17w2AwYPLkyfj++++xePFi7NixA507d/ZSUACQnZ2Nf/7zn3C73dybCADk5uZizpw5fnNs27Ytzp49i86dO2PQoEFYsmQJgPo3HfbL1KpVK7Rv3z6ie0VEjlqtRvv27eletyBisZhzP7Vp0ybW07lkYC3C9PT0WE+l2VEqlaitrfXb3uKvGgzDoKqqCmlpaX77dDodTCaT3/Zwx1dXV4NhGC93YWlpKRiGwfLly5Gfn48tW7bgo48+wtKlSzlfdbBjO51OP9dRMNcda5bL5XJUVVUFuXqCIAgiEGq1OuAaXotbUiKRCMnJyaipqfHbV1NTEzBQQWh8x44dub8rKysBwEtJ3XfffRg4cCDuuusubtvo0aMxefJkHDp0CMnJyTAajQGPza5Z8MnOzsaoUaO4YAB2YZE9t0qlCmuxlCAIgmggmOyMibuvdevWKCoq8tpmt9tx7NgxzJ49O6zxLpcLBQUFmDZtGretrq4OQL0VxNK3b1+/47lcLgD11k/r1q1x7tw5OJ1Or8XSQ4cO4ZprrvHzv44dOxZjx44Nem1yuTzkwj1BEAThTTDZGZOVxeHDh+N///ufV7jt5s2bYbPZcP311wccv379ei9X25YtW1BbW+s1ns1r4l8oGxbKZ+PGjZBIJOjbty9uueUWmEwmbNu2jdvvdDqxfv36gAouFGwEHkEQBBE+wWRnTJTUlClTUFRUhMmTJ6OgoAC5ubmYMmUKhg0bhk6dOgGAVyjxpEmTUFxcjPHjx+PIkSNYu3Ytxo0bhwEDBqBbt24AgLNnz2L37t0AgDVr1uD48eMAgMceewyDBw/GxYsX4XA4sH79esyaNQt33XUXMjIy0L59e4wYMQKPPPIIvv76axw8eBC33347zp8/j4ceeijiaxOLxWGFchIEQRANBJWdTIz49ttvmSuuuIIBwABgxowZw5SUlHD7VSoV07p1a+7vbdu2MR07duTG33XXXcz58+e5/ZMnT2YMBgOj0+kYnU7HvPrqqwzDMExBQQHTrVs3BgAjkUgYAMwdd9zBVFVVcZ8tLy9nHnjgAe7YV1xxBbNu3bpGXdeAAQOYwYMHN+qzBEG0HOerLUyV2R7raRB/EEx2xiRPisVut+PEiRMBw1b/7//+D1lZWZgxYwa3zeFw4MSJE9DpdLjsssvCPo/H48GOHTtQXl6Oa665Bl26dAk47vz58zCZTOjYsSPkcnmjrunGG2+EXC73ch8SBBE/lNfa8cbXBVj76wVcnaXD5pkDYj0lAsFlZ0yV1KVI//79oVQq8d1338V6KgRBBODuRT/ilzPV3N+n/3F7DGdDsASTnYmfkh1n+Cb/EgQRXxTXNJQOa2NQCYwkWpJgspOUVJSx2+1QKBSxngZBEEFwuhsiyORSEoHxQjDZSf9DUcZms/3pWnwQRCLh9jSscMglJALjhWCyk/6HooxvUjBBEPEF35JSyck1Hy8Ek52kpKKMw+FodGQgQRDND8+QQrKaXijjhWCyk5RUlCFLiiASB4OaXijjBbKkWgir1epXlJYgiPiBX41TKSN3X7wQTHaSkooibOO3SFrOEwTRsvBrRicpY1Jjm/BBSHaSkooibAtmvV4f66kQBBEEsbhBS+lV5JqPB4RkJympKML2pSIlRRDxi4RnSpElFR8IyU5SUlGkoqICAJCamhrjmRAEEQy+JaVVkJKKB4RkJympKFJdXV8PjJQUQcQvPB1F7r44QUh2kpKKIuzbQEpKSoxnQhBEMKTiBrFHllR8ICQ7SUlFEdavmpycHOOZEAQRDAnPlErVUp3NeEBIdpKSiiIWiwUAoNFoYjwTgiCCwS8qm0ZKKi4Qkp2kpKJIaWkpZDIZdDpdrKdCEEQQNDwXn5ai++ICIdlJSiqKlJaWIj09HWIx3VaCiFf4gRN81x8RO4RkJ0nTKFJcXIzMzMxYT4MgCAE81Is87hCSnaSkokhZWRmysrJiPQ2CIATwkJaKO4RkJympKFJeXo5WrVrFehoEQQjAgJRUvCEkO0lJRQmGYVBWVob09PRYT4UgCAFEoHWoeCKU7IxpaMsPP/yAVatWQSwW46GHHkKvXr0Ex+/ZswfLly8HAEydOhXXXXed136n04k333wTffr0wfDhw732HTp0CJs2bYJMJsOoUaPQsWNHbt/HH3+MqqoqMAwDu90OkUgEi8WCMWPGoHfv3mFdS01NDRwOBykpgohzxBQsEVeEkp0xsaQYhsETTzyBgQMHorCwEAcOHMC1116LhQsXBv3Ms88+i379+uHw4cM4fPgw+vbtizfeeIPbX15ejltuuQUvvfQSCgoKuO0ejwfPP/88cnJy8PHHH2P+/Pm46qqr8OWXX3Jjli5dirlz5+KTTz7B2rVrkZubi127dnGx++FQVlYGAMjIyIjkVhAE0cK4eO3jidgTUnYyMWDTpk0MAGbNmjXctoULFzJKpZIpLy/3G79161YGALNy5Upu26JFixiZTMZcvHiRYRiGGTt2LNO+fXtGpVIxCxcu5MZ98cUXjEajYT777DPG4/Ewdrudufnmm5nu3btzY4YMGcJMnjy5SdeUn5/PAGA2bNjQpOMQBNG8DH17B9Pu2U1Mu2c3ReV4i3ecYJ7+/DfmbGVdVI73ZyOU7IyJJbVy5UoMHz4c99xzD7ft4YcfhlQqxZo1awKOHzRoEMaPH89tmzJlCrRaLVavXg0AePfdd3H48GHI5XJ4PA1vSnfeeSfOnz+P++67DyKRCHK5HMnJyZBIGjpyVlRUIDMzE5988gmeeuopvPjiizh37lzAuRcUFOB///sf1q9fj2+++QZ79uxBaWkpampqAFCbDoKId6wOd1SP98H2E8j95TwGzN+O9b9diOqx/wyEkp0xUVLbt2/Hbbfd5rVNpVLhyiuvxIkTJ8IaL5fL0alTJ258eno6FAqFX3dHiUTi9feWLVvw5ZdfYvTo0dy24uJizJ8/H48++ij27duHTz/9FNnZ2SgsLPSbS25uLkaPHo0777wTt956K/r164f//ve/MJlMAICkpKRG3BGCIFoKhSy6Ys/CU3qf/Xw2qsf+MxBKdra4kmIYBhUVFQEXyfR6PadV+YQ7vrq6GgzDBPRtut1uzJ07FyNGjMDtt9+Op59+GgDgcrlQUVGBDh064OjRo9ixYwcKCwtxxRVX4K233gp4nEDzYG80lUQiiPhGI49uvNhlySrud6ebwtsjJZTsbPHoPpFIBIPBEFAZ1dTUBDT5hMZ36NCB+7uqqgqA/wJcaWkpxowZg19++QXvvPMOHnvsMa78hkQiweuvv45x48bhsssuAwAolUqMGjUKmzdv9jtndnY2Ro0aBbfbDbvdjurqalx22WU4cuQIN1eCIOIXD9OgSFxuD6SSpr2rK6UNSwcUlBE5rGwPJjtjEoKelZWFM2fOeG1zOBw4fvw4nnzyybDGu91uFBYWYsqUKdw2s9kMwNu36fF4cMstt8BqtWL//v3o0qWL13FEIhGef/55v3MyDMMdj8/YsWMxduxYv+0//fQTALKkCCLeMdtdXr8b1PImHU8l5ykpqmYRMaySCiY7Y7Imdcstt2D9+vVgeG80eXl5sFqt6Nu3b9Dx/ICI7du3w2QyeY1XKpUA6hUey48//ojff/8dn376qZ+CYuGPB+oV2zfffBNwLsEwm82Qy+WQyajTJ0HEM/w1pFqbS2BkeGgUDUrKTUoqYkLJzpgoqSlTpuDo0aOYPn06Tp8+jU2bNmHy5MkYNGgQp0hsNhtcrvoHaNKkSTh37hymTZuGoqIifPPNNxg3bhz69u2LHj16AABKSkqwb98+AMC6detw+vRpAMDBgwchk8lw8OBBvPnmm3jxxRfxwgsvYNu2bQDqXYR6vR7Lly8HwzCoqanB9OnT8euvv+LBBx8M+5qcTicpKIJIAByuhpfdGquzycdTyUhJNYVQsjMm7r6cnBysXbsW06dPx0cffQQAuOOOO7B48WKIRPXZ4K1atUJqairOnDmDLl26YP369Xj44Ye5ihPDhw/HRx99xI1/+umnkZubC6lUijlz5sBut+PVV19Fp06dIJfL8cwzzyAlJQVarRYejwdr165FYWEhUlJS8PTTT+Phhx/GY489hrq6OiQlJeGDDz7AwIEDw74mu93OWXIEQcQv/AKzFWZ7k4+nJCXVJELJzpiVRfrLX/6CoUOHoqCgAAaDwatMEVBf9ohfFffWW29FYWEhCgoKoNPp0KlTJ6/xq1atwqpVq/zOM2zYsIBrS3xeffVVTJ06FTt37oRKpcKIESMiDiWvq6uDWq2O6DMEQbQ8/HWjaFtStCYVOaFkZ0xr92k0mqC18d577z2/bWq1Gtdee22zzKVdu3aYMGFCoz9vs9nIkiKIBIBv7URjTUotJ0uqKYSSnVQFPUrYbDaoVKrQAwmCiCn8Vh0mW9MtKX4Len4wGBEeoWQnKakoYbFYSEkRRIJRZ2+6JZXcxBD2PzuhZCcpqShB0X0EkXhYolDHL0kZ01WThCeU7CQlFUXYKhYEQSQGNZamu/vUvDJLbLQxERlCspOkapQgXzRBJAZiniKJ9poUvadGTijZSbeUIIg/FVJeZ16bs+m19vSqBleVhCypqENKKkqIRCKvsk0EQcQnEp6SsjiaHjiRplUEPDYRHqFkJympKCEWi0lJEUQC4K2kmh44oeNZUnJeRXQiPELJTlJSUYKUFEEkBhLewlE0lBQ/mVcuJZEaKaSkWgipVMoVxCUIIn6RSxosKXMU8qRkvH5Udmd0W9P/GQglO0lJRQlSUgSRGPALwkZDSfGJRi3APxukpFoIUlIEkRjwXXIOlyeq9faiUQvwzwYpqRZCJpPB6aS3KIKId3zXjfj9pZoKxfZFTijZSUoqSiiVSthstlhPgyCIECh8lJQ7ion4Hkrqj5hQspOUVJRQKBSw25veQI0giOaFvyYFRLe9hoyi+yImlOykOxol5HI5HA5HrKdBEEQIVD5KykM9oGJKKNlJSipKqNVqWK3WWE+DIIgQaBTeVcuj6e7zVYBEaELJTlJSUYK90ZTQSxDxjVbRfK01+Im9RHiEkp2kpKKEWq0GAAqeIIg4R61oPkXCT+wlwiOU7KQ7GiWSkpIAALW1tTGeCUEQQvi65KIZNk4FZiMnlOwkJRUltFotAMBsNsd4JgRBCKGRe7v7pFG0fsiSipxQsjOmfY/z8/Px6aefQiKRYNq0abj66qsFx+/fvx8rV64EAEyePBndu3f32u92u/Huu++iZ8+euOmmm7z2bd26FevWrUNSUhKmT5+Oyy+/3Gt/bW0tPvzwQ5w+fRrXX389xo0bB4kkfLeAUqkEAAqeIIg4R+vT7l0maZr143Q3rKX45mARoQklO2N2R59//nlcd9112L17N7Zt24acnBwsWrQo6Pg5c+bg2muvxc6dO7Fz505cc801ePfdd7n91dXVuOOOOzB79mwcOHCA2+7xeDB+/HjcfPPN+O2337B27Vp07twZ69ev58b8+uuv6NSpE95++20cP34cjzzyCPr374+6urqwr0elUgEgJUUQ8URVnQPzvjqCvq9vxZYjpQAAnVLmNUbRxPYa/Hp90cy5+rMQSnbGREnl5eXhH//4B1asWIH8/Hzs378f8+bNw+zZs1FVVeU3/vvvv8crr7yCRYsWYd++fdi3bx/eeecdPPfccygtrX/wZsyYgd9//x1KpRIiXnfMZcuWYc2aNfj222+xa9cuFBQUYOrUqZgxYwacTifcbjceeOABdOvWDceOHcPWrVtRWFiI48ePY/HixWFfEykpgogvTpTVYtD87VjyQxFKTXb8eLICAKCUNYg9iVjU5HUko6VBSdlcVAU9UuJSSa1YsQJDhw7FxIkTIRKJIBKJ8Pjjj0MkEmHNmjUBx/fv3x8PP/wwN3769OlQqVRYvXo1AGD+/Pk4cuQIlEqlVyjj8uXLMX78eNxyyy0A6nuXPPvsszh//jy2bNmCvXv3orCwEO+//z4MBgMAoF27dhg3bhyWLVsW9jVpNBoAiMj6Igii+Thwrga1vCrnbNt4fp5UNPKaymsbqiU4XWRJRUoo2RmTNalt27bhueee89qmVqvRoUMHHD9+POD4Rx55xGubQqFA586dufFt2rSB2+1GTU0N9Ho9gPqL3rt3L2bPnu312csvvxx6vR7Hjx+H2WxGhw4dkJ2d7TUmJycHixcvhsfjgZjXJK2goACFhYUQi8VQKBQwGAxo27YtdDodAIruI4h4wdf1xrr1+HlSepW3668xlJsblBTV7oucULKzxZUUwzAoKytDZmam3z69Xg+j0ei3PdzxRqMRDMNwY6uqquB2uwU/W1VVFXS/w+GA1WrlND0A5ObmYs6cOV5jJ06cyG0jS4og4gMG3gqDdfOpeAm3BnXTlZTR0lDSh1RU5ISypFrc3ScSiWAwGGAymfz2mUwmTqvyCXd8ZWUlACA9PZ37HDsu2GeFji2VSjl/KYvb7e9zrqur48IoSUkRRHzCFpY18Kwn32KzjYHfgp4sqcgJJTtj4u7LzMzE2bNnvbY5nU6cOHECM2fODGu8x+PBsWPHMHHiRG4bG2fPKietVgu1Wu332ZKSEhiNRvTo0QNHjx7FuXPn4Ha7vULOCwsL0b17dy9XHwBkZ2dj1KhRcLvdsNvtqK6uhkajgcFggFgsRllZWSPuCEEQ0Ubkk6bLuvkMajm3LRrJt9UWKizdFELJzpgETgwbNgwbNmwAw3vr+O6771BXV4e+ffuGNf77779HdXW113iFQgEAXAMtkUjEfZbP+vXrIRKJ0Lt3bwwbNgxGoxG7du3i9rvdbmzYsAHXXXed31zGjh2LdevWYePGjcjLy0N+fj5WrFgBqVSKVq1akZIiiDhB7KOAkv4IPecrpmg0PDTWNUT3Ub2JyAklO2OipCZPnozDhw/jqaeeQllZGbZt24YpU6bghhtu4AIYXC4Xp5QmTpyIoqIizJgxA6Wlpdi5cyfGjx+PXr16oWfPngCAiooKHDlyBACwefNmXLhwAQAwbdo0bNq0CfPnz0dlZSW++OILPPPMM7j33nuh0+nQqVMnDBw4EI8++ih+/vlnXLx4Effffz+OHz+OqVOnRnRdWq2WAicIIk6Q+igpfug5CxMF91ydoyGCUCwiNdUYhGRnTJRUz5498dlnn+HTTz9FRkYGhg4diuzsbPznP//hcpxSU1PRsWNHAEDXrl2Rm5uLL7/8EpmZmRg8eDCuvPJK5ObmcuP/9re/4Z577gEAPP300/jggw8AACNHjsT777+PuXPnolWrVhgzZgyGDRuGf/3rX9x8/vOf/6Bt27bo27cv2rRpg+3bt2P58uXo3bt3RNel0WhoTYog4gSpTyWJQEm7rigk39bZSUk1FSHZGbOySPfccw+GDx+OAwcOIDk5GTk5OV777777brRu3Zr7+6677sLQoUNx4MAB6PV65OTkeCXtrlmzBqtXr+bejPjrSzNmzMC9996LI0eOoHXr1ujUqZPXuS677DJ8/fXXOHDgAGpra9GzZ09uMS8SNBoNLBZLxJ8jCCL6SH3WkxUBLCmbs+nJt1W8ZF5fFyMRHkKyM6a1+/R6PQYOHBhw37///W+/bTqdDgMGDAh6PKFae2lpaRg0aFDQ/SKRCNdcc43AbEOTlJRE7j6CiBP83H0BLClRFCyfugAJw0RkCMlOqoYYRfR6PWpqamI9DYIg4B9eHihxNxq19uy8UkjUqqNxCMlOUlJRRKfTkZIiiDjBt7mhb/XzaMEvhdTUiup/VsOdnqwAACAASURBVIRkJympKJKcnBywYgZBEC0P370nEYu8knhZkqKguKy8da2mVlT/syIkO0lJRRGtVguLxeJV4JYgiNjAL3+kV8kCBjVEo/+ThReCHijMnQiNkOykOxpF2OZdNpstxjMhCIKvMIIVkm3qkpTHw8DpbjiInJoeNgoh2Ul3NIpQ/T6CiB/4bThSNfKAY9TyprnnHG7vN39y9zUOIdlJSiqKpKamAgDKy8tjPBOCIPjuvmSekopGlQkW3+hAuYREamMQkp10R6MIe6Orq6tjPBOCIPgKgx8gwa9azs9xagy+Vc9pTapxCMlOuqNRhDVZ2WrsBEHEDn6iLl+X8JWUzdm0ICffdX5+118ifIRkJ93RKJKUlASAuvMSRLzBLyzBj8ZzuhuvpLYfLcPOo97uqVgpKY+HQbHJhjYGVejBcYiQ7CQlFUVSUlIA1FdkJwgifuAHNPDzmnwDH8LB4nBh1prf8O3h0gDniY1zav63R7F450n86/5euL17Vkzm0BSEZCe5+6JIWloaAAqcIIh4gx/PUGdvUFIud+RBFN8fqwiooICGxootzeKdJwEA878tjMn5m4qQ7CQlFUXkcjm0Wi2qqqpiPRWCIHjw16RMtoaq5a5GJN4LVU5vrtJL4dJKq4jp+RuLkOwkJRVltFotBU4QRJzB7/NksvKUVCMsKX5BWV9iZUmxXJ2li+n5m0Iw2UlKKsrI5XI4HI5YT4MgCB785oZGXv8neyPaxzsEFJsuSGWLluLKNE1Mz98UgsnOiJSU0+mkyLUQKJVKKotEEHEAP2nXa02KF93XmMAJu4C7T6eMrZJKS0pMdx8QXHaGraTefPNNaDQarjHghQsXMHbsWLRv3x7PPPMMFVX9A1JSBBEf8K0kOy8fysbLk2pM9QkhxRaNquqRwr+GVM2fWElpNBo4nU6sW7cOADBhwgRs374dY8aMwYIFC7B+/frozTaBIXcfQcQHtbYGi6ma5+Kr81JSkR/XI1CV1rfRYktQy6uakZYUuEZhItBkd9/o0aMBACdOnMCpU6ewbds2fPzxx1iwYAGuvvpq7N69O3qzTWCkUilcrqaVWiEIounwk3Yr6+zc7/w8KZeHwao9Z6J2TlUMlFSVuUGwJ6sTV0kFk51hK6nWrVvj+uuvx9y5c/HFF19AKpViyJAhAACZTAa73R7iCH8OJBIJ3O7gPmuCIFoGB8/dV8OzpPjuPgD4cPsJ7neGYVBpbrwsi0Vn3hpetGKsQ+CbQjDZGVHgxBtvvIFTp07h2WefxV/+8hckJSXhzJkzOHLkCHr27Bm1ySYyEomE1ucIIg7g1+WrsjRYGyab99t6VV3Dvrwjpbj2ta34YNvxRp2TXy+wpSivbVCqiVyFPZjsjEjtDh48GEePHkVBQQH69u0LoN79d/PNN+P+++8P6xgejwdr1qxBfn4+OnXqhMmTJ3MNr4KN/+KLL7Bnzx5ceeWVmDp1KlQq7/pUxcXFWLp0KUwmE0aPHo1+/foBqH8r2r17N1wuF+RyORQKBRQKBaRSKUQiEdq2bQulUomVK1fC7XbD5XLBbrdDIpHAbDZj/PjxaN26dSS3iCCIOMHMW6upsTrh9jCQiEV+eU783Kbc/HMAgLfyjmHGTZ0CHjcWikiIcp7lF29ziwZhKymHw4HCwkJ4PB4MHjwYQL0C6d27NzZt2hTWzamursaIESPw22+/4dprr8WyZcuwYMEC7NixA5dffrnfeJPJhNtuuw0///wzevfujRUrVmD+/PnYsWMHrrzySgDAF198gcmTJyM5ORlpaWl46623MHPmTLz77rs4d+4chgwZEnSN6OWXX8ZLL72Ep556CmazGTqdDnK5HG63G5mZmbj55psjVlIejwdSaeKa3ARxqWB1NnzvGaZ+XSo9SelXMYIfBRhOnpMkQBv6WFJSc2lEEweTnWHbhi+88AJ69OiB++67DwCwb98+5OTkwGAwYMSIEbBarSGPMWvWLBQXF+PAgQP48ccfUVRUBJ1OhxdffDHg+GeeeQanTp3Cr7/+yo3PzMzEc889B6Degpo4cSLGjRuH48ePY9++fdiwYQPee+89/Pzzz7j88sthMplQXl6OCxcu4PTp0zh06BDatm2Lq6++Gk8++SSkUik3B3ZcSUkJfvvtN/Tq1Svc28PhdrshkVB3ToKINQ6fRF220oTvdr5lVWoKLfClcaakqi2XRjRxMNkZtpJi86N27NgBhmEwceJEyOVyrFq1Ctu2bcN///tfwc/X1dXhs88+w6uvvoqrrroKAJCcnIyZM2fis88+80sSttvtWLVqFV5++WV069YNAKDX6zFr1ix8+eWXqKqqwpo1a6BSqbBw4UIoFPX5ASNHjkSvXr2wdOlSAIBKpUKrVq3QunVrtGvXDkuXLkVVVRXWrl0LvV4PoL7ybmZmJj7//HPMnTsXH330ESwWS7i3xgtSUgQRH/hWhmDXqJw+251uBq4/cp8qwgiaiDdLymy7NKKJg8nOsP1SI0eOhFQqxVdffYU+ffrg8OHD+P777zFgwAC8/fbb2L9/PyZPnhz087t374bD4cCoUaO8tvfo0QMulwunT59GTk4Ot33v3r2wWCwBx3s8Hpw8eRLbtm3D8OHD/da0evTogWPHjvnNobCwEP/85z/x1ltvoUuXLgDqlafZbMYTTzwBl8uFq666CidPnsSCBQuwb98+TpGxFBQUoLCwEGKxGAqFAgaDAWq1Gm3btkVycjI8Hg/E4sRdvCSISwXfyhBsF95AybgOtwdSiRhltaGVlDTOghMq6y4NSyqY7AxbSen1egwfPhxz5szBbbfdBrVajeuuuw5AvdUTah2mpKQECoUCycnJXtvZv33bBpeUlAAAMjIygo4vKSnhlI3vmEDVdP/+97+jbdu2+Nvf/uZ3ns6dO2Pz5s3IysrChQsXkJOTg6VLl+Kpp57yOkZubi7mzJnjd2y9Xg+j0Qin0wmZLLalUQiCANw+Sbdscm+gZFx2LFvXTy0P7g1RxyAXSgh+VfdEJpjsjOiV4J133oFer8fHH3+M6dOnQ6FQYN++fSgsLMSAAQMEP6vX62G32/0yik0mE4CGzoz88UC9pRNsvF6vD1hL0GQyQafzrgZ87tw5fPHFF5g1axbk8oaEt/T0dDz++OP45ptvkJVV3yysTZs2GDZsGPbu3et37GA5UOzNJSVFEPGBy0cZsUm87gBlJnz1VmuBDrfx1iLe6rg08jKDyc6I7nbnzp1x4MABmEwmzqKRSqV4/vnn/dxyvmRmZgIAzp8/z0XmAcDRo0chlUq5dSrf8efOncPVV1/tNV4kEqFr167IzMzE2bNn/c519OhRdO/e3WvbkiVLoNPpMHXqVK/tSUlJeO+99/yOIZPJUFlZ6bc9Ozsbo0aNgtvtht1uR3V1NcxmM6d8XS4XKSmCiAM8PsqIdff5bgf8a/hl6YOnxejiLGFWqL9VIhFMdkbsXBWLxSguLsbXX3+NvXv3olOnTpg3b17IdZiePXtCr9dj48aNXts3bNiA7t27++U+devWDWlpaQHHZ2dnQ6fT4aabbsL27du9rKnS0lLs2bMHffr04bY5nU4sXboUkydPhlarDXmNdrsdu3fvDpigPHbsWKxbtw4bN25EXl4e8vPzUVBQgJMn6ztjWq1WwbwvgiBaBl93H5s3FSjXftS/dmPfmYYlgrYp6qDHVceZJeUbCJKoBJOdESmpU6dOoVevXujatStGjhyJG264Ae3atcOSJUtCflYul2P8+PGYN28etmzZAovFgjfffBOrV6/GlClTuHFsxrFEIsHEiRMxf/58fP3117BarVi4cCGWL1/OjR81ahSUSiUmT56M0tJSFBUV4c4774RGo8Fdd93FHXPjxo0oLi7GuHHj/OZ16NAhtG7dGvn5+QAAi8WCadOm4fz585gwYUIktwdA/Y32VbgEQcQei4Bb7EylBet/u8j93SEt+MusVhFfa1KXSv5uMNkZ0SvBE088AaPRiG+//RY333wzrFYr3nvvPTz00EPo0aMHF0gRjNdffx21tbW45ZZbANS71GbOnInp06cDAH755Rf06dMHa9aswT333IM5c+bAaDTitttuA1CvuKZPn46ZM2cCAFJSUrB+/XpMmTKFcw9eeeWVWLt2LQwGA3fe1atXo1evXgEto86dO+O6667Dddddh/bt2+PChQuQyWRYsmQJsrOzI7k9AOqTnvlrXgRBxAe++VG+8GW9kLvPEGdFXGVxFm3YWILJzoiU1JEjR/DXv/6VUzJqtRrPP/88li9fjtWrV4dUUklJSVixYgVmz56Ns2fPIicnB23btuX2t2nTBtnZ2ejYsSOA+vYgS5YswZNPPomioiJ07doV7dq18zrmwIEDcfDgQfz4448Qi8Xo37+/34U++OCDyMjICFgVQy6XY926ddi1axf27NmDlJQUjB492kvJhQvDMKirqwvLpUgQRPMi9vm+s6HnwSyPCl4ot0Gg8gR/n1wiblTjxGiikCa+khKSnREpqaFDh2L16tWYNWsW2rRpA6A+hLu0tJT7Oxy6devGJejyycrKwpEjR/y2Z2dnC1o1KpUKQ4cODbp/+PDhIed044034sYbbww5Tgir1Qq32+0XqUgQRMsj9km6ZfOmghVhvWhsqJqTJNBhl5/Mm5akwAVj6Go7zUm8RRs2BiHZGZEKfvnll6FQKNChQwfcfvvtuPfee9GtWzekpKTgwQcfjNqEExU2PN43/J0giJbHt3wRW6PPV3mxlJkaEnk1AutO/MK1vsEZzhhYVbFoDxJthGRnREqqTZs2OHToEBYsWACdTgez2Yxnn30Wv/zyS6PcY5caRqMRAOheEEQcIAni7gu2hlNW21C3Ty/g7uO3xvB19dXZW75EkVwaX4EcjUFIdkZsJ2o0Gjz22GN47LHHANS7+/bs2YPbb7+9idNMfGpqagDAr5QSQRAtj2+NPdbqkQexPPih3EIuNH4/Kt/SS1anGy39iqqSNShdh8uDLUdKMeSqNKjlieMGFJKdTV5x+/rrrzFy5EhcvHgx9OBLHNZkJSVFELFH6qOM2HzdcKLhhIIR+BUe6nzC2mNR7DVF0xAo9sSa/fjb6l/x7tbGNW2MFUKys8lKasCAAWAYBlu2bGnqoRIetoSTRqOJ8UwIglD51Nhj7SR5GNFwQv3xhCo88Fu5txTJvJD4zQfra5F+9Xtxi8+jKQjJziYrqQ4dOqBt27b45ZdfmnqohIcto+RbRJcgiJbH12XHlj7SNjEaziyw7iS0r7kwqP3Xz0wxUJZNQUh2NllJsXX0ArXG+LNRVlYGwL9yO0EQLU8wiymQUI8EoYKusSj2mqpRcL+zdQVrY6Asm4KQ7BR8pbBYLPjqq6/AMAwkEgkkEgkUCgXUajU0Gg20Wi3S0tKQnp6O33//vXlmn0AYjUYoFAoqi0QQcYAySNSbb3X0SBFK3vVdo2oJ+C3vk5Qyr8COREFIdgoqqX379uHee+/1qxAciEGDBjV+hpcIgVqEEAQRG3wtJjbar6kFWX1zo/jUxqC3E99ibKorM1YIyU7BKxowYACcTidcLhfcbjfXnsJiscBiscBsNqOmpgZ1dXW4/vrrm2XyiURFRQVSUlJiPQ2CIACofBoXskrKHagMegQIJewKFbFtLvjJvBl6JY6W+vfYi3eEZGdItcu6+Vio5E9wqqqqkJqaGutpEAQBf6tC9kc7odomusPsAoVqY9HbScmLYszUKQRGxi9CsjPxKxPGEXV1dRR+ThBxgkIqBj+fVyJhLammuft8E3j5xEJJ8ZswJsdZhfZwEZKdpKSiiNlspgroBBEniEQiKHjBE2wtvzCW2AURCryIReAEv7KEbwJzoiAkO0lJRZHKykpakyKIOIKfK8VWmrA20dpxC2i5WISg8+sM8tuTuGLcQiQShGQnKakoYjQaSUkRRByRzIvwY6PgmrNSeSwKzKp5ASL8ihclJlug4XGJkOwkJRUlnE4nbDZbQgeWMAyD3ScqsOT7U7C7Wv6NkCCiTRJvvUb2h7svnLJIjSUWZZH4JZyO8SL7qnhNHOOZULIzMYPq45BEr4B+osyMF9YexM+nqwAAnTK0GNwlPcazIoimwW9eyLr7fFt4RJOmRg42lbOVFu53oyUxSiOFkp1kSUWJRC8uu2jHSU5BAYCriQmPBBEPpPIqhMv+sKCa+mT7tqXn09T1rqZSZWmwnhLFkgolO0lJRQmbrd7/q1QqYzyTxuFb6kWSoFFCBMFHzeuwy67dhJNwK1Rlx7fjLx+HQA5Vc6L8o6eUzdlw/kRRUqFkJympKJHoSsoXeRg9dwgi3uG362Cj4CyO0C45odJJQv2ohBJ9m5O0JP8k3moLKSmCR6KvSfkSTmM4goh3tIqGNSnNH/lE4Vg7QkVkfTv+en0uRgFH7VP9XWWXyppUTAIntm/fjm3btiEzMxOTJk0KuY7zww8/YMuWLUhLS8OkSZP8okDq6uqwYsUKlJSUYOjQoRg8eDC37/Tp0/j+++8hlUrhcDjgcrkgFouhVCpx//33c+PsdjtWrVqF06dPo1+/frjtttsEG5/5YjQaAQAGQ0s3j44Ovt8734ZxBJGIpGob1qTYSL9wQtBrrM6gxVqF1qSaWMyi0WTo/K0QUwyK3TaGULKzRZWU0+nEAw88gM8//xzdu3fHuXPnMG/ePOTl5aFbt25+410uFyZNmoT//Oc/yMnJwYULF/Daa6/hm2++Qc+ePQEAe/fuxejRo2GxWHD55ZfjtddewwMPPICVK1dCLBbj+++/x8SJEyGXy6FUKiEWi+HxeHDLLbdwSqqgoAAjR45EaWkpOnbsiDfeeAPDhw/HunXrIJeHV2Yk0QMnpGJvy4nvyyeIRCVL3yC8tcrwLalKsx1tDIFb7gitSXmaWs6ikVyW7D/XWCQWN4a4CpyYP38+Nm/ejO3bt+PAgQM4ffo0srOz8fjjjwcc/9577+HLL79EXl4efv/9d5w5cwa9evXC3/72NwD1vsx77rkH3bt3x6lTp3DgwAH88MMPWLNmDdatWwcASEtLAwBcvHgRNTU1qK6uRk1NDT7//HMAgMfjwf3334/09HQcP34c+/fvx/79+7Fr1y78+9//DvvaWJM1US0pudT7i5eoJf8Jgg8/BF33x++2MJSUkKtMqPRQjHQUrmjlL+Cb2jerpQglO1tMSTEMg48//hgzZ87k3HE6nQ7PPfcctm/fjhMnTvh95uOPP8ajjz6Km2++GQCg1Wrxwgsv4KeffsKhQ4eQl5eHCxcuYMmSJVzb4RtvvBG33XYblixZAqC+BLxarYbNZsPy5csxf/587Nu3jztHfn4+fvvtNyxatAiZmZkAgO7du+O+++7jjhEOtbX1SXSJmszra0nplE3rXkoQ8QA/mVcll8DjYcKypDYfLMbKn04HjPJTBGmmGEvStP6BE7Gy6iIllOxssdflkydP4uzZsxg9erTX9l69egEAjh49io4dO3Lbz58/j2PHjgmO/+GHH9CnTx9cdtllfmNWrFgBACguLobT6USHDh2gVquh1Wrx7LPP4o033sBzzz2Hbdu24YorrkCPHj38jvHpp5+CYRivtamCggIUFhZCLBZDoVDAYDAgJycHJpMJYrEYarW6aTcqRih4WfhiUUNIK0EkMnyPgEYuFQyI4PPf/HMAgL5XpKJLprfwVDRjxYrGkpTAL5WhZGeL3e2LFy8CANq0aeO1nTXxKisrwxqv0Wggk8lQWVmJixcv+u0HgOTkZO54rJKaPXs2ysrKcPbsWcybNw9z586F0WgUPIbVauXCI1lyc3MxevRo3Hnnnbj11lvRr18/nD59GlVVVTAYDBCL4+8BDgelT6huJEEjBBGv8AvMKmXiiK2LQEEWSoGgIoHlqmbFtwsxILx2Fk+Ekp0tJlHZ1sBms9lrO/u3b5n2YOPtdjucTie0Wi10Op3ffqDefGSPN3jwYMyfPx9z586FVFr/wE6bNg0WiwUHDx4UPIZEIvGL3Xe7/Rcj1Wo1LBZLwlpRgHfIeaL2pCEIIRgm8jWjQHX+1PLgSkooPL05CeSeVyRIhG4o2dli7r6MjAwA9W48vlvv+PHjAICcnJyg4/mRf/zxhw8fxp49e/zOdfz4cXTv3h0AMGrUKL/9CkW9/7a6uhoZGRm4cOGCn1vv+PHjyMnJ8bMosrOzMWrUKLjdbtjtdlRXV0OlUsHpdEImS1yTW8OL5tMqKWiCuDTgrz/V2l0RK5GkAN8FISUlj9F6VaBo3ERJIwklO1vMksrMzESXLl3w1VdfeW3ftGkTkpOTvRQXUO9u69GjR8DxGo0GV199NYYMGYLDhw/j9OnT3H673Y68vDz06dMn6Fx27NgBoD5AYsiQISgvL0d+fj633+Px4Kuvvgp4jLFjx2LdunXYuHEj8vLykJ+fj8zMzIRXUjpeTxp+fxqCSGRqeblCpkZUKC8sqcU7eUdxwWjltukEvh+xWq8KlHyfKBG6caOkRCIRJk2ahEWLFmHjxo3weDzIzc3FggULMH78+IBrIJMmTcLSpUuxdu1aeDwerFu3DvPmzcMDDzwAiUSCQYMG4fLLL8eDDz6IixcvwmQyYdy4cSgtLcUDDzwAAJg9ezbGjx8Ph6O+RMihQ4cwc+ZMDBkyBO3bt0e3bt3Qs2dPPPbYYzh58iSsViseffRRHD58GBMmTAj7+lwuF+dOTET4Xy6hN0WCSCSqeaHkjalQvnjHSfxz2wn0/8c2vLbpCDweRlD4x1Nn3ET5HoeSnS0qVZ988kmcOXOGc8ExDIPRo0dj3rx5AOrdb2lpaZgzZw5efPFFzJgxA0VFRbj77ru58SNHjsSCBQsAADKZDGvXrsWECRO44Ifk5GQsX74cV111FQBg5MiRuOuuu9C6dWsYDAacPHkS2dnZWLZsGYB65fnZZ59h/Pjx6NixI0QiETQaDd59913ceOONYV9boltSfNdAIkcKEQQfvvVktrsQqQrhVzVfuqsI9/RpGzBxliWeal76uu1/O2fEqxsP49VR3dCtTfyUbwslO1tUScnlcixatAgzZsxAYWEhOnXqxK0dAfVrRR06dEDnzp3rJyeV4r333sMjjzyCI0eOoEOHDrjmmmu8jnnttddi//792LZtG+x2OwYNGuSVFDZo0CAUFRVh9erVqKioQK9evTBixAhIJA1CuUuXLvjpp5+wY8cOmEwm9O/fH+npkfVScjgcYVeniEdUvLcuypEiLhUsPCVjdbojWpNSysRQ+qwx2Z0ewbUeVRxZL77f4zv/tRsA8NL6Q/jfo/1jMaWAhJKdMfFPde3aFV27dvXbrlarcfToUb/t2dnZyM7ODno8uVyOESNGBN2v1+sxffp0wTlJJBIMHTpUcIwQie7uM6gaHpIUDSkp4tLAzlNSDpdHsO6eL2lJCvgOd/sEWPkST5YUv5cWH1/FG2tCyc74uaMJjtvt9rLOEg2dquEhIXcfcanAL4HkcHkisqRaaRURrzHFU/eAQO07ACAliPKKFaFkZ/zc0QSHYZiETeQFAAMvNypRFlwJIhT86L46R2Qh6MlquZ9lVFBsEvxMPAVOBFNS6brA22NFKNmZuFI1DknkKg38iKVACYwEkYiYrA0RfRZ7ZNF9OqUUUh8llfvLOcHPxFO5vGBRiPzK8PGCkOwkaRRFhFpOxzv8N0xTI0J1CSIe4XfhDaf6OR+JWOxnSbGNE4PBjwbk89qmI1i2qyii8zeVYEEcmw+WoLzW3qJzCYWQ7CQlFUUSWUnxqYvwjZMg4hV+CLorzOKyLDVWh59XIZQrPFCF9aMltVi6qwivbjoS0fmbSrAoxN/OGdFn3lb8a/uJuJFZpKRaAIlEErCuXyLijPCNkyDiFSNfSUXYX8locfpZI3qVDHM2HA76mUAFbC/yqlW0JKGWHxZ8exQny/3rlrY0oWQnKakoIZVKLxklZXNdGtdBEFV1Du53T4RKqs7hhsZHSWXolNh5rDzoZwJVHj9fbYnovM1BsB5aTnfsLalQspOUVJSQy+Ww2+PLz9tYrA6ypIhLA36HXXeEri2z3enXlsOglgm24whUefx8dWwsKT78tTk+8dDOI5TsTNzs0zhDpVLBao39w9hY+G+ZcfDcEkRUMFoaLKlIl1/cbsbP3dc2RS2YC8V+j1b+dBrvbT2Obm30cdFVgF/DkE+sWovwCSU7Y3/3LhE0Gg3q6upiPY1Gwy8fQ+4+4lLA5fY0KVK13Gz3q86QqpELpmi4GQZ/+8+v+OpgMQBg57FyXJUZuC16S3K6MrBsksZBbmco2Rn7GV4iqNXqhLak+DkkJTWXhtuS+HNT5/B+2WKthmDlgnxxuhnIfBTS7hOVgkVqlVIJp6BYKnnrYrEiWMh5PCQfh5KdpKSihEwm49qBJCIW3hd6a0FpwLbZBJFI2HxyllirITtLF/YxFD6uvYVbj+HA+Zqg4wO1cY9FTlKSTyJvpTmwbNLEQc+pULKTlFSUkMvlCa2kKuu8v0j5p6tiNBOCiA5WH0uKtRqys8J3vylkkYlIIVdg6xas9NDGp51IqckWcFysmjTyCSU7Yz/DSwT2RsdLclyklJm8lVRRReKurxEEANh9wq4XfHsUgxZsx69njWEfwze6D6gvlxSMYKHeQPBaes1BK633uaotgZVAPFRtDyU7Yz/DSwSFQgGGYeByJWa1hiqfhzhWCYgEES0CuazPVFqw70x12McIVGGiXaom6HhhJdVyllSyz7pboK7ESUopxHEQ3RdKdpKSihJJSfUuBJNJuEpyvOJbCul0ZewTEAmiKTiisK4aSEkJdeZ1CiQMt2SftitaeStSc4BSZ77WVqwIJTtJSUWJ1NRUAEB1dfhvafGExcd/f6ykNkYzIYjoEI3yXiqZv2uvfavglpTbE/ycmfrgyi3aZPi04+DXMGQJ5MqMBaFkJympKJGcnAwAqKpKzIADfksDALhAHLo+owAAIABJREFU7j4iwYm0wkQg+M1AWTqla4OOd7qZoFUc2qWomzyfYPiu5/haSb4voUB8BE0AoWVnfMzyEkCv1wMAamqCh6fGM77uvkAPNUEkEu4Ia/UF4vZ/7vLb1lZA2Xg8jFcDUT6p2ubriOtbg883FyxQC5GkOKiEAYSWnaSkooRGU+8CSNSqE5YgfXAIIlFxRaF4ak0AN1lygFwoFg/DID1IFF+qpvnWgOw+VWJ887VsAV46Q/XGailCyU5SUlEi0S2pSLuWEkS8E6htRjTQKYUDIILlVjWnJeVrKaVpvSMJnQHWypJbMJBDiFCyMyaq9KeffsKOHTuQmZmJ+++/HwqF8BtGfn4+vvvuO6SlpeH++++HSuW9AGmz2bB69WqUlpbipptuQt++ff2OUV1djR07dkAmk2Hw4MHQahv8yt999x1qa2vhdrvhcDggEolgsVhw0003oX379mFdE7v4V1FREdb4eCMakVAE8WdApwou3EUiEWRB1qSCNSGMBha7t5LSKLzPFUhfZ+jio418KNnZokrK5XJh6tSpWLlyJTp27Iji4mK8+uqr+Pbbb9G5c2e/8R6PB4888giWLFmCDh06oKysDK+88gq+/vprdO3aFQCwf/9+jB49GuXl5WjdujVeeOEFPPTQQ1i8eDHX9GvZsmV4/PHHYbPZ4PF4kJKSgg0bNuCGG24AADz++OM4cqS+a6ZcLofH44HBYMDSpUvDVlJ6vR5KpRLFxcWhB8chvomPBJHohOj512iESoaJ4N3Dik9zRtOZbN5uSalPkm6ge5ESZg3D5iaU7GxRd9/ChQuRm5uLzZs34/jx4zh79izatm2LGTNmBBz/r3/9C5988gnWrVuHEydO4OzZs+jcuTOmT58OAHA4HBgzZgyuvPJKnDp1CseOHcOWLVvw73//G5s2bQIAbNq0CQ8++CCeeeYZ1NbW4uLFi2jXrh2efvpp7jwZGRmYOHEiPB4P7HY7nE4nysvLMWrUqLCvTSQSISsrCyUlJU24Q7GDavURlxriZtBSBrUM1XWB214A9d1/j5cF7nbbnG0xAq2d8QnUXqQ5LbtICCU7W0xJMQyDDz/8EI8//jhuvfVWAEBKSgpeeOEFbNmyBadOnfL7zKJFi/DII49wysJgMODFF1/EDz/8gIKCAmzZsgVFRUVYtmwZ0tPTAQDDhg3DiBEj8NFHHwEABg4ciAMHDuDll1+GSqVCZmYmcnJyUFvbkAdUWVmJyy+/HHv37sXHH3+MVatWwWYLXOtKiOTkZBiN4ZdciSWr9pzBNa/m4Z0tx2B1uKMSCUUQ8URztKG4LFkFozV4nTmhihPNqaR8o3N9CVT+KB6Ky7IIyc4Wm2VRURFOnz6Nu+++22t77969AQCFhYW48sorue0XLlxAQUEBp2x8xxcUFGDXrl3o06cP2rVr5zdm5cqVAACdTodu3bpx+w4dOoTc3FzOGgOAkpISfPjhh5g7dy6ysrJQUVGBl156CT/99BOysrK8jl1QUIDCwkKIxWIoFAoYDAao1Wrk5ORAp9MlTODEjycqYLQ48c/vjmPT7xcDPsQMw3AuU4JINCItDhsOKRqFoCJSCpwzlpaUSi4BfILngkUhxgIh2dliltT58+cBAJdddpnXdjaRy3fR7MKFCwHHa7VayGQyVFRU4Pz58377gXoLLdAi3H//+1/0798f2dnZeOmllwDUr5OVl5dDLBZjz549uHjxIs6fPw+Px4O3337b7xi5ubkYPXo07rzzTtx6663o168fevToAZPJBJ1O52WhxTN88/9UeR3KArQTCJRbQRCJQnMUT01WywSVVKwcEqHagegDBHukxklZJACCsrPFlBRbn8k3Fp79m42VDzXe4XDA6XRCo9EgKSkpYGy92Wz2it6zWq2YNGkSxo4di/Hjx+OHH36AwWAAUO8PHT9+PPLy8riowPT0dIwZMwa7dvkn8rndgQV3dXU1UlNTUVZWFvwmxBG+Pmq29w6/wrPNSetUROIi1Oa9seiUMsEgo5KayJcJokGoDsTaAK69QNtihZDsbLFZZmRkAAAuXryIjh07cttPnjwJAF4uOd/x/H388YcPH0Z+fr7fuU6ePMl9hmEYjBo1Cvv378fWrVsxdOhQr7ESiQSffPKJ3zHUanXAWlLZ2dkYNWoU3G437HY7qqurYTabUVJSgszMTJSVlSWEm8zXLcG+HWoUUu6Bp2AKIpERcr01liSlVPB7YXHEJt8wWEQhS6AcrUDWVawQkp0tZkllZWWhY8eO2Lx5s9f2r776Cnq9Hp06dfLanpKSgm7dugUcr1ar0bVrVwwaNAgHDx7EuXPnuP1OpxPffvst+vTpA6A+RH3Lli1YtWqVn4IKBsMw2LVrl5/iBICxY8di3bp12LhxI/Ly8pCfn4+CggJcf/31yMjIgNvtRmVlZVjniSVan5Iorj/8FPyqz69uOtKicyKIaNIcloJBLUO5ObhrzRajVA5zCEsqUMVzoQaNLY2Q7GyxWYpEIkyYMAGLFi3Cd999B6A+PHz+/Pm4//77IQ4QiTNhwgQsXboU33zzDQAgLy8Pr732Gu69915IpVIMGTIEbdq0wfTp01FRUQGr1YqpU6fi4sWLGDt2LABg7969kMvl0Gg0+N///odVq1bhk08+QVFREYD6ooZdu3bF7t27AdQrqLfeegs7duzAxIkTI7pG1vorLy9v3E1qQZTSwOGn/FyOr35PzJwvggCApBCVIRrDuv0X8X//OxR0f6yiZIt5nXdlEn8vTrzkRAVDSHa2qFPy6aefxsmTJzFs2DCo1WpYLBaMGDEC//jHPwDUl8VIT0/H3Llz8cwzz2DmzJk4duwYbr31Vm780KFDuYAGuVyOL774AuPGjUNmZibEYjGUSiUWL17MWUGpqalwOBwYOHAggHo3HlBvqZ07dw4GgwFdu3bFwIED0aNHDxQXF6OkpASzZs3CyJEjI7o+dh3MbA6cJxFPBEssjHMvJUGETXO4+44UC/eLk4hEcKPlFdWF6ob+b4EsyPQWbLjYGIRkZ4sqKaVSiRUrVmDGjBkoKChAly5dcN111zVMRipFRkYGMjMzAdQroSVLluDRRx/FoUOH0LFjR/Tr18/rmNdffz0OHjyIr7/+Gna7HTfffDNatWrF7R8zZgz69+8PpVKJ5ORkiMViuN1uLtxRLBYjNzcXO3bswJYtW6BWqzFmzJiAFTBCodPpACRG48PT1B6euMSJxbqwRCwCggTFVprtSNHIm2VeFeaGNSl1gMKxwdafHC4PjFZHzJWYkOyMSXhH7969uXwnPhqNBmfPnvXb3rNnT/Ts2TPo8VQqFUaPHh1wn0gkQps2bby2SSQSpKSkeG0bPHgwBg8eHMbsg5NISmrNL+cCbo9G5WiC+LMSrJcUAFz72lY8MawTnhgW+QtwJATKD/Otig4Aub+cw+ubC1Brc+GbmQPQKSOpWeclhJDsjJ+Vs0sA1pWYqO06gIZQdCC+FlYJIlJ+Lmr5BqShEohPlUdfNnh81sEC5YfxA6JYnvnidxgtTrg9TMg8q+ZGSHaSFIoi7NtAoiT08mHfAPm5UV1i+GZFEE3hRFkt7v34pxY/L9/tFghpgKCGpuKbtxWoskWo4raxfiEVkp3xk811CcAmICeiklLKJDDbXV7VlDsHUVJnKuvw711FuDpLh/uuu7ylpkgQYVNSYw/YniLWyKJcT/BUuRkHL3iXEwq05BXI3ccn1kpKSHaSkooibJ8ri8USYmT8oVHUKyl+2/i0ALW93vr2KD7YfgIAkKVXkpIi4pLmanjYVGTS6FlSZyrrcNPbO/22B7r0UMVkmyNcPxKEZCe5+6IIGwIf72tSTICnONBDvHjnSVTyEhcZhsGHO05wf1eGcG0QBOGNQurbjJDBRztP4oNtxyM+1kff+3eOqD+m/7ZQreJjnUclJDvJkooyarUaVqs11tMQJFDh2GAP8Z0f7saWWYOglEkgEokglYi5Ekr6EC4EgiC8YV8GTTYnVuw+DbvLjX9try/1NuXGKwKGjwcjWJ3AQFZkqArsSXFQxy+Y7Iz9zC4xtFptTJN5iyrq8NHOk1DLpXh55NUBxwSq4hwsj+JclRWHL9bg2nb1IftquUSwCjRBxAPxmpSuVUjg9jAYsmAHKn3q7dmdHqjl9VGJr28uQN8rUvD8bdlBjxWsXp8jwpqbSQopxM3YRiRcgslOcvdFGY1GEzMltXDLMQx7Zyf+m38Oy3YXeYWT8wlkSSULmPt/XfQTTvzRbVTH812H81zbnG4s2nGS+zxBtATN0ZU3GihlEhy8UOOnoADA5nL/f3tnHhd1tf//12wMAwMMDJsiuOKCigouYW7hnqbmVpl52yzv16yr3frdupZW3zK/mmVpat0W9VrmkpqKZWqKmmuiAoLKpqCAwOzMPvP+/YHzkWFmWBRk1PN8POahfM75fOacM+fzfp/lfd5v/N+v2Zi65hjOFqqwJjUPx/MqPMaK8vROmSx2EBEOXy7Dv7el13lwP8DXO+YqnmQnU1KNjEgkgsVSewCy2rhSUYlZ6//CDyeuNnjGsuHElXr5DnN3YFdWh0fk0ctTYbLanGZc9Yl8uul0IRb/mo1hy1w3eBmMpsJLdRR8RQLszXQfJl1rtOLLg7lO15786jj6fLgPmdddAwLqPETjvaYyYOqaY3jmm5PYcOIqtvxVVGuZAu/QG7rOZMXrm87hoY/242rF7RuNeZKdTEk1Mj4+PjCbb9+g4NeMEvyaWYK3t6XjkaUHkV5U/0i/9Y2fk1vmOlpxZ8lXHYuNcLFE6zTjqk+k0SOXXYNPMhhNjbfOpCQigYsicqDSux/cmq12/GPj2QZ9z6mCW2GGHHtUnszQ7yRkBxFh8qo/sfVMEUo0Rhdz+IbgSXYyJdXI3OlMylptJnRNZcB3R/ORXVI/N0viGmcdPL2o649dcblWl5Kqer4AUvEt6yR33par83VqHvZeKOX+dmdVyGgYBrMNj395FG9vS2/uong1dfXN5mLDCdd3z4FKb/a49Hb5hg7/PX7r3ppeJmpDeHPw2jNa5jbdXRiP+mKzE7JLbp1tstpvf7+azaTuEgKBwGP03tvh57RrGPXZYRQqap9GX63Qu0z/Pc10LG46eFg9Omqwn8hpT8pXJEClyYpLpVoYzM51LlEb8WFKltM1vZmFo79Tfjp1FWlXVfjhhKuPS8YtmiIqb2NwPM+zqyat0Vqrwpi/PYMLqlhbdOCaOAavHcKkbtPDAz1/56I9WZi36Sxybrh3UCAU8CGp5s3Ccge+Pz3JTu/YMbuP4PP5dzRj8DT7URssiHZzvVChx0cpWdiT4brO7dBRJWojZH4izjWKxI1/MXeRO2sSKBE5zbhUegu6LviN+7vg4zHc/935Aqs0W+s8VMioHZWHTfSGUKw2IL1IjaFdIuq1ZHsvUvM8krfg7yNApYfBmslqr9MzxHWVAR3CAzwaRbnDoaRayiRu0787WoAJPaPQw81Ma82hqrNY29Ku4ZVHOuD1EZ1c8gRJRJwx1p1Y/nqSnd453LiHsdvtd+SK39MyRc2lPAfPfX/KrYIS8nm4VKrDc9+dxEOL9jutabvz4yVx44CyJr4igdOhv2sq5zMNf+bc2n9yt6mrNzXNTMpktWHNoVwcvlx23y8pRlUTNNYGmhrb7YRVB3ORtOgAXlr/F1IveX9wztvF0/vS3HhSUECV1W1dBkyamxF4rQ1Y7nO877UZSEz48mit/YEI2OBh9l594OqY6d0OnmQnG9Y2MjabDWLx7a/xeprGe1q+8GSGarUTRn6WeitfmQ45N3S4oTW67eCB9XCLklumQ5HS80HlC8Ua9O9QFcvLaHV9Gav7BWxMjucpsGhPNgAgqZ0cS6bEo1WwX5N8V3NTfcZrsNgQ0IBlrT9zK7D412zub5Xh/vUY0ty+6G4Hk9VW595wscoIxDQsArBjn8tdMEQHRFWyZFDHMO5azS0Gk4fZW/UyN2QZsiaeZCdTUo2M1WqFUHj7zeppr+H8NTXWHitAq2A/bDhxBY92a4GZA9vV+7n+PgLM+u9fyLmhc+vK/72dmXU+Y/Tyw7VO5x1LEIUKvVsz24IKPeJbud+8vRO01ZTfsbwKrD9+BW+N9nwI8k65oTXivV8u4G/926Bv25C6b2hEgv1uKSmjxY6GxKqrubJnsd6/s05vnUnVht1OiJLVPria/cMZ/Jkbg9mPdKj3c/19hChS6vHmlnO15vMVCfDd0Xz8mlGCE27CnFSabXh7Wzr+/WgXp2X7iMBbnbAhyrMmnmQnU1KNjMlkuqOZlKfZxsoDObhYemvzcsUfOfhvLZZCNREK+Mi5UWUe6u5E+m+ZpS7XalLXenNeeSX+tfU8Np5yH1BxwY4MDI4NQ/o1NZLay132Q66rDPjn5nMY37MlnugTg+N5FbhUqsWMpDYNKldTB25ccSAHu9OLsTu92Gkfjohgp/qZ5t8u1ZdsTG5mq7VR88B2Qz0T3EtIveSAakMwWe0e942qs+HEVbw0qP4DVKXejCFLDta5RFgfi9EfTlzFiLgIDOkUzl2rbsJ+J0rKk+y8935JL8doNMLX9/ZCMZutdmiN7td0qysoB57OVbjD3eypsfn5zLVa05V6C3q8vxcAsGxqD0xMaOWUfjSnHH/mVuDP3ApcuK7B2pum8gM6hEJtsKBHK5lb9y13ssTQUKw2OzZ5iGr89rYM7Dh7DdMfao2XBrVzsdTKLtFgW9o1PNe/LSKD6u4jJqsNm08XYUCHULQJ9QfgPJNSGyxoFVx3mZWVZsj8RC57EvUVKESEC8UadI4MbDZDi6sVegT5iVzO9OhMVny+/zIGdwzDwzeXmgHvNZyojYxranSODKxX3sFLDtb7ucdyKxq0h1UXNWNmVT+Wciff4kl2MiXVyOj1ei7KZEOZ/p8TjVyaW7hzhdSczNt0Diq9BU8/FIOT+Qos+e0izlc7uLy22lkuRziCt0Z3xsuD2wOoWhr5NbME3x7JR34Nty819+8W/5qNMq0J74/vWqsDT6PFBl9RlW81T8I4JaPEKTAkEXGbvcdyy6E32/BVah5+OlWIA68Phryaovo6NR9bzxRhzaE87JozAN2igtx+x5a/isDnVdVj/vYMAMC3z/ZGcucIp431ugLsAcDu88WY/cMZxLcKwgsD2jql1TecxZwf07DrfDE+mdIDkxJb1X1DI1Ko0OODXRew90Ip+rQJxuZZ/bk0IsKQJX+gXGfGiXwFvgj2Q3mlCQkx9dDcXsgfF8vQqwnK/nNa7YPHhvLPzecw+WY/0BotWLr3Epd2J0MYT7KTKalG5k6U1MmCpgt3fbZQ1WTPvl3e33UB7++6UO/8i/Zk4+XB7WGzE8Z+cQRZxe4POVc3EDKYq3wHAlUbw5890RMAuJmJgxnfnkTqpTIESURQGyx4tn8bvDM2zklZvb0t3WXP8GyhCr1ignEguxQF1VzCqA0WFCkNTkpKpb+lVP5zOA+fPdkLRITNfxVh9cFcvJLcAY/1aIl/bq7aO3iq761DB+/vvIDkzhFOM8nLpVoMrrbR7cBktXEziUs3Z+Dni9R4rYbXgvoIFIvNjl3niwEAJ/Ir6qWkdpy9hj+yb+CZpDZIbH1nQvfp/5zA1Zsb+NUPjQJVe5wORW002zB1zTGUaIzYMisJvdvc3b3CxmLZ75fqzuQFfHkwBzMHtsOU1c7Rj+/E0Ycn2Xnv7S56OWazGT4+DY/NcvDijSYozf1J5nW1RwUFAIKbb8qu89fR64O93PWzhSoMWXoQQ5YexIm8Cu56idrImd86nHl+/2cBVv6RAyLizNrdGbUs+e0iyrQmLN93Kx5QYLX9kLwyHVLSq4R89c1mh4PRt7el480t55FXXon92TecluB+OXud+7/jIHR1s///3Z2Fd3dkcGW22OxYlJKFru/+hq9T8/DLues4Vq2eLu1Uj6W76uf2IoPq3i8BgP8czsf2s9cxadWfeHHt6VpN5e12qtV7QmW1oww1zxBVPysU1zIQJZqq0BUr/8gBo2n5v18v4tPfL7kMHAR3oKU8yc5mmUmdPXsWBw8eRIsWLTBx4kSIRLWbP6enp+PAgQMIDw/HpEmTXCpisViwdetWlJSUIDk5GfHx8S7PuHz5Mvbs2YPAwEBMnTrVRWPb7Xbs2LEDBQUF6N+/P/r163dbdbtdw4mTbqxpGK60+dfuOvOs+CMHsRFSl5lDdbJLtOjXTo6/rigwadUxt3mW/X6JG9lueNF9f/gztwLfHMnHuWpLlY6zLONXHuWuzRve0cni7PDlcuxJL8bBi7fOpoQHiJ2UQvUzNQI+D5/vv+wy0l537ArWHbuCQ28Mwef7c7D1TJUz0ZrePtzxzo5MmKx2tA+TYkinMPyWWYoPUy5g1dOJ3FJkdUVWqNDjw91VM9+XBrV3Mj3WGi1YtCcbZ64onQTXvqxS5JZVolNkgNsyPPv9KVwu1eKTqT2Q2DoYPgK+01kZf7GQU+iFCgN0JitnSl3dT1x1ZfbHxTKoG7Bfy7g93PkgvBPTf0+yk0d38fSj3W7H7NmzsWbNGrRo0QIVFRVo06YNUlJS0K6dq7UKEeEf//gHvvjiC0RGRkKpVCIqKgopKSno2LEjACAjIwOTJ09GQUEB5HI5iouL8eqrr+LTTz/lOvsHH3yADz74AMHBwaisrIRMJsO2bdvQp08fAMCVK1cwceJEnD9/HpGRkSgqKsL06dPx/fffQyCo/was1WqFSCTCe++9h3fffbdBbVMf4cu4v5nQsyVeSY5tdo/xUxJbYcmUHrDZCUaLzcmrSHWCJCK0CpbgqkLv0eDHweJJ3TGhVxQul+qwO70YT/WJwbG8cvy/ra4WZRtfeggPtZMDAEZ+mupiNBQZ6Itvn+2DRz8/7PH7PhjfFe/sqPtYhYM2cj+n5VrG7TF/TBe82ICjMQ5qk513VUktX74cb775Jn744QdMmjQJJSUlePzxxxEcHIyUlBSX/GvWrMErr7yCdevW4cknn0R5eTkmTZoEoVCIAwcOwGKxoHv37ggNDcVPP/2Eli1bYufOnXj88cexa9cujB49Gtu3b8fjjz+OlStXYtasWdBqtZgxYwZyc3ORnl71ggwYMABarRY///wzOnTogNTUVIwaNQpffvklnn322XrXT61WQyaTYenSpXj99dcb1DZMSTEYt/hgfFdM6R2NpEX7oWyiWdHfklrDVySAyWrHm6M64Y0t57H75v4b4/apfiyjvtQmO++qkoqNjcW4cePwySefcNd2796NsWPHIj8/H23atHHK3717dwwaNAgrV67kru3fvx/Dhg1DdnY28vPz8eijjyInJ8dpJjZ69Gj4+Phgx44dGDZsGGQyGbZs2cKlZ2RkoHv37khNTYVUKkVCQgKOHz/utMT37LPPIjs7G8ePH693/fLz89GuXTt88803eP755xvSNExJMRh3mRcGtMU7Y29FrzZZbVh/7Ar+d/etpdJOEQH4/KleTt5b3CEW8nHsraF49ruTTlaqDyK7Xx2Ari3dW656ojbZedf2pAoKCpCTk4OpU6c6Xe/bty8AICsry0lJFRcXIyMjAytWrHCb/8KFCzh69Ch69+7tslTYt29fbNiwAQaDAYcPH8b69eud0rt27QqJRILMzExotVrExMRwz63+jK1btzqZGDvKmZ2dDT6fD7FYDJlMBj8/P3To0AEKRdW+UmhoKO4mfdoEO8WPqS8bXuyHtKtKJxNSd7w1ujP8fAQel086RQS4PcfFYHgzkho+LMVCAV4c2A5P9o3Bh7svoGe0DJMSWkEo4GNYlwicyK/At8/2cbFoO/vucFjthBB/H+yY/TB+OlWIf/3svIwZ4Cusc0m0Pozr0RItZRKsPuQ+JpU3EB3ScOvm2mTnXVNSV69WWUbFxMQ4XQ8JqTIVLStzdm5YWFjoNr9UKoVIJEJZWRmuXr3qkg4AcrkcN27cwI0bN2A2m13y8Hg8yOVylJWV4caNG4iJiXFxbCiXy6HT6WA0GiGR3LJq2rRpExYuXOjynQsXLsSgQYMAAIGB9TuQV539rw/Gqz+mgccDXk2ORb92cuzPKsXIrpFOVmHb064hOkSCxNYhSC9S44qiEmO6twCPx8M1lQHL9l5Cv3YhmNo7GulFalxT6RHXIggxcj/Y7YSfTheiS4tALrZMYutgCAV8HLx4A+EBvnionRyJrYMh4PPw5pZzSCtUoWvLIAyIDcXkxGgnR7Q6kxUSkQACPg83tEZs/esaMq6pMSA2FFN7R+NSqRZKvRk2OyEhJhgaowU+Aj6C/XygMVrQ96P9Tt4inuobAwEf8PMR4qvUPKf2GREXgUUTuwMA/rqixEvr/wIAPNu/DYrVBo8eM775W2+cLFBgfI8orDqUi53nrrvNBwCfPtEDJ/MVyLyuQatgCVLS3UdQ9cTCx+IQ7O+DIR3D8em+S/j+zwKn9Ee7R6JntAwxIX7ILavEkt8uNuj51ZmS2Ao2ojoPUAPAqqcTYLTaMPenW25xtv69P9qH+bv8BvcLreV+sNoI11QGRMkknFXky4PbQa23wGyz4/GEKLf3SsVCLJrobHz19YxE2OxU5bnlw9FY/Gs2vj6cjwCxEIG+Iu5oAI/Hw5N9YzCyayTWHivAyK6R6NIiEESE01eUaCmTIEomgdVmx89p1/DmlvMAgNHdIjlH0RN7RSEiyBc+N5XjYyuOoGe0DEun9ECH8KpwG6FSH3x5MBeju0XileQOEPB42HW+GB3CpZD5iaCoNGNfVinmDe+EG1ojyrVm8PlA79YhWHHgMvzFQvytfxuo9Bbk3NBh+jcn0CkiAPNGdMTIrpFY+EumS/9tG+qPXtEyPPdwWxzNLcfHe7Kd0rtFBWJ0txb18gNaE42mylrXreyku8Rff/1FACgnJ8fpukajIQC0efNmp+uZmZkEgDIyMpyum0wmAkDr16+n5557jkaPHu3Oe3D/AAAgAElEQVTyXR999BFFRkZSeXk5AaD9+/e75AkODqalS5fSv/71L0pISHBJ/+abb4jP55PNZnO6/s477xCqDlY7fZYvX07btm0jAHTmzJl6t4s3Y7fbSVVpbvLvSS9SUaGissm/pzp2u53KtUZKL1KR3mT1mM9ms3tMUxvMpNCZSG+ykt3ums9osdKOs9do48krbtNrlsday3dpjZZa7yci0hktdK5QSeuPFZCy0uQ2T6nGUOszbDY7VZosXHm1N5/pqfwlagN9nZpLlSYLGcxWMpitVKo2kMlic8mbV6ajPenFpDNaqExrJI3BTFqjhQoVlS7PL1Lqad2f+fTBzkw6e1Xpsbwn8iro1R/P0KqDObRs70WyWF2/t6lIu6qstWz1oa5+cbfwVI6rFZV0NKfstu5tCLXJzrs2kwoPr/L1VFxcjPbt23PX8/KqRsxdu3b1mL96Wn5+Ppc/IyMDZ86ccfmuvLw8dO3aFcHBwRAKhSgudt4MVSgUUCqV6Nq1K7KysnD9+nWXZb28vDzExcWBz3c2qezSpQvGjx8Pm80Gk8kEpVIJnU6H0NBQKJVVS27Bwffmifea8Hg8BNUR36Yx8OR5oSnh8XiQS8VOh23d4c4Nk4O6RoxioQDjerSsd3lqCyZbmwdrB/5iIeJbyWp14hteh0daPp/n5JVDevOZnogI9HWx5nIXCgaoGom3vXmIuvrqgLu6RckkeKYOn40A0LdtyF138uvAU6TbhnAnYX0aE0/liA7xq3P5rjHqUJvsvGuHeaOiotC2bVvs2bPH6brj7FKnTs7BtEJDQ9GlSxe3+X19fdGtWzcMHDgQ586dw7Vrt5Y8rFYr9u7di759+4LP52PAgAEuz/jttyqT2j59+mDgwIEoKSnBuXO3lkKICCkpKS77VADw1FNPYfv27di5cyf27t2LU6dOISsrC9OmTYNOVxU2Qyp1HwGTwWAwGK7UKjvveJ7WAN59910KDg6mI0eOEBHR77//TjKZjF588UW3+T/66CMKCAiggwcPEhHRwYMHSS6X0/Tp04mIyGg0UkREBE2cOJFUKhWZTCaaOXMmAaC0tDQiqlq2E4lEtGPHDiIiOnPmDLVq1YqGDx9ORFVT1c6dO1NycjKVlpaS1Wqlt956iwDQb7/91qD6LV68mACQTqdreOMwGAzGA0ptsvOuKqnKykqaMmUKAaDQ0FACQIMGDSKFQkFERGq1miQSCS1fvpyIqpTQtGnTnPInJSXRjRs3uGcePHiQoqOjSSKRkFQqJV9fX/rss8+4dJvNRrNnzyYej0dyuZx4PB7Fx8dTfn4+l+fMmTPUsWNH8vHxocDAQBIKhfTOO+80uH5vv/02CQQCr1lnZjAYjHuB2mTnXXWL5Ofnh02bNuHw4cO4cOECOnXqhMGDB3NrmjweDwEBAZy7drFYjA0bNmD27NlIT09Hhw4dkJyc7LQGOnjwYFy4cAE7duyA0WjE6NGj0bLlrX0APp+PFStW4IUXXsCpU6fQqlUrjBo1ymmvqVevXjh//jx27NgBjUaD5ORktx4w6kKr1SIgIMBr1pkZDAbjXqA22XlXD/Pe7/ztb39DamoqZ9zBYDAYjLqxWKq8irjz48qUVCNjs9ka5O/PmyAiqNVqVFRUQK1Wo7KyEmq1GkqlEhUVFdBqtTCZTDCbzTCbzbBYLNDr9aisrITBYIDZbIbVaoXN5hy7isfjQSAQQCgUwsfHByKRCEKhECKRCCKRCH5+fggJCUFgYCACAgIQFBQEf39/yGQyBAUFwdfXF76+vvD390dQUFCdDonvVaxWK1QqFXQ6HSorK6HRaLi2NRgMMBqN0Ol00Gq10Ov13MdsNsNkMsFoNMJiscBqtXIfu90Ou93OeXJ3jFQd7V69bcViMUQiEaRSKYKCghAUFITAwEAEBgZy/w8PD0dQUNA9u1qg1WqhUChQWVnJffR6PbRaLbRaLde+jv872tRoNMJkMsFiscBsNjv1cR6Px/VtHx8fSCQSBAQEcJ/q7SeTySCTybj/BwcH3xf92WQy4fr161AqlVAoFCgtLeX6r9Fo5PqqyWTi+rSjr9psNtjtdsTHx2PJkiUuz2bxpBqR1157DRkZGZBIJJDJZAgJCeGErkQigVQqRXBwMNdhQ0JCEBISAn9/fwiFjfNT2O12GAwGaLVaaDQa6PV6aDQaaDQa6HQ6lJaWorS0FCUlJaioqODSlEoliouLYTQaa30+j8fjXkbHC+nv7w+JRAKxWAyBQACBQAAejwcejwci4sz1rVYrp9ysVissFgun6FQqFez2+h0q9fX1hUwmg1wuh1Qqhb+/P0JCQhAaGsq9/OHh4ZDL5fD39+eEhEM4SCSSRheyZrMZZWVlUCgUnICrqKhARUUFJ+x0Oh2USiU0Gg3UajW0Wi0nKHU6HcrLy+vdBgAgkUggkUjg4+MDsVgMX19fbgDg+PD5fO4D3Axxf7OPlJaWcspPr9dzAtlsrj2Yoo+PD8LDwxEWFobw8HC0aNECERERiIiIgJ+fH2QyGUJDQxEcHIzQ0FDIZDJIpVKX4xy3CxHBZDJxAySHonEMsIqLi1FSUsL9W1JSAoVCwf0W9UEsFkMqlUIikUAoFMLX15dT4j4+PlwfB6oGpkajkRu8GY1G7v0zGAx1fFPVNohUKkVAQADXpnK5HCEhIfDz80NYWBhCQ0O5vh4UFITg4GBO4TVGuxIRzGYz9Ho9dDodNBoNysrKuCM2Go2Gq5Nj4FpcXMw5RKjpjKEmAoEAfn5+EIvFnLyo3lcFAgH0evcOftlMqhF57bXXcPr0aRiNRigUCqhUKmi1WpeZhTtEIhHEYjF8fHzg5+fHjXLFYjH3I/L5fNjtdthsNk6YWCwWTsg5BE1dCAQChIeHIzw8nFOiMpkMkZGRaNGiBUJDQ7nZTFBQEEJCQhAcHIzAwEAIhcImGUXb7XZuRKtSqVBZWQmVSgW1Wg2j0Qij0cjN7ByjYYVCwc06KioqoFAooNFoYDKZ6qy/v78/p2Qdgsgxs+Pz+ZyydQgAm80Gm83GKVpHmcxmM3Q6Xb2En0OAO2YpAQEB8PPzg7+/PwICArjfxN/fn7vmeKEdH4cw8/X1bTShXxOLxQKNRgOVSsUJJ7VaDbVajdLSUs6bS3l5OacIbty4wS3ZuIPH43EDBIegF4lEXB93CH0+vypUh2MGaDabYTAYOOHpGIXXJbb4fD7Cw8PRsmVLREZGIjQ0FCEhIWjZsiXkcjnX7v7+/vDz8+Nm8VKplPNq0xjYbDanQYlKpeLaVaVSQalUcnJCq9Vy7VpWVgaVSuVRcNdsV39/f65dHXLEoQQcKzvV+7DJZILJZILBYOBm7/VRBUKhkJMXERERXNtGRUUhKiqKG5xEREQgKCiIk2Mikei25QZTUk0MEUGv18NgMHAjabVaDY1Gg/LyciiVSm4k6FhKc0yNHUsMjikx3TxwLBAInF50x4vlmNX4+flxSw2OmURgYCCkUinCwsIgl8u9brnGarViwoQJCAwMRHBwsJNT4Yai1+tx48YNrm0dAra60NXpdJwAdMwgHB/HQMDR5gA4xeVY1nEsk/n4+EAqlSIkJIQb8TqEXXBwMMLCwuDv718vpWK327F9+3ZuOe/vf//7bbfB3cZut3PLO44lH8dMsnr7O5Z5HAMsRx93tLXj41BYYrHYSUE7+rejrzv+dvRzuVzOKfumUuJNCdVwKmC321FeXs7NAqsvwatUKm4wXFlZyfVfx+DJsWLhmJ1X78NisRhisZgbOEmlUvj6+nKyw9GWISEhkEqlnBJvilWIumBKyguwWCz466+/uCWbLl26NHeR7jpKpZLz4+jv71/vZZn7CdYGztzL+7u3i1KpRLt27RAWFobIyEikptbuff1BgO1JeQFZWVlISkoCAHTu3BlZWXVHVb3fcDgUBoDo6OhmLEnzwdoAOH/+PHr06AHgwXwXCgsLuSXBB01Be+Lemw/fh1Rfd35QXSqxNmBtALA2eNDr7w6mpLwAlUrF/f9+cU7bUFgbsDYAWBs86PV3B1NSXkB1azSxuHav3PcrrA1YGwCsDR70+ruDKSkvoPrZmAd1HZq1AWsDgLXBg15/dzDDCS+gc+fOeO+992Cz2dC5c+fmLk6zwNqAtQHA2uBBr787mAk6g8FgMLwWttzHYDAYDK+FKSkGg8FgeC1MSXkBJ06cwGOPPYbExES88cYbUCgUzV2kRmPlypWYNm0apk+fjilTpmDy5MmYMGECPv30Uy4PEWHHjh145JFH0LdvXyxZssTFB6FCocDcuXPRu3dvjBs3DidPnrzbVWkwp0+fxoABA1BQUOB0vaioCC+88AISExMxbdo0ZGdnO6UTEbZs2YIhQ4agX79+WL58uYs/wrKyMsyZMweJiYmYOHEizpw509TVuS1++uknDBkyxOna//zP/+CZZ57BtGnTMHnyZEyZMgVjxozBgQMHuDxmsxmffPIJ+vXrh0ceeQTbtm1z8S2XlZWFJ554AomJiXjppZdQVFR0N6pUb8xmM9asWYMnn3wSL7/8Mv7880+XPIcOHcKoUaPQu3dvzJ8/HxqNxim9srISCxYsQJ8+fTBixAjs27fP5RmnT5/G+PHjkZiYiLlz56K8vLzJ6tQcMCXVzHz11Vd46KGHYLFYMGLECGzevBm9e/d2Oi9xL5OXl4ft27fDYDBwTlwDAgLQs2dPLs/cuXMxYcIEtGzZEgMGDMCiRYswcuRIztKpsLAQXbt2xZYtWzB8+HDYbDb069cPO3bsaK5q1ckPP/yAgQMH4ujRo7Bardz1tLQ0dO7cGcePH8eoUaNw5coV9OjRA6dOneLyzJo1C1OnTkVMTAySkpKwYMECjBs3jhPSubm5iIuLw65duzBy5EjodDr07t0bv/32212vpyfsdjveeustPPnkk0hPT3dKO3HiBA4dOgSz2QwfHx/w+XzExMSgTZs2AACj0YhBgwZhwYIFSEpKQkxMDKZOnYoFCxZwz0hJSUF8fDyuXr2KUaNG4dixY4iPj3cZEDQXpaWlGDBgAF5//XXYbDacOHECDz/8sFOfXbp0KYYMGQKxWIyhQ4fiu+++Q1JSEuc5XaVSISEhAStWrMCQIUMgk8kwYsQIrFq1invG2rVr0adPH1RWVmLEiBH45ZdfkJCQgIqKirte5yajKUIBM+pHWVkZ+fn50fz587mwySqVikJDQ2nx4sXNXLrG4fXXX6eEhASP6adOnSIA9OOPP3LXsrOzicfj0a5du4iI6IknnqAuXbqQWq3m8sycOZO6dOniNtx0c2Oz2SgyMpLGjh1LACgrK4tL69evHyUnJ5PRaCQiIrvdTiNGjKDRo0cTEVFqaioBoO3bt3P3nD17lgDQ/v37iYho3Lhx1LNnT9LpdFye6dOnU0JCgte0x8WLFykoKIiGDh1KAQEBTmmJiYk0b948j/cuWbKEpFIpXbp0ibv21Vdfka+vL5WVlZHZbKZWrVrRjBkzyGazERGR2WymuLg4+vvf/940FWogy5cvp379+lF+fj4REVmtVurbty+NGTOGiIiuXLlCIpGIlixZwt1TWlpK/v7+tGbNGiIimjdvHkVGRlJxcTGX57333qOwsDAyGAykVCopMDCQ3njjDe5312q1FBkZSe+///5dqmnTw5RUM7Ju3TqSSqWk1Wqdrs+aNYsSExObqVSNyzPPPEPDhg2jRYsW0WOPPUbDhg2j9evXcy/Vm2++Sd26dXMRrgMHDqQZM2aQwWAgiURC33//vVP68ePHCQClp6fftbo0BJvNRocPHyYAlJ2dTURVggkAHTp0yCnvxo0bCQBpNBqaM2cO9enTx+V5iYmJNGvWLNJoNCQSiWjTpk1O6X/88QcBoJycnKarVAOx2Wy0dOlSCgoKcroeHR1N8+bNo7lz59Lw4cNp4sSJdPToUS69b9++9MorrzjdU1lZST4+PvTtt99Samoq8Xg8TgE4+Pjjj0kulzdZfe4Eq9VKsbGx9MwzzxAR0eeff07h4eFkMpmc8j355JM0dOhQstvtFB0dTR9++KFTelFREQGgPXv20E8//UQSiYSUSqVTntdee426devWtBW6i7DlvmbkyJEjSExMdPHR1a5dO69ZtrhTiouLsW/fPixduhQREREIDw/HjBkz8MUXXwAAjh49ikGDBrm4/3e0QVpaGgwGAwYPHuySDsBr24nP50OpVAIA59n86NGjEAgE6N+/v1NeR12uXLmCI0eOuNTVkaegoACnTp2CxWLBoEGD3D7Dm9rD0QaO+gNV+20lJSVYtmwZ9uzZg06dOqGiogKDBg3CmTNnYDKZcPr0aZf6+fn5oUWLFigoKMCRI0eclgcdtGvXjosg7U3YbDb885//xOXLl/Hcc88BqHr3k5KS4OPj45TX8TsXFRWhsLDQpR1atmwJHx8frh169OgBmUzm9hl0n5wuYod5mxGdTufSwYCqF7I+wQvvBcrKytC6dWscPnyY8+zdokULfPbZZ3j11VfrbANHuIqgoCCXdABe3U4VFRUQCASQy+UAqn5vqVTqEoW5el10Op1LXR15ysrKuPao2Wbe2h4VFRWIiIjg/lar1bBYLHj00Ufx888/QywWw26346GHHsKKFSuwbNky2O32WvuE1Wr1mA5UuRYKCAhouko1gMLCQkyfPh3Hjh3D6tWr8cgjjwCo6guhoaEu+Wv2+5r15PF4Tnk89RWj0eh1MeNuFzaTakbkcjk32q6OQqHgBNu9zhdffOGkoAAgOTkZ+fn50Ov1dbaB40WumcdhAenN7VRRUYGwsDAu+J5cLodGo3EypACc6xIaGlprezjqe6+0h0KhcFJSQUFB2LZtG7Zs2cL5puPz+RgyZAgyMzMRGBgIgUBQZxt4ShcKhQgMDGy6CjWA33//HfHx8VCr1Th58iRefvllLu12f2ebzQa1Wl2vvnK/wJRUMxITE4NLly45+esCqizAevfu3UylalwGDhzoEhvJYWZLRIiJiXExwQZutUGrVq0AABcvXnRJB4CEhISmKHajYDAYnJZyY2JiQES4fPmyU760tDSEhISgTZs2btuDiJCWlobExETExMQAgEuetLQ0CAQCLhaTt6DX653agMfjYcKECZBIJE75NBoNiAh8Ph/R0dEu9bt+/TpKS0vRu3dvxMTE4Pr161Cr1U550tLSEB8f77KE1hxcv34d48ePx6hRo3Dy5Ekna1agqi/U7NPArX4vl8shkUhc2uHcuXMgIq4dcnJyXAY995P8AJiSalbGjBmDkpISHDp0iLtWUlKCvXv3ckEQ73Vyc3NhNpudrv3444946KGH4O/vjzFjxiA1NRXXr1/n0k+cOIFLly4hKSkJYWFh6NevHzZu3MilExHWr1+Prl27ul3u8BakUqlTfKCEhARERkY61cVms2HDhg1ISkoCj8fDmDFjsG/fPqezLocOHUJRURGSkpIQHR2N7t27u22PXr16uQj/5qZmGxCRi+DV6XTYuXMnhg4dCqDqvdi0aRNsNhuXZ/369RAKhUhMTMSwYcPA4/Hw888/c+l6vR5btmzxmvdm69atEAqF+Prrr90qzTFjxuDSpUvcYAsA8vPzub0qgUCA0aNHY+PGjU57S+vWrUNwcDA6deqEMWPGQKFQOJ2dKi8vR0pKite0Q6PQTAYbjJuMGjWKIiMj6euvv6bNmzdT27ZtKSwsjMrLy5u7aI1CbGwsjR07ljIzM+ncuXM0c+ZMAkAbN24kIiK9Xk8dOnSguLg42rRpE/3nP/8hmUxGPXv2JKvVSkS3rN/mzJlDKSkp9PTTTxMAWrduXXNWzSN2u51+/PFHmjRpEolEIlq4cCGVlJQQUZUFmo+PDy1YsIB2795Nw4cPJx6PRwcPHiSiKhPimJgYio+Ppy1bttDq1aspICCA+vXrx1lAfvfdd8Tn8+n111+nlJQUmjx5MgGgzZs3N1uda2IwGGjlypXUt29fatu2LS1evJgsFgsVFhYSAPr3v/9N+fn5lJqaSv379yc/Pz/OMjEzM5N8fHxo7NixtHv3bpo/fz7xeDyaM2cO9/xZs2ZRQEAALVu2jH755RdKSEggX19fzpKyuZk7dy6Fh4fTnDlz6KmnnqLHHnuMJkyYQFu2bCGiKsvHpKQkiomJobVr19IPP/xAUVFRFB0dzR0tOHToEAGg6dOn0549e2jOnDkEgD766CPue8aPH09hYWG0evVq2rp1K8XGxlJwcDDX3+4HmJJqZrRaLc2dO5eEQiEBoNGjR1NGRkZzF6vROHbsGMXHxxMAAkBRUVG0evVqpzzXr1+nKVOmEI/HIz6fT88++yxdu3aNS7fb7bR161Zq1aoVAaCYmBgXk3RvwmKxUHJyMiUkJFCvXr0oMTGR0tLSiKhKOK1Zs4ZCQkIIAMXFxdHOnTud7r969SqNHz+eAJBQKKSZM2dSaWkpl+5Qgi1atCAA1LZtW9qwYcNdrWNdXL58mfr168e1wYABA0ij0RAR0bfffkvh4eFcn0hISHAxyz958iQlJCQQAAoMDKT333+fO1tGRGQymejDDz8kPz8/AkBJSUlOZuzNzc6dOykxMZGSk5Np0qRJNGPGDJoyZQrNnj2by6NQKOjll18mPp9PAGjixIl0+fJlp+fs37+fOnXqRAAoLCyMPv30U+5sGBGRTqejN954g0QiEQGg4cOH07lz5+5aPe8GzAu6l2C1WmGz2e7LQGd2ux0XLlyAxWJBt27dIBKJ3OYzm83g8Xge04kIBoMBEonknrdcctTFYZHmDpPJBIFA4GINWPMZ92J76PV6ZGVlISAgALGxsR7LbzAYIBaLOeOTmtjtdphMJq9b5mwIFosFRFTrXpper6/1d7bZbLBYLPD19W2qYjYbTEkxGAwGw2thhhMMBoPB8FqYkmIwGAyG18KUFIPBYDC8FqakGAwGg+G1MCXFYDAYDK+FOZhlMLwEk8mEwsJC2Gw2hIaG3hX/a+Xl5W4dnTIY3gKbSTEYzcyQIUPQvXt3+Pv7IzY2Fp07d0ZoaCg6derUpKE3CgsLERYWhqysrNu6/6WXXsLixYu5v41GIzp37oxLly41VhEZDDaTYjCam9zcXISFheHTTz9Fx44dIRQKUVhYiMzMzCYNOeHwFn7lyhV06dKlQfeazWb897//xcqVK7lrhw4dQk5ODiIjIxu1nIwHG6akGIxmRi6XY/DgwZgzZ85d/V6H9wJP3hzcUVhYCIVCgczMTBgMBoSFheHcuXMAqhwHx8XFIT8/H61bt3Yb84nBaChMSTEYzYzJZKrVHVZWVha++uorLFu2DBkZGSAixMfHu+Q7f/48lEolHn74YRdXSrm5udi1axfatm2LUaNGwcfHByaTCUBVkDyLxYK0tDRERUUhKirKbTkMBgO6dOmCyspK7tpjjz3mkq9nz56YM2cOPv/883rVn8GoDaakGIxmRqPRQCwWIz8/H8XFxSgpKUFpaSni4uIwePBgpKSk4Ouvv8Yff/zBzVqmTJmCH3/8EQKBAGVlZZg2bRoXsqFnz57Yv38/F7b9119/xeTJkyGTyaBSqdC2bVucPXuWCwGRnZ2NGTNmID8/HwKBAHv27MHw4cNdyimRSKBWq2G325GUlISRI0di4cKFAICcnBzExcXh/PnziI2N9YqYToz7hObybMtgMIiMRiPnwbrm5/nnnyciojVr1hAAGjduHJ08eZI2b95MAGjv3r1ERDRt2jRq0aIF7dmzhw4fPkxisZg++OADIiKqrKyk8PBweu2118hut5NKpaLVq1eT3W6nkpISAkAikYgmTJhAp0+fpoSEBJo5c2atZXbcd+LECe7aypUrKTIykgsnwmA0FmwmxWA0I3w+HxaLBbNnz8bYsWMRERGBiIgIhIWFcd7gHYEdt2zZApFIhD59+qBLly5ITU1F7969sXHjRnz11VcYNWoUAOC5557DgQMHMH/+fKxbtw52ux0ff/wxeDwegoKCuDDmdHMm1a1bN2zatIl7dklJiduyajQalJaWYufOnQgKCkJQUBAXZXjnzp3o2bMncnJyIJPJEBYW1qTtxnhwYEqKwWhGHFFrhw0bximZmvj7+wOo2hNyKK6wsDAoFAoUFBTAbrejf//+XP42bdrg1KlTAIBjx45h/PjxbkM4OJTU/PnzuedarVbodDq35Xj++eexdetW7u/OnTu75OnYsSOCg4NRUVFxz4UPYXgn7JwUg9GMOAS53W73mMdhVGG1Wrl/c3JyEBUVxZmRl5WVcfmzsrLQvn17AFWKjTxE43GEZ68+6wkMDERpaanb/OvXr8fFixchFAqxdu1aFBUVoaioCGvXrgUAnDlzBkVFRcjLy2MKitFosJkUg9GMSKVSyGQyFBYWOl23WCzg8/kQCAScifjixYsRFxeHX375BWVlZXjqqafQtm1bJCYmYt68efjkk09w7tw5rF+/Hnv27AEATJ48GTNmzMDQoUORnJyMkydPYtWqVVi3bh0MBgOAWzM1AIiJicGVK1dARC6KRiKRIDMzE0KhEJMmTeLuO3v2LOLi4tCrV68mayfGA0zzbokxGIynn36aAJBMJiOZTEZ+fn7E5/PplVdeISKiffv2EQCSy+UEgDp27Ejbtm3j7s/IyKCOHTsSABKLxbRw4ULOgMFut9M777zDGWeIRCJ66aWXyGKxUHZ2NrVv354L605ElJqaSgCorKzMbVlXrVpFs2bNcrr2zDPP0OLFixu7WRgMImLh4xmMZsdms2H//v3Izc2FWq2GUCiEVCrF8OHD0b59e+zbtw/Dhw+HQqGAVCrl9o9qPiM3Nxdyudytzz+NRoPc3Fy0bt2aM013BxEhJycHsbGxjVpHBuN2YUqKwfBy9u7di5EjR0KpVDIvDowHDmY4wWB4OQ6jCmaMwHgQYUqKwfByYmNjMWHCBCcDBwbjQYEt9zEYDAbDa2EzKQaDwWB4LUxJMRgMBsNrYXcEmm4AAAAnSURBVEqKwWAwGF4LU1IMBoPB8FqYkmIwGAyG18KUFIPBYDC8lv8PwyZ5zP8SqEkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.xkcd();\n",
    "plt.xlabel('Epoch #');\n",
    "plt.ylabel('Loss');\n",
    "plt.plot(losses);\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "准确率: 91.0000 %\n"
     ]
    }
   ],
   "source": [
    "cnn.eval()\n",
    "correct = 0\n",
    "total = 0\n",
    "for images, labels in test_loader:\n",
    "    images = images.float().to(DEVICE)\n",
    "    outputs = cnn(images).cpu()\n",
    "    _, predicted = torch.max(outputs.data, 1)\n",
    "    total += labels.size(0)\n",
    "    correct += (predicted == labels).sum()\n",
    "print('准确率: %.4f %%' % (100 * correct / total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "损失小了，但是准确率没有提高，这就说明已经接近模型的瓶颈了，如果再要进行优化，就需要修改模型了。另外还有一个判断模型是否到瓶颈的标准，就是看损失函数，最后一次的训练的损失函数明显的没有下降的趋势，只是在震荡，这说明已经没有什么优化的空间了。\n",
    "\n",
    "通过简单的操作，我们也能够看到Adam优化器的暴力性，我们只要简单的修改学习率就能够达到优化的效果，Adam优化器的使用一般情况下是首先使用0.1进行预热，然后再用0.01进行大批次的训练，最后使用0.001这个学习率进行收尾，再小的学习率一般情况就不需要了。\n",
    "\n",
    "## 总结\n",
    "最后我们再总结一下几个超参数:\n",
    "\n",
    "`BATCH_SIZE`: 批次数量，定义每次训练时多少数据作为一批，这个批次需要在dataloader初始化时进行设置，并且需要这对模型和显存进行配置，如果出现OOM有线减小，一般设为2的倍数\n",
    "\n",
    "`DEVICE`：进行计算的设备，主要是CPU还是GPU\n",
    "\n",
    "`LEARNING_RATE`：学习率，反向传播时使用\n",
    "\n",
    "`TOTAL_EPOCHS`：训练的批次，一般情况下会根据损失和准确率等阈值\n",
    "\n",
    "其实优化器和损失函数也算超参数，这里就不说了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "deep learning",
   "language": "python",
   "name": "dl"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}