AntiSym Comparison.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Antisymmetric Weights Comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comparison of different h values (1 and 0.1) for ResAntiSymNets with given backprop learning rates of 0.1, 0.5 and 1. The stability with regard to the depth of the net is considered. For stable archtitectures a maximum depth of 120 layers is considered.\n",
    "The hidden layers have 8 neurons, gamma is set to 0.3 and the net is trained for 100 epochs. The results are averaged over 10 computations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from Dataset.Dataset import makeMoonsDataset\n",
    "\n",
    "dataset_size = 600\n",
    "batch_size = 40\n",
    "test_set_size = dataset_size * 0.2\n",
    "\n",
    "train_loader, test_loader = makeMoonsDataset(dataset_size, batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning rate 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### h = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of layers: 2\n",
      "Max training accuracy: 0.9474999904632568\n",
      "Max test accuracy: 0.9450000000000001\n",
      "Time elapsed: 12.789678981999941\n",
      "Number of layers: 3\n",
      "Max training accuracy: 0.9856249928474426\n",
      "Max test accuracy: 0.9791666666666666\n",
      "Time elapsed: 16.46214900300015\n",
      "Number of layers: 4\n",
      "Max training accuracy: 0.9989583313465118\n",
      "Max test accuracy: 0.9966666666666666\n",
      "Time elapsed: 22.298688333999962\n",
      "Number of layers: 5\n",
      "Max training accuracy: 0.9989583313465118\n",
      "Max test accuracy: 0.9975\n",
      "Time elapsed: 25.69672525599981\n",
      "Number of layers: 6\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 28.811471888000142\n",
      "Number of layers: 7\n",
      "Max training accuracy: 0.9904166638851166\n",
      "Max test accuracy: 0.9891666666666666\n",
      "Time elapsed: 33.66371358399988\n",
      "Number of layers: 8\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 37.99518894700009\n",
      "Number of layers: 9\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 44.13530271700006\n",
      "Number of layers: 10\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 52.030954417999965\n",
      "Number of layers: 11\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 57.97349936699993\n",
      "Number of layers: 12\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 57.997160353000254\n",
      "Number of layers: 13\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 62.3626230660002\n",
      "Number of layers: 14\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 62.98143228400022\n",
      "Number of layers: 15\n",
      "Max training accuracy: 0.990625\n",
      "Max test accuracy: 0.9908333333333333\n",
      "Time elapsed: 67.46802209599991\n",
      "Number of layers: 16\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 70.90346432900014\n",
      "Number of layers: 17\n",
      "Max training accuracy: 0.9997916638851165\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 76.54248418800034\n",
      "Number of layers: 18\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 82.29970667899988\n",
      "Number of layers: 19\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 84.87181233599995\n",
      "Number of layers: 20\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 89.30745967799976\n",
      "Number of layers: 21\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 97.66976505599996\n",
      "Number of layers: 22\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 98.70815581200031\n",
      "Number of layers: 23\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 100.48527177100004\n",
      "Number of layers: 24\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 106.61353602899999\n",
      "Number of layers: 25\n",
      "Max training accuracy: 0.9916666686534882\n",
      "Max test accuracy: 0.9908333333333333\n",
      "Time elapsed: 116.69122896499994\n",
      "Number of layers: 26\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 114.44726604300013\n",
      "Number of layers: 27\n",
      "Max training accuracy: 0.9908333361148834\n",
      "Max test accuracy: 0.9908333333333333\n",
      "Time elapsed: 121.69212188099982\n",
      "Number of layers: 28\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 125.68200113900002\n",
      "Number of layers: 29\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 125.54167403500014\n",
      "Number of layers: 30\n",
      "Max training accuracy: 0.990625\n",
      "Max test accuracy: 0.9891666666666666\n",
      "Time elapsed: 132.14633950799998\n",
      "Number of layers: 31\n",
      "Max training accuracy: 0.940416669845581\n",
      "Max test accuracy: 0.9408333333333334\n",
      "Time elapsed: 133.8021340539999\n",
      "Number of layers: 32\n",
      "Max training accuracy: 0.9399999976158142\n",
      "Max test accuracy: 0.9416666666666667\n",
      "Time elapsed: 140.7406529130003\n",
      "Number of layers: 33\n",
      "Max training accuracy: 0.8264583289623261\n",
      "Max test accuracy: 0.8366666666666667\n",
      "Time elapsed: 145.203601274\n",
      "Number of layers: 34\n",
      "Max training accuracy: 0.818958330154419\n",
      "Max test accuracy: 0.8275\n",
      "Time elapsed: 146.2198423689997\n",
      "Number of layers: 35\n",
      "Max training accuracy: 0.8370833337306977\n",
      "Max test accuracy: 0.8450000000000001\n",
      "Time elapsed: 143.88568928800032\n",
      "Number of layers: 36\n",
      "Max training accuracy: 0.8293749988079071\n",
      "Max test accuracy: 0.8358333333333333\n",
      "Time elapsed: 150.14144980400033\n",
      "Number of layers: 37\n",
      "Max training accuracy: 0.6877083361148835\n",
      "Max test accuracy: 0.6991666666666667\n",
      "Time elapsed: 151.7786509309999\n",
      "Number of layers: 38\n",
      "Max training accuracy: 0.8366666674613953\n",
      "Max test accuracy: 0.8441666666666666\n",
      "Time elapsed: 160.2307341959995\n",
      "Number of layers: 39\n",
      "Max training accuracy: 0.8150000035762787\n",
      "Max test accuracy: 0.8233333333333334\n",
      "Time elapsed: 177.75609722299941\n",
      "Number of layers: 40\n",
      "Max training accuracy: 0.5870833337306977\n",
      "Max test accuracy: 0.6041666666666666\n",
      "Time elapsed: 184.6048217460002\n",
      "Number of layers: 41\n",
      "Max training accuracy: 0.5462500035762787\n",
      "Max test accuracy: 0.565\n",
      "Time elapsed: 177.88541118299963\n"
     ]
    }
   ],
   "source": [
    "from Networks.ResNet import ResAntiSymNet\n",
    "import numpy as np\n",
    "import timeit\n",
    "\n",
    "num_epochs = 100\n",
    "num_features = 2\n",
    "num_classes = 2\n",
    "size_hidden = 8\n",
    "gamma = 0.3\n",
    "h = 1\n",
    "lr = 0.1\n",
    "\n",
    "max_layers = 40\n",
    "train_results_per_layer = np.zeros(max_layers)\n",
    "test_results_per_layer = np.zeros(max_layers)\n",
    "basic_save_key = 'AntiSym_Comparison_results//AntiSymNet'\n",
    "\n",
    "for num_hidden_layers in range(1,max_layers+1):\n",
    "    tic = timeit.default_timer()\n",
    "    num_layers = num_hidden_layers + 1\n",
    "    print('Number of layers: '+str(num_layers))\n",
    "    train_results_per_epoch = np.zeros(num_epochs)\n",
    "    test_results_per_epoch = np.zeros(num_epochs)\n",
    "    for i in range(10):\n",
    "        net = ResAntiSymNet(features=num_features, classes=num_classes, num_layers=num_layers, gamma=gamma, h=h, bias=True, hidden_size=size_hidden)\n",
    "        net.set_test_tracking(True)\n",
    "        net.train(num_epochs, train_loader, test_loader, print_output=False)\n",
    "        train_results_per_epoch += net.avg_correct_pred.numpy()\n",
    "        test_results_per_epoch += net.test_results.numpy()\n",
    "        \n",
    "    train_results_per_epoch = train_results_per_epoch / 10\n",
    "    test_results_per_epoch = test_results_per_epoch / 10\n",
    "    train_save_key = basic_save_key+'_train_per_epoch_layers_'+str(num_layers)+'_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "    test_save_key = basic_save_key+'_test_per_epoch_layers_'+str(num_layers)+'_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "    \n",
    "    np.savetxt(train_save_key, train_results_per_epoch, delimiter=',')\n",
    "    np.savetxt(test_save_key, test_results_per_epoch, delimiter=',')\n",
    "    \n",
    "    train_results_per_layer[num_hidden_layers-1] = train_results_per_epoch.max()\n",
    "    print('Max training accuracy: '+str(train_results_per_epoch.max()))\n",
    "    test_results_per_layer[num_hidden_layers-1] = test_results_per_epoch.max()\n",
    "    print('Max test accuracy: '+str(test_results_per_epoch.max()/test_set_size))\n",
    "    \n",
    "    toc = timeit.default_timer()\n",
    "    print('Time elapsed: '+str(toc-tic))\n",
    "\n",
    "train_save_key = basic_save_key+'_train_per_layer_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "test_save_key = basic_save_key+'_test_per_layer_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "np.savetxt(train_save_key, train_results_per_layer, delimiter=',')\n",
    "np.savetxt(test_save_key, test_results_per_layer, delimiter=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1c46e54cb70>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8lPW5///XNZOQAGEPIBAgILvsRkRBRFwq1lo9tdbt2PZrxXOqre3p6Vc9tS79fc851p6jHntaty5qW9uqbd2KilZQ6gZBgbCHTYkghH3PNtfvj/tmHENIhpDJTML7+XjMI3PP3HPPO3dgrvncy3WbuyMiIgIQSXcAERHJHCoKIiISp6IgIiJxKgoiIhKnoiAiInEqCiIiEqeiICIicSoKIiISp6IgIiJxWekOcLTy8/O9sLAw3TFERFqUBQsWbHX37g3N1+KKQmFhIcXFxemOISLSopjZh8nMp81HIiISp6IgIiJxKgoiIhLX4vYpiIg0RlVVFWVlZRw8eDDdUVIqNzeXgoICsrOzG/V6FQUROS6UlZXRoUMHCgsLMbN0x0kJd2fbtm2UlZUxYMCARi1Dm49E5Lhw8OBBunXr1moLAoCZ0a1bt2MaDakoiMhxozUXhEOO9XdUURARkTgVBRGRZrBz505+/vOfH/XrLrjgAnbu3JmCRHVTURARaQZHKgo1NTX1vm7mzJl07tw5VbEOo6OPRESawS233MKaNWsYO3Ys2dnZ5OXl0atXLxYuXMiyZcu4+OKL2bBhAwcPHuSmm25ixowZwKetffbu3cv06dOZPHkyb7/9Nn369OG5556jbdu2TZpTRUFEjjt3vbCUZRt3N+kyR/TuyB1fOOmIz999990sWbKEhQsXMmfOHD7/+c+zZMmS+KGjv/rVr+jatSsHDhzglFNO4Utf+hLdunX7zDJKS0v5/e9/z6OPPspll13Gn/70J66++uom/T1UFERE0mDChAmfOZfggQce4C9/+QsAGzZsoLS09LCiMGDAAMaOHQvAySefzPr165s8l4qCiBx36vtG31zat28fvz9nzhxee+013nnnHdq1a8fUqVPrPNcgJycnfj8ajXLgwIEmz5WyHc1m9isz22JmS47wvJnZA2a22swWm9n4VGUREUm3Dh06sGfPnjqf27VrF126dKFdu3asWLGCd999t5nTfSqVI4XHgP8FnjjC89OBweHtVODB8KeISKvTrVs3Jk2axMiRI2nbti09e/aMP3f++efz0EMPMXr0aIYOHcrEiRPTltPcPXULNysEXnT3kXU89zAwx91/H06vBKa6+6b6lllUVOS6yI6IHK3ly5czfPjwdMdoFnX9rma2wN2LGnptOvcp9AE2JEyXhY/VWxTS4UBlDW+sKmfLnoPgTsSricSq4j/Nq4nUVGGVe4hU7iZSsZusyt1kVe4iu2oPbar3kBWrJNZ7PCeMm07hwCFJnYp+oLKGktI1bCn5G77jI2osq+4bWRhO1GuIehVRqol4DVkeZIx6NfW/mxMN5wtuVZ+ZNpyqSA6Vlktl5NCtbfx+DVlEqf70/Wsvq+MJ5A89nREjx5KX26bB39tjNWxY9QGfLHkD37gIj9UQi2QRs2xqLEoskk3MsqixbNwi4d+hhqhXEvEaoof+Nl5NhFjSf+dUSPwbffbvlo1DsM74dJ0n/t1q2nUnZ8Bp9Bk9lZ49Tmjw30xldYyVm3azck0pe8qWkdejkKHDRzGid2eyog1vKT5QWcOiD8tZt+IDDpSvCzMl/k0/vU+HnnQZcRajho+gU9vGdeOUzJTOolDXv/A6hy1mNgOYAdCvX79UZoo7WFXDnJXlvLpoLZ1XPs0VvMzZtplsq/9Ek7rspR0xjI7bn4Mld7CGAtZ1OpXYgLPoM+5chvXtSTRi7DlYxcJV6yhf8jo5G97ixP0fMME2NPwGTaiaKFVkUU2UarKoJgvHyKWCHCrIoeroF7oZKIVdL7Tj/TZD2dttDO0HTKD/mDPIP6EfVQf3sX7x39m58k1yNs2n//6l9GMv/YCd5FFFNsHHajXZVIf3D/87HMqdmD9G9JjXSeP5YbkPpaqtMlzX1WRRRRY1ROiyYxfZGx+Ht2ANfSnrMJqq3hPoNHQKQ4aOZPu+g6xZsZida4uJbCmh+95VDLf1jLLwUMvVsOettiykP1vzhmEnjCJ/yCkMGVlEh/btKd+2jdUl77Jz7QKytpRwwv5SxlkZEy2Jv/FmYPUdrHv2BN7KHcO+XqfRccQ0xgwbygmdcpt2NUqz0uajBBXVNcxdtZUXF2+kZPlyLq2ZyZVZs+nEXvbkjyV64pkQyYZoNh7/mQXRNhDJxnI7kt2+M9F2nYm27QxtO0NOR4hE8ViMTaXvs2XhS7T96A0K9y0ih0oqPIsPGE55Tl8GHlzKcPuIiDkV5LCx4xhihWfQc/Q55BWMhFh1cKuphJqq4BarCqaxIEc0Gw5limZ/Om0NfFMMfx8aGsHUVEPV/k9vlfuDDOE6IJqVcD947/1bP6RsyVwOrJtHp22LKahaR5YFH4zl1pXOsV3xYrveCvik01is/0R6jZxK3xNPwiJ1ZI/FPl0f8d+xhTQ7i8WCdeaxYD1FonVmP7h/DxtK5rKn9C3abppP331LyGMfAOXekXZU0N4qAKgmi63tBlLVfSR5hePp3Pckdn2ylh1rioluWUL3faW0JTiapdKjbLMu9PRtRCz4/7/HOrCtwzCs92jyTyyifa+hkJWT8G8o+zP/vg6Wr2XTwln4ujfpueN92nuQa3WsNyVtRtNp6reZNnlSc6zNpGnzUXKbj9JZFD4P3AhcQLCD+QF3n9DQMlNRFNydh95Yy8/nrKawYhX/lPMy5/MOEXN82BeInH4j9G0w2tGpOsC2ZXPYUfIy7cvm0rWijM0dR+H9J9NzzLnk9p8AWQ1vammJKg/sZd2Sd9i28m0im0uoatuTnBNPp3DsWfTo2Tvd8TJXLMaeshI2l7xB9UfzsLYd6Vg4nu6DTyGr5/D6/73Eatj3ySo2LH2PPesXEN1dRix/CF1OLKLv8FNp07Vv44tqrIbqjxdSXvIq1Wvmkr+tmB3enp1XvcKIIYMbt8wUUFFIc1Ews98DU4F8gsHmHUA2gLs/ZMEG0v8Fzgf2A1939wY/7Zu6KLg7d7+8ggVvzuQ/Ov6FIRUleJsO2Phr4NQZ0KWwyd5L5Hiwa+18cp64gJU2kL7f/RtdO+alOxKgopD2Hc3ufkUDzztwQ6rePxnuzl0vLGPPu0/wx5xHieT0gqn/gY37R8jtmM5oIi1Wp4Gn8NG0exnz+o28/vA3mPLd35GVlc59O3I0jtsuqbGY829/LqHLvP/iv9s8RGTAZOyf34LTblBBEDlG/ab8I8sGXce0fS/x+m/+I91xMkJjW2cD3H///ezfv7+JE9XtuCwK1TUxbn5qPkULb+WmrD/jY6/Ernom2DEsIk1ixJX3sLzjJKatv5d5r/8l3XHSrqUUheOu91FVTYxbn5zLpaU3MzG6HM66DZvyry3nyBWRliISYeD1v2PjvVMY/Ma3WN9vKIWDRqQ7Vdokts4+99xz6dGjB0899RQVFRVccskl3HXXXezbt4/LLruMsrIyampq+OEPf8jmzZvZuHEjZ511Fvn5+cyePTulOY+rolBRXcOdj/+Vf/7w/1IY3QqXPAqjL0t3LJFWK6d9F3Kv+SORX59L7MnL2fvdN8nrkAEj8pdugU9KmnaZJ4yC6Xcf8enE1tmzZs3imWeeYd68ebg7F110EW+++Sbl5eX07t2bv/71r0DQE6lTp07ce++9zJ49m/z8/KbNXIfjZvPRwaoa7n70t3zvoxsoaLOP6FefVUEQaQY9+o/g47N/Rv+ajyh96Co8dvQngLY2s2bNYtasWYwbN47x48ezYsUKSktLGTVqFK+99ho333wzc+fOpVOnTs2e7bgZKbz89KPc/MltVLfrQc61z0F+5hw/LdLajTjjYt756HucVvpfFD9xM0Vf+69Pn4zVhCdjVn56MmJOh9QGqucbfXNwd2699Vauv/76w55bsGABM2fO5NZbb+W8887j9ttvb9Zsx01RmD6ukAM7xtD5q3+AvO7pjiNy3Jl4xQ945/4lnLb+Ufbf+duwGUkN0VptP6rJ4uB1c8nr07r2PyS2zv7c5z7HD3/4Q6666iry8vL4+OOPyc7Oprq6mq5du3L11VeTl5fHY4899pnXNsfmo+OmKOQMP5+cYZ/TDmWRNLFIhNHX/4q//fZOOLA97FF1qN9T2LMq5lyx65ds+NsjDL/m/nRHblKJrbOnT5/OlVdeyWmnnQZAXl4ev/3tb1m9ejXf//73iUQiZGdn8+CDDwIwY8YMpk+fTq9evVK+ozmlbS5SQa2zRVovd+ft/3ceI3w1XX5QGvTSaiI6ozm5M5qPmx3NIpL5zIxtgy+lS2w7O0teTnec45KKgohklBFnfpntnseOtx9Ld5TjkoqCiGSUQb26Mjf3LAq2zIb925t02S1tc3ljHOvvqKIgIhmnctQVZFPN9vf+0GTLzM3NZdu2ba26MLg727ZtIze38Rc6Om6OPhKRluP0ydNYNq8/3d7/DZz1zSZZZkFBAWVlZZSXlzfJ8jJVbm4uBQUFjX69ioKIZJw+ndsyq+Pn+PqeR2DzMuh57OcsZGdnM2DAgCZI17pp85GIZKT2RZdT5VG2a4dzs1JREJGMNK1oJHN8HDnLng7aYEizUFEQkYyUn5fDkh4X0r5qO776tXTHOW6oKIhIxuo74Yts9Y7sfOfxdEc5bqgoiEjGOm90X17wSXT88NUmP2dB6qaiICIZq2NuNmX9LiHq1cQWP5XuOMcFFQURyWjjJ0xhSayQ/fOeSHeU44KKgohktLOH9+B5ppK3fSl8siTdcVo9FQURyWi52VH2D72EKqLUfPBkuuO0eioKIpLxzj55BH+rGU/1wj/onIUUU1EQkYw3eXA+L2VNI6diG5S+evQLOLAD/n4f3D8KHhgPz34T3n8CyldBUzXIW/M6PPkVWPdm0ywvTdT7SEQyXnY0QodR09m66CG6zHuUaL+J0K5rwy/cuhreexAWPglV+9nWYyI1We3JX/ESkYW/C+Zp1w36ngr9JkL/yVBwcqMyVr/7MFmlL8Oql2HgWXD27dBnfKOWlU4qCiLSIlw4th9Pvn8W3177LNwzALoPCz7I+04MfnYpDK7B7g7r58I7P4dVL+PRbEp7Tufft03ljY96hktzTm63lQs6r2dCdBUDP15C+5Uzg6e+9EsYdenRhaupombNmzxbM4V9nYdx1cZnyHr0LBh+EUy7DboPbcpVkVK6RrOItAixmDPpP1/jws7r+M6Q7bTfXAwb5kHFrmCGvJ7BN/4d6+CTEmJtu/Fet0u47eMJrDmQR1H/Llw3ZSD5eTks3biLJR/vYsnHu1m1eQ/VMSefXbyQexs5/Yro+n+O8pyID9+BX5/PbW1u5uXYKRzcu5N7es/l/D3PEKk+AGOuhKm3QOe+Tb9ikpTsNZo1UhCRFiESMb58Sj8eeL2SRzf0oXuHIkb2+lemdN5KUWQVA/aX0H5TMVXRPP5acDN3rD+JPTuzOHd4T+45cyAn9/90c9PJ/bvE7x+sqmHV5j0s+Xg3b8wczyVlb0J1JWS1STrbvuWzyHVjwCkXMGfKSB55Yw3fm9uBu2Kn8z99ZnNqydNYyVNw3r/DqTOadL00NY0URKTFiMWceeu3s3TjbpZu3MWyjbtZvWUv1bHgc6x9myj7q2rIjkT4h/F9+MYZAxnUIy/p5f/0wQf41uYf4tc8hw2cmvTrtv/PGazftp/sGa8zqqATAJt3H+S+V1fxVPEGBubs5LcdH6Knb8W+t/wofuOmkxEjBTM7H/gfIAr8wt3vrvV8P+BxoHM4zy3uPjOVmUSk5YpEjIkDuzFxYLf4YweraijdvJdlm3axdONuOrdrw9Wn9qNHx6O/JGX+qHOp+OQuDiyeSeeBU5N70YEddN6xhPmRf+AbvTvGH+7ZMZe7vzSa/zN5AD9+aQW/LB3JD7KfDHo4JbOTPE1SVhTMLAr8DDgXKAPmm9nz7r4sYbbbgKfc/UEzGwHMBApTlUlEWp/c7CijCjrFv6Efi8kj+vPeK8MZfRSHvfraN4gQY0/BFKIRO+z5IT078MuvncKvHl8E657ENy/BBkw55qypksrzFCYAq919rbtXAn8AvlhrHgcOldZOwMYU5hERqVffru1Y3PZUOu9bCzvWJ/WaPUtnscfb0vukSfXO12nAOAB2rV94rDFTKpVFoQ+wIWG6LHws0Z3A1WZWRjBK+FYK84iINMgHnQNA1YpZSczsRNbN5p3YCE4f3KveWfv3LWSrd2TfR4uaImbKpLIoHD6OCkYGia4AHnP3AuAC4DdmdlgmM5thZsVmVlxeXp6CqCIigZNGjWN9rCd7SpLYvbl9LXkHNrI452T6d2tX76xDenVkRawvWVvTs6M5WaksCmVA4kG5BRy+eeha4CkAd38HyAXyay/I3R9x9yJ3L+revXuK4oqIwMQT83nTx9Hhk7eh6kC988ZWvx78HDAVs7q+B3+qY242ZW0G0mXvaojVNFneppbKojAfGGxmA8ysDXA58HyteT4CzgYws+EERUFDARFJm3ZtstjUczLZsQpY/1a98+5ZOosNse4MO2lsUsve23kobbwCtq9riqgpkbKi4O7VwI3AK8BygqOMlprZj8zsonC27wHXmdki4PfA17ylnTghIq1OtxHTOOBt2Le0nk1INVXkfvwWc2OjOH3QYRs46hTpNRKA6k0lTREzJVJ6nkJ4zsHMWo/dnnB/GVD/LnsRkWY2aXhf3p59EqeueiXopVTXpqGPF5BTs4+1nU4lPy8nqeV27T+a6pIIuz/8gK6jLmni1E2jwZGCmY1sjiAiIpli2AkdKM4uIm9/GWxbU+c81ateo8aN3MFTk17uoN75rPVeVH6cuSOFZDYfPWRm88zsm2bWOeWJRETSzMyoOfFsAGKrXqlznv0rXmOxn8jJwwcmvdxBPfJY6f1ouz1zj0BqsCi4+2TgKoIjiYrN7EkzOzflyURE0mjkyDGUxvqwd+lLhz95YCd5Wxfxlo9mQmHyLStys6N80nYQnSo2wcFdTZi26SS1o9ndSwlaUtwMnAk8YGYrzOwfUhlORCRdzhiUz5zYWNptfBcq9332yXVvEiHGlh6n0z7n6HbNVnYbHtzZkpmjhWT2KYw2s/sIjiCaBnzB3YeH9+9LcT4RkbTo0r4NH3abRJZXHXaJzYqVr7HXc+kxbPJRLzenYBRAxu5XSGak8L/A+8AYd7/B3d8HcPeNBKMHEZFWKX/4mez1XCqWv/yZx6tXv847sZM4bcgJR73MPn0HscvbsefDzOyBlExRuAB40t0PAJhZxMzaAbj7b1IZTkQknc4Y3pu/x0ZRs3JWcGgqwPa1tN+3gfciYxjTiM6sQ3p1ZIX3wzcvaeK0TSOZovAa0DZhul34mIhIqzamoDPvRMfT7sBGKF8RPLgmaG1xoO8UsqJHf/5vYbf2rKI/HXatglisKeM2iWR+o1x333toIrxff+cnEZFWICsaoXLANAB8VdA1df+K1yjzfAYNG9OoZUYjxo68weTE9sOuj5osa1NJpijsM7PxhybM7GSg/i5RIiKtxJjhI1gW68+BZS9BTTVZH87lzZpRTB7c+OacsR7hOcGfZN4mpGSKwneAp81srpnNBf5I0NNIRKTVmzKkO3NiY8jdNB/WzqZN9V5KcsYf1bWfa+vQbxQxNw6WLW7CpE2jwQNs3X2+mQ0DhhJcI2GFu1elPJmISAbo3bktpR1PI7L/efy1O3CM6IkNt8quz8A+PVjvPelctoijv5J0aiW7l2QoMAIYB1xhZtekLpKISGbpNnwyu70dtnkpi2MDGTc0+dYWdRnaswMrvB9Z5Zl3AlsyJ6/dAfw0vJ0F3ANcVO+LRERakTOG9uLN2GgA5sZGMSnJVtlH0qtTLuuiheTt/+jws6XTLJmRwqUEF8L5xN2/DowBkusTKyLSCpw6oCtvcDIApR0mckKnY9voY2bs7TSUCA5bVjRFxCaTTFE44O4xoNrMOgJbgGMbO4mItCC52VG2FF7ElyruoMuwM5pkmYcuuJNpJ7ElUxSKw5bZjwILCFpezEtpKhGRDHPGkB4s8KHHvOnokJ79hrDXc9m/YVGTLK+p1Hv0kQW71//T3XcSXFfhZaCju2fecVQiIin05ZP7sreimjOHNv78hERDTujESu/LiRszqzFevSOF8HrJzyZMr1dBEJHjUad22XznnCHkZEWbZHlDe3ZgeSy84E4GXZo+mc1H75rZKSlPIiJyHOnSvg0ftxlITvUe2P1xuuPEJVMUzgLeMbM1ZrbYzErMTKMFEZFjVJE/IrizeWl6gyRI5pJB01OeQkTkOJTbeyRsgdgnS4gM+Vy64wDJjRT8CDcRETkGhQW92BDrnlFHICUzUvgrQREwIBcYAKwETkphLhGRVu9Qu4uJGbT5qMGRgruPcvfR4c/BwATg76mPJiLSug3umccK70v73eug6mC64wDJN8SLC6/RrKORRESOUbs2WWxtP5gINbB1ZbrjAElsPjKzf0mYjADjgfKUJRIROY7UdD8JygiOQOrVuKu5NaVkRgodEm45BPsYvpjKUCIix4suBUM54G2o3pQZR/onc5Gdu5ojiIjI8WhIr86s9AIGlZXQ+Gu5NZ1krqfwatgQ79B0FzN7JbWxRESOD0NP6MCKWD+yy5dmRLuLZDYfdQ8b4gHg7juAHqmLJCJy/BiQ355S609O5Q7YuyXdcZIqCjVm1u/QhJn1J8mT18zsfDNbaWarzeyWI8xzmZktM7OlZvZkcrFFRFqH7GiEPR2HBhMZcG2FZE5e+wHwdzN7I5yeAsxo6EVmFgV+BpxLsG99vpk97+7LEuYZDNwKTHL3HWamEYiIHHcivU6C1QRHIA06O71ZGprB3V8mOAz1j8BTwMnunsw+hQnAandf6+6VwB84/Kil64CfhZukcPf0j51ERJpZ3z4FrIn1IvbO/8LW0rRmSWZH8yVAlbu/6O4vEFyW8+Iklt0H2JAwXRY+lmgIMMTM3jKzd83s/CNkmGFmxWZWXF6uUyREpHUZ2rMD/1T1XWqqa+CxC6F8VdqyJLNP4Q5333VoItzpfEcSr7M6Hqu9LyILGAxMBa4AfpF4pFPCez7i7kXuXtS9e9Nc9UhEJFMMPaEDpV7ArFN+AR6Dxy+E8vSc4ZxMUahrnmT2RZQBfROmC4CNdczznLtXufs6gkZ7g5NYtohIq9Gnc1vat4nyzEftqbnmheDQ1McuhC0rmj1LMkWh2MzuNbMTzWygmd0HLEjidfOBwWY2wMzaAJcDz9ea51mCi/hgZvkEm5PWJh9fRKTli0SM75wzhNkry7l1bgWxr74IZsGIYfOyhhfQlFmSmOdbQCXBjuangYPANxt6kbtXAzcCrwDLgafcfamZ/cjMLgpnewXYZmbLgNnA991929H/GiIiLdt1UwZy09mDeaq4jNvfrsS/+iJYFB7/QrNemc38KM+gM7Nc4Avu/nRqItWvqKjIi4uL0/HWIiIp5e7c/fIKHn5jLddOHsBtE7Oxx78ANZVwzfNwwshGL9vMFrh7UUPzJdU628yiZjbdzJ4A1gNfaXQyERGpk5lxy/nD+Nrphfzy7+v4SXFNMGKI5gQjhk9KUp6h3h3GZjYFuBL4PDAPmAQMdPf9KU8mInIcMjPu+MIIKqpj/HzOGnKzh/Dtr70Iv7kYtq2GE0al9P2PWBTMrAz4CHiQYFv/HjNbp4IgIpJaZsa/XzySyuoY9766ipysYVx/w3zIzk35e9c3UvgTcDHBpqIaM3uOJHseiYjIsYlEjHsuHU1FdQ3/+dIKcrIifG3SgNS/75GecPebgELgXoLDRlcB3cMGdpnQ9ltEpFWLRoz7vjKW80b05M4XlvHMgrKUv2e9O5o98Lq7X0dQIK4kGD2sT3kyEREhOxrhp1eO4ytFfRnf77CGD00umTOTAXD3KuAF4AUza5u6SCIikignK8qPLx3dLO+V1CGptbn7gaYOIiIi6deooiAiIq1T0kXBzNqnMoiIiKRfMtdTOD3sTbQ8nB5jZj9PeTIREWl2yYwU7gM+B2wDcPdFBJfkFBGRViapzUfuvqHWQzUpyCIiImmWzCGpG8zsdMDD6yJ8m3BTkoiItC7JjBT+CbiB4PrKZcDYcFpERFqZBkcK7r4VuKoZsoiISJo1WBTM7IE6Ht4FFLv7c00fSURE0iWZzUe5BJuMSsPbaKArcK2Z3Z/CbCIi0syS2dE8CJgWXnMZM3sQmAWcC6T+MkAiItJskhkp9AESz2ZuD/R29xqgIiWpREQkLZIZKdwDLDSzOYARnLj2H2Hbi9dSmE1ERJpZMkcf/dLMZgITCIrCv7n7xvDp76cynIiINK9kG+IdBDYB24FBZqY2FyIirVAyh6R+A7gJKAAWAhOBd4BpqY0mIiLNLZmRwk3AKcCH7n4WMA4oT2kqERFJi2SKwkF3PwhgZjnuvgIYmtpYIiKSDskcfVRmZp2BZ4FXzWwHsLGB14iISAuUzNFHl4R37zSz2UAn4OWUphIRkbSotyiYWQRY7O4jAdz9jWZJJSIiaVHvPgV3jwGLzKxfM+UREZE0SmZHcy9gqZn9zcyeP3RLZuFmdr6ZrTSz1WZ2Sz3zXWpmbmZFyQYXEZGml8yO5rsas2AziwI/I2icVwbMN7Pn3X1Zrfk6EFzN7b3GvI+IiDSdBkcK4X6E9UB2eH8+8H4Sy54ArHb3te5eCfwB+GId8/1/BP2VDiYbWkREUqPBomBm1wHPAA+HD/UhODy1IX2ADQnTZeFjicseB/R19xeTSisiIimVzD6FG4BJwG4Ady8FeiTxOqvjMY8/GRzZdB/wvQYXZDbDzIrNrLi8XCdTi4ikSjJFoSLc/AOAmWWR8OFejzKgb8J0AZ896a0DMBKYY2brCXoqPV/XzmZ3f8Tdi9y9qHv37km8tYiINEYyReENM/s3oK2ZnQs8DbyQxOvmA4PNbICZtQEuB+JHLbn7LnfPd/dCdy8E3gUucvfio/4tRESkSSRTFG4haIBXAlwPzAQRCxWHAAAMKklEQVRua+hF4eU7bwReAZYDT7n7UjP7kZld1PjIIiKSKuZe/5YgM7sEmOnuGXHpzaKiIi8u1mBCRORomNkCd2/wXLBkRgoXAavM7Ddm9vlwn4KIiLRCyZyn8HVgEMG+hCuBNWb2i1QHExGR5pfUt353rzKzlwiOOmpLcBLaN1IZTEREml8yJ6+db2aPAauBS4FfEPRDEhGRViaZkcLXCFpUXJ8pO5tFRCQ1krnIzuWJ02Y2CbjS3W9IWSoREUmLpPYpmNlYgp3MlwHrgD+nMpSIiKTHEYuCmQ0hOAv5CmAb8EeC8xrOaqZsIiLSzOobKawA5gJfcPfVAGb23WZJJSIiaVHf0UdfAj4BZpvZo2Z2NnV3PhURkVbiiEXB3f/i7l8BhgFzgO8CPc3sQTM7r5nyiYhIM0rmjOZ97v47d7+QoP31QoImeSIi0sok0/sozt23u/vD7j4tVYFERCR9jqooiIhI66aiICIicSoKIiISp6IgIiJxKgoiIhKnoiAiInEqCiIiEqeiICIicSoKIiISp6IgIiJxKgoiIhKnoiAiInEqCiIiEqeiICIicSoKIiISp6IgIiJxKgoiIhKnoiAiInEpLQpmdr6ZrTSz1WZ22HWdzexfzGyZmS02s7+ZWf9U5hERkfqlrCiYWRT4GTAdGAFcYWYjas32AVDk7qOBZ4B7UpVHREQalsqRwgRgtbuvdfdK4A/AFxNncPfZ7r4/nHwXKEhhHhERaUAqi0IfYEPCdFn42JFcC7xU1xNmNsPMis2suLy8vAkjiohIolQWBavjMa9zRrOrgSLgJ3U97+6PuHuRuxd17969CSOKiEiirBQuuwzomzBdAGysPZOZnQP8ADjT3StSmEdERBqQypHCfGCwmQ0wszbA5cDziTOY2TjgYeAid9+SwiwiIpKElBUFd68GbgReAZYDT7n7UjP7kZldFM72EyAPeNrMFprZ80dYnIiININUbj7C3WcCM2s9dnvC/XNS+f4iInJ0dEaziIjEqSiIiEicioKIiMSpKIiISJyKgoiIxKkoiIhInIqCiIjEqSiIiEicioKIiMSpKIiISJyKgoiIxKkoiIhInIqCiIjEqSiIiEicioKIiMSpKIiISJyKgoiIxKkoiIhInIqCiIjEqSiIiEicioKIiMSpKIiISJyKgoiIxKkoiIhInIqCiIjEqSiIiEicioKIiMSpKIiISJyKgoiIxKkoiIhIXEqLgpmdb2YrzWy1md1Sx/M5ZvbH8Pn3zKwwlXlERKR+KSsKZhYFfgZMB0YAV5jZiFqzXQvscPdBwH3Aj1OVR0REGpbKkcIEYLW7r3X3SuAPwBdrzfNF4PHw/jPA2WZmKcwkIiL1SGVR6ANsSJguCx+rcx53rwZ2Ad1SmElEROqRlcJl1/WN3xsxD2Y2A5gRTlaY2ZJjzJYq+cDWdIc4AmVrHGVrHGVrnFRm65/MTKksCmVA34TpAmDjEeYpM7MsoBOwvfaC3P0R4BEAMyt296KUJD5GytY4ytY4ytY4yla/VG4+mg8MNrMBZtYGuBx4vtY8zwNfDe9fCrzu7oeNFEREpHmkbKTg7tVmdiPwChAFfuXuS83sR0Cxuz8P/BL4jZmtJhghXJ6qPCIi0rBUbj7C3WcCM2s9dnvC/YPAl49ysY80QbRUUbbGUbbGUbbGUbZ6mLbWiIjIIWpzISIicS2mKDTUMiOdzGy9mZWY2UIzK86APL8ysy2Jh+6aWVcze9XMSsOfXTIk151m9nG47haa2QXNnSvM0dfMZpvZcjNbamY3hY9nwno7Ura0rzszyzWzeWa2KMx2V/j4gLB1TWnYyqZNBmV7zMzWJay3sc2dLSFj1Mw+MLMXw+m0rzfcPeNvBDuq1wADgTbAImBEunMl5FsP5Kc7R0KeKcB4YEnCY/cAt4T3bwF+nCG57gT+NQPWWS9gfHi/A7CKoD1LJqy3I2VL+7ojONcoL7yfDbwHTASeAi4PH38I+OcMyvYYcGm6/82Fuf4FeBJ4MZxO+3prKSOFZFpmSMjd3+Tw8z0SW4o8DlzcrKE4Yq6M4O6b3P398P4eYDnBGfeZsN6OlC3tPLA3nMwObw5MI2hdA+lbb0fKlhHMrAD4PPCLcNrIgPXWUopCMi0z0smBWWa2IDz7OhP1dPdNEHzIAD3SnCfRjWa2ONy81OybZ2oLu/WOI/hmmVHrrVY2yIB1F24CWQhsAV4lGNXv9KB1DaTx/2vtbO5+aL39e7je7jOznHRkA+4H/i8QC6e7kQHrraUUhaTaYaTRJHcfT9AR9gYzm5LuQC3Ig8CJwFhgE/Df6QxjZnnAn4DvuPvudGaprY5sGbHu3L3G3ccSdC2YAAyva7bmTRW+aa1sZjYSuBUYBpwCdAVubu5cZnYhsMXdFyQ+XMeszb7eWkpRSKZlRtq4+8bw5xbgLwT/MTLNZjPrBRD+3JLmPAC4++bwP24MeJQ0rjszyyb40P2du/85fDgj1ltd2TJp3YV5dgJzCLbbdw5b10AG/H9NyHZ+uDnO3b0C+DXpWW+TgIvMbD3B5vBpBCOHtK+3llIUkmmZkRZm1t7MOhy6D5wHZGLDvsSWIl8FnktjlrhDH7ihS0jTugu35/4SWO7u9yY8lfb1dqRsmbDuzKy7mXUO77cFziHY5zGboHUNpG+91ZVtRUKRN4Jt9s2+3tz9VncvcPdCgs+z1939KjJgvaV973uyN+ACgqMu1gA/SHeehFwDCY6GWgQszYRswO8JNidUEYyyriXYXvk3oDT82TVDcv0GKAEWE3wA90rTOptMMFRfDCwMbxdkyHo7Ura0rztgNPBBmGEJcHv4+EBgHrAaeBrIyaBsr4frbQnwW8IjlNJ1A6by6dFHaV9vOqNZRETiWsrmIxERaQYqCiIiEqeiICIicSoKIiISp6IgIiJxKgqS0czMzey/E6b/1czubKJlP2ZmlzY85zG/z5fDDqezaz1emNgxViQTqChIpqsA/sHM8tMdJJGZRY9i9muBb7r7WanK05CEs2RF6qWiIJmumuAShd+t/UTtb/pmtjf8OdXM3jCzp8xslZndbWZXhb31S8zsxITFnGNmc8P5LgxfHzWzn5jZ/LBp2vUJy51tZk8SnPxUO88V4fKXmNmPw8duJzj57CEz+0kyv7CZXRe+9yIz+5OZtTOzDuE1ALLDeTpacB2PbDM70cxeDhsyzjWzYQnr595whPJjMzvTPr2GwAeHzsQXSaRvD9IS/AxYbGb3HMVrxhA0ZtsOrAV+4e4TLLhAzbeA74TzFQJnEjSWm21mg4BrgF3ufkrYQfMtM5sVzj8BGOnu6xLfzMx6Az8GTgZ2EHTNvdjdf2Rm0wiue5DsBZj+7O6Phsv9f8C17v5TM5tD0Gr5WYLWCH9y9yozewT4J3cvNbNTgZ8T9NIBGAKc4+41ZvYCcIO7vxU21zuYZB45jmikIBnPg46gTwDfPoqXzfeg8VkFQWuUQx/qJQSF4JCn3D3m7qUExWMYQf+qa8KWy+8RtLoYHM4/r3ZBCJ0CzHH3cg9aH/+O4KJCjTEy/MZfAlwFnBQ+/gvg6+H9rwO/Dj/cTweeDvM+THBRnkOedvea8P5bwL1m9m2gs3/aolkkTiMFaSnuB94n6Gp5SDXhF5uwuVnipQsrEu7HEqZjfPbffe0+L07Qwvhb7v5K4hNmNhXYd4R8dbU9bqzHgIvdfZGZfY2gNw7hN/xCMzsTiLr7EjPrSNCD/0iXlIzndfe7zeyvBH2T3jWzc9x9RRPmllZAIwVpEdx9O8GlCq9NeHg9weYaCK6Qlt2IRX/ZzCLhfoaBwErgFeCfE7bfDwk74NbnPeBMM8sPd0JfAbzRiDwQXHJzU/j+V9V67gmCxoK/hvgoap2ZfTnMamY2pq6FmtmJ7l7i7j8GiglGRSKfoaIgLcl/A4lHIT1K8EE8DziVI3+Lr89Kgg/vlwi2yx8k2EyzDHg/PGT0YRoYVXtwVbZbCVofLwLed/dk2h4PNbOyhNuXgR8SFJlXgdrf5H8HdCEoDIdcBVxrZoc69R7pUrXfCXeCLwIOhL+zyGeoS6pICxIebfVFd//HdGeR1kn7FERaCDP7KcElXy9IdxZpvTRSEBGROO1TEBGROBUFERGJU1EQEZE4FQUREYlTURARkTgVBRERifv/ASSGmsp5u+nsAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "layers = np.arange(2,max_layers+2,1)\n",
    "ax.plot(layers, train_results_per_layer, label='train')\n",
    "ax.plot(layers, test_results_per_layer/test_set_size, label='test')\n",
    "ax.set_ylim(0,1.1)\n",
    "ax.set_xlabel('Number of Layers')\n",
    "ax.set_ylabel('Average Accuracy')\n",
    "ax.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### h =  0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of layers: 2\n",
      "Max training accuracy: 0.940416669845581\n",
      "Max test accuracy: 0.955\n",
      "Time elapsed: 11.628685269999991\n",
      "Number of layers: 3\n",
      "Max training accuracy: 0.9229166686534882\n",
      "Max test accuracy: 0.9450000000000001\n",
      "Time elapsed: 14.960609403000007\n",
      "Number of layers: 4\n",
      "Max training accuracy: 0.9445833265781403\n",
      "Max test accuracy: 0.95\n",
      "Time elapsed: 18.329642236999987\n",
      "Number of layers: 5\n",
      "Max training accuracy: 0.9210416734218597\n",
      "Max test accuracy: 0.9283333333333333\n",
      "Time elapsed: 22.024457772999995\n",
      "Number of layers: 6\n",
      "Max training accuracy: 0.926458328962326\n",
      "Max test accuracy: 0.94\n",
      "Time elapsed: 24.795526373\n",
      "Number of layers: 7\n",
      "Max training accuracy: 0.9287499904632568\n",
      "Max test accuracy: 0.9366666666666668\n",
      "Time elapsed: 29.42907770100001\n",
      "Number of layers: 8\n",
      "Max training accuracy: 0.9362499952316284\n",
      "Max test accuracy: 0.94\n",
      "Time elapsed: 33.696739648999994\n",
      "Number of layers: 9\n",
      "Max training accuracy: 0.9208333313465118\n",
      "Max test accuracy: 0.9283333333333333\n",
      "Time elapsed: 38.653537838999995\n",
      "Number of layers: 10\n",
      "Max training accuracy: 0.9291666746139526\n",
      "Max test accuracy: 0.9358333333333333\n",
      "Time elapsed: 38.49169696199999\n",
      "Number of layers: 11\n",
      "Max training accuracy: 0.9343749940395355\n",
      "Max test accuracy: 0.9408333333333334\n",
      "Time elapsed: 42.408528811\n",
      "Number of layers: 12\n",
      "Max training accuracy: 0.9375000059604645\n",
      "Max test accuracy: 0.9408333333333334\n",
      "Time elapsed: 48.674559146000036\n",
      "Number of layers: 13\n",
      "Max training accuracy: 0.9429166615009308\n",
      "Max test accuracy: 0.9566666666666667\n",
      "Time elapsed: 53.017350943999986\n",
      "Number of layers: 14\n",
      "Max training accuracy: 0.9431250035762787\n",
      "Max test accuracy: 0.9575\n",
      "Time elapsed: 53.721814804000076\n",
      "Number of layers: 15\n",
      "Max training accuracy: 0.9254166662693024\n",
      "Max test accuracy: 0.9333333333333333\n",
      "Time elapsed: 60.64417378799999\n",
      "Number of layers: 16\n",
      "Max training accuracy: 0.959375\n",
      "Max test accuracy: 0.9716666666666666\n",
      "Time elapsed: 58.44810755200001\n",
      "Number of layers: 17\n",
      "Max training accuracy: 0.9587500035762787\n",
      "Max test accuracy: 0.9633333333333333\n",
      "Time elapsed: 66.8953957\n",
      "Number of layers: 18\n",
      "Max training accuracy: 0.943958330154419\n",
      "Max test accuracy: 0.9466666666666667\n",
      "Time elapsed: 67.04302446100007\n",
      "Number of layers: 19\n",
      "Max training accuracy: 0.9299999952316285\n",
      "Max test accuracy: 0.9358333333333333\n",
      "Time elapsed: 72.94822167300003\n",
      "Number of layers: 20\n",
      "Max training accuracy: 0.9354166805744171\n",
      "Max test accuracy: 0.9475\n",
      "Time elapsed: 76.85616754599994\n",
      "Number of layers: 21\n",
      "Max training accuracy: 0.9285416662693023\n",
      "Max test accuracy: 0.9358333333333333\n",
      "Time elapsed: 80.828388453\n",
      "Number of layers: 22\n",
      "Max training accuracy: 0.932708328962326\n",
      "Max test accuracy: 0.9441666666666666\n",
      "Time elapsed: 81.23192526499986\n",
      "Number of layers: 23\n",
      "Max training accuracy: 0.9352083325386047\n",
      "Max test accuracy: 0.9458333333333333\n",
      "Time elapsed: 86.15025867300005\n",
      "Number of layers: 24\n",
      "Max training accuracy: 0.9418749928474426\n",
      "Max test accuracy: 0.9391666666666667\n",
      "Time elapsed: 88.071327376\n",
      "Number of layers: 25\n",
      "Max training accuracy: 0.9377083420753479\n",
      "Max test accuracy: 0.9433333333333334\n",
      "Time elapsed: 93.33065287199997\n",
      "Number of layers: 26\n",
      "Max training accuracy: 0.9637499988079071\n",
      "Max test accuracy: 0.9725\n",
      "Time elapsed: 96.52796877800006\n",
      "Number of layers: 27\n",
      "Max training accuracy: 0.9539583325386047\n",
      "Max test accuracy: 0.9608333333333333\n",
      "Time elapsed: 100.50080501299999\n",
      "Number of layers: 28\n",
      "Max training accuracy: 0.9414583325386048\n",
      "Max test accuracy: 0.955\n",
      "Time elapsed: 102.95860201400001\n",
      "Number of layers: 29\n",
      "Max training accuracy: 0.9331249952316284\n",
      "Max test accuracy: 0.9416666666666667\n",
      "Time elapsed: 105.94558353599996\n",
      "Number of layers: 30\n",
      "Max training accuracy: 0.9627083301544189\n",
      "Max test accuracy: 0.9658333333333334\n",
      "Time elapsed: 109.06207024900004\n",
      "Number of layers: 31\n",
      "Max training accuracy: 0.9820833265781402\n",
      "Max test accuracy: 0.9841666666666666\n",
      "Time elapsed: 116.19427680499984\n",
      "Number of layers: 32\n",
      "Max training accuracy: 0.9445833325386047\n",
      "Max test accuracy: 0.9508333333333333\n",
      "Time elapsed: 110.85007906200008\n",
      "Number of layers: 33\n",
      "Max training accuracy: 0.9435416519641876\n",
      "Max test accuracy: 0.9533333333333334\n",
      "Time elapsed: 122.74245930799998\n",
      "Number of layers: 34\n",
      "Max training accuracy: 0.9522916674613953\n",
      "Max test accuracy: 0.9641666666666667\n",
      "Time elapsed: 123.708737724\n",
      "Number of layers: 35\n",
      "Max training accuracy: 0.9539583206176758\n",
      "Max test accuracy: 0.9558333333333333\n",
      "Time elapsed: 124.7754761189999\n",
      "Number of layers: 36\n",
      "Max training accuracy: 0.9399999976158142\n",
      "Max test accuracy: 0.9483333333333334\n",
      "Time elapsed: 133.43835641799978\n",
      "Number of layers: 37\n",
      "Max training accuracy: 0.9322916626930237\n",
      "Max test accuracy: 0.9299999999999999\n",
      "Time elapsed: 132.9082774379999\n",
      "Number of layers: 38\n",
      "Max training accuracy: 0.9520833313465118\n",
      "Max test accuracy: 0.9625\n",
      "Time elapsed: 136.96099494600003\n",
      "Number of layers: 39\n",
      "Max training accuracy: 0.95625\n",
      "Max test accuracy: 0.9616666666666667\n",
      "Time elapsed: 139.14721957200027\n",
      "Number of layers: 40\n",
      "Max training accuracy: 0.9591666638851166\n",
      "Max test accuracy: 0.9608333333333333\n",
      "Time elapsed: 143.6257962740001\n",
      "Number of layers: 41\n",
      "Max training accuracy: 0.9589583396911621\n",
      "Max test accuracy: 0.9616666666666667\n",
      "Time elapsed: 147.88158249199978\n",
      "Number of layers: 42\n",
      "Max training accuracy: 0.9489583194255828\n",
      "Max test accuracy: 0.9566666666666667\n",
      "Time elapsed: 151.21110700999998\n",
      "Number of layers: 43\n",
      "Max training accuracy: 0.9458333313465118\n",
      "Max test accuracy: 0.9575\n",
      "Time elapsed: 153.97183844999972\n",
      "Number of layers: 44\n",
      "Max training accuracy: 0.9616666555404663\n",
      "Max test accuracy: 0.9716666666666666\n",
      "Time elapsed: 159.0852983990003\n",
      "Number of layers: 45\n",
      "Max training accuracy: 0.9745833218097687\n",
      "Max test accuracy: 0.9833333333333333\n",
      "Time elapsed: 161.20173576299976\n",
      "Number of layers: 46\n",
      "Max training accuracy: 0.9572916746139526\n",
      "Max test accuracy: 0.9633333333333333\n",
      "Time elapsed: 166.10677258900023\n",
      "Number of layers: 47\n",
      "Max training accuracy: 0.9537500023841858\n",
      "Max test accuracy: 0.9608333333333333\n",
      "Time elapsed: 168.9513080679999\n",
      "Number of layers: 48\n",
      "Max training accuracy: 0.9356249928474426\n",
      "Max test accuracy: 0.9583333333333334\n",
      "Time elapsed: 169.5284848909996\n",
      "Number of layers: 49\n",
      "Max training accuracy: 0.9456249892711639\n",
      "Max test accuracy: 0.9575\n",
      "Time elapsed: 176.40269118800006\n",
      "Number of layers: 50\n",
      "Max training accuracy: 0.95625\n",
      "Max test accuracy: 0.9625\n",
      "Time elapsed: 176.96777740500056\n",
      "Number of layers: 51\n",
      "Max training accuracy: 0.9529166638851165\n",
      "Max test accuracy: 0.9583333333333334\n",
      "Time elapsed: 181.88168530199982\n",
      "Number of layers: 52\n",
      "Max training accuracy: 0.9689583361148835\n",
      "Max test accuracy: 0.9758333333333333\n",
      "Time elapsed: 184.07610964200012\n",
      "Number of layers: 53\n",
      "Max training accuracy: 0.9539583325386047\n",
      "Max test accuracy: 0.965\n",
      "Time elapsed: 188.853444589\n",
      "Number of layers: 54\n",
      "Max training accuracy: 0.9410416662693024\n",
      "Max test accuracy: 0.9516666666666667\n",
      "Time elapsed: 190.77266577299997\n",
      "Number of layers: 55\n",
      "Max training accuracy: 0.9656249940395355\n",
      "Max test accuracy: 0.9716666666666666\n",
      "Time elapsed: 197.64772129099947\n",
      "Number of layers: 56\n",
      "Max training accuracy: 0.9491666674613952\n",
      "Max test accuracy: 0.955\n",
      "Time elapsed: 196.45266620500024\n",
      "Number of layers: 57\n",
      "Max training accuracy: 0.9552083253860474\n",
      "Max test accuracy: 0.9633333333333333\n",
      "Time elapsed: 199.68249412000023\n",
      "Number of layers: 58\n",
      "Max training accuracy: 0.9643750011920929\n",
      "Max test accuracy: 0.9683333333333334\n",
      "Time elapsed: 207.11003170700042\n",
      "Number of layers: 59\n",
      "Max training accuracy: 0.9341666638851166\n",
      "Max test accuracy: 0.9466666666666667\n",
      "Time elapsed: 208.2817393819996\n",
      "Number of layers: 60\n",
      "Max training accuracy: 0.943958330154419\n",
      "Max test accuracy: 0.9525\n",
      "Time elapsed: 213.32367455700023\n",
      "Number of layers: 61\n",
      "Max training accuracy: 0.9520833313465118\n",
      "Max test accuracy: 0.9633333333333333\n",
      "Time elapsed: 213.84854820200053\n",
      "Number of layers: 62\n",
      "Max training accuracy: 0.9643750011920929\n",
      "Max test accuracy: 0.9758333333333333\n",
      "Time elapsed: 221.6414672810006\n",
      "Number of layers: 63\n",
      "Max training accuracy: 0.9449999928474426\n",
      "Max test accuracy: 0.9566666666666667\n",
      "Time elapsed: 227.17016447800052\n",
      "Number of layers: 64\n",
      "Max training accuracy: 0.9758333206176758\n",
      "Max test accuracy: 0.9825\n",
      "Time elapsed: 224.68142282600002\n",
      "Number of layers: 65\n",
      "Max training accuracy: 0.9599999964237214\n",
      "Max test accuracy: 0.9683333333333334\n",
      "Time elapsed: 233.68502481799987\n",
      "Number of layers: 66\n",
      "Max training accuracy: 0.9558333396911621\n",
      "Max test accuracy: 0.9674999999999999\n",
      "Time elapsed: 231.817409753\n",
      "Number of layers: 67\n",
      "Max training accuracy: 0.9625\n",
      "Max test accuracy: 0.9666666666666667\n",
      "Time elapsed: 236.2705853870002\n",
      "Number of layers: 68\n",
      "Max training accuracy: 0.9316666543483734\n",
      "Max test accuracy: 0.94\n",
      "Time elapsed: 235.1397873450005\n",
      "Number of layers: 69\n",
      "Max training accuracy: 0.9393749952316284\n",
      "Max test accuracy: 0.9491666666666667\n",
      "Time elapsed: 243.93588946099953\n",
      "Number of layers: 70\n",
      "Max training accuracy: 0.970208328962326\n",
      "Max test accuracy: 0.9741666666666667\n",
      "Time elapsed: 245.96755173400015\n",
      "Number of layers: 71\n",
      "Max training accuracy: 0.9562499940395355\n",
      "Max test accuracy: 0.9633333333333333\n",
      "Time elapsed: 251.83261913800015\n",
      "Number of layers: 72\n",
      "Max training accuracy: 0.9585416615009308\n",
      "Max test accuracy: 0.9625\n",
      "Time elapsed: 251.7219314949998\n",
      "Number of layers: 73\n",
      "Max training accuracy: 0.9447916686534882\n",
      "Max test accuracy: 0.9491666666666667\n",
      "Time elapsed: 255.99749294700086\n",
      "Number of layers: 74\n",
      "Max training accuracy: 0.9591666698455811\n",
      "Max test accuracy: 0.9758333333333333\n",
      "Time elapsed: 262.55268666399934\n",
      "Number of layers: 75\n",
      "Max training accuracy: 0.9316666603088379\n",
      "Max test accuracy: 0.9325\n",
      "Time elapsed: 266.5499626799992\n",
      "Number of layers: 76\n",
      "Max training accuracy: 0.964166659116745\n",
      "Max test accuracy: 0.9716666666666666\n",
      "Time elapsed: 267.45401303200015\n",
      "Number of layers: 77\n",
      "Max training accuracy: 0.9693750023841858\n",
      "Max test accuracy: 0.9725\n",
      "Time elapsed: 268.9482905570003\n",
      "Number of layers: 78\n",
      "Max training accuracy: 0.9556249976158142\n",
      "Max test accuracy: 0.9608333333333333\n",
      "Time elapsed: 273.71390450500076\n",
      "Number of layers: 79\n",
      "Max training accuracy: 0.9741666555404663\n",
      "Max test accuracy: 0.9825\n",
      "Time elapsed: 281.26134984999953\n",
      "Number of layers: 80\n",
      "Max training accuracy: 0.9558333218097687\n",
      "Max test accuracy: 0.9641666666666667\n",
      "Time elapsed: 281.6825851070007\n",
      "Number of layers: 81\n",
      "Max training accuracy: 0.9443750023841858\n",
      "Max test accuracy: 0.9533333333333334\n",
      "Time elapsed: 287.7709215800005\n",
      "Number of layers: 82\n",
      "Max training accuracy: 0.9770833373069763\n",
      "Max test accuracy: 0.9825\n",
      "Time elapsed: 288.3404934530008\n",
      "Number of layers: 83\n",
      "Max training accuracy: 0.9610416531562805\n",
      "Max test accuracy: 0.9691666666666666\n",
      "Time elapsed: 292.8539756090013\n",
      "Number of layers: 84\n",
      "Max training accuracy: 0.9358333230018616\n",
      "Max test accuracy: 0.9408333333333334\n",
      "Time elapsed: 294.8310821089999\n",
      "Number of layers: 85\n",
      "Max training accuracy: 0.9689583301544189\n",
      "Max test accuracy: 0.9716666666666666\n",
      "Time elapsed: 302.378126489999\n",
      "Number of layers: 86\n",
      "Max training accuracy: 0.970208328962326\n",
      "Max test accuracy: 0.9791666666666666\n",
      "Time elapsed: 303.352288991\n",
      "Number of layers: 87\n",
      "Max training accuracy: 0.9627083241939545\n",
      "Max test accuracy: 0.9708333333333333\n",
      "Time elapsed: 301.16030641900034\n",
      "Number of layers: 88\n",
      "Max training accuracy: 0.9572916567325592\n",
      "Max test accuracy: 0.9633333333333333\n",
      "Time elapsed: 306.22613750299934\n",
      "Number of layers: 89\n",
      "Max training accuracy: 0.9570833325386048\n",
      "Max test accuracy: 0.965\n",
      "Time elapsed: 315.77566529199976\n",
      "Number of layers: 90\n",
      "Max training accuracy: 0.9495833337306976\n",
      "Max test accuracy: 0.9591666666666666\n",
      "Time elapsed: 315.8843512000003\n",
      "Number of layers: 91\n",
      "Max training accuracy: 0.9727083325386048\n",
      "Max test accuracy: 0.98\n",
      "Time elapsed: 316.10622666200106\n",
      "Number of layers: 92\n",
      "Max training accuracy: 0.9472916603088379\n",
      "Max test accuracy: 0.9541666666666667\n",
      "Time elapsed: 331.1514584810011\n",
      "Number of layers: 93\n",
      "Max training accuracy: 0.9770833253860474\n",
      "Max test accuracy: 0.9841666666666666\n",
      "Time elapsed: 325.98184867099917\n",
      "Number of layers: 94\n",
      "Max training accuracy: 0.9631250023841857\n",
      "Max test accuracy: 0.9725\n",
      "Time elapsed: 331.6532540799999\n",
      "Number of layers: 95\n",
      "Max training accuracy: 0.9497916638851166\n",
      "Max test accuracy: 0.9525\n",
      "Time elapsed: 334.3661037410002\n",
      "Number of layers: 96\n",
      "Max training accuracy: 0.9533333241939544\n",
      "Max test accuracy: 0.965\n",
      "Time elapsed: 337.9216146300005\n",
      "Number of layers: 97\n",
      "Max training accuracy: 0.9541666567325592\n",
      "Max test accuracy: 0.9658333333333334\n",
      "Time elapsed: 334.56876583300254\n",
      "Number of layers: 98\n",
      "Max training accuracy: 0.9377083361148835\n",
      "Max test accuracy: 0.9416666666666667\n",
      "Time elapsed: 341.52659573300116\n",
      "Number of layers: 99\n",
      "Max training accuracy: 0.956458330154419\n",
      "Max test accuracy: 0.9600000000000001\n",
      "Time elapsed: 348.8388315089978\n",
      "Number of layers: 100\n",
      "Max training accuracy: 0.9464583337306977\n",
      "Max test accuracy: 0.9516666666666667\n",
      "Time elapsed: 346.89844970800186\n",
      "Number of layers: 101\n",
      "Max training accuracy: 0.9777083218097686\n",
      "Max test accuracy: 0.9858333333333333\n",
      "Time elapsed: 352.53626513400013\n",
      "Number of layers: 102\n",
      "Max training accuracy: 0.9633333325386048\n",
      "Max test accuracy: 0.9725\n",
      "Time elapsed: 356.47908818699943\n",
      "Number of layers: 103\n",
      "Max training accuracy: 0.9304166615009308\n",
      "Max test accuracy: 0.9450000000000001\n",
      "Time elapsed: 359.5727004210021\n",
      "Number of layers: 104\n",
      "Max training accuracy: 0.9397916674613953\n",
      "Max test accuracy: 0.9491666666666667\n",
      "Time elapsed: 361.8862167099978\n",
      "Number of layers: 105\n",
      "Max training accuracy: 0.9504166662693023\n",
      "Max test accuracy: 0.9575\n",
      "Time elapsed: 364.6247943769995\n",
      "Number of layers: 106\n",
      "Max training accuracy: 0.9735416650772095\n",
      "Max test accuracy: 0.9816666666666667\n",
      "Time elapsed: 370.83505268099907\n",
      "Number of layers: 107\n",
      "Max training accuracy: 0.9431250035762787\n",
      "Max test accuracy: 0.9466666666666667\n",
      "Time elapsed: 375.58876416699786\n",
      "Number of layers: 108\n",
      "Max training accuracy: 0.9637499988079071\n",
      "Max test accuracy: 0.9741666666666667\n",
      "Time elapsed: 378.6039586450024\n",
      "Number of layers: 109\n",
      "Max training accuracy: 0.9714583218097687\n",
      "Max test accuracy: 0.975\n",
      "Time elapsed: 382.57079584200255\n",
      "Number of layers: 110\n",
      "Max training accuracy: 0.9816666603088379\n",
      "Max test accuracy: 0.9891666666666666\n",
      "Time elapsed: 392.05546256800153\n",
      "Number of layers: 111\n",
      "Max training accuracy: 0.9612499892711639\n",
      "Max test accuracy: 0.9700000000000001\n",
      "Time elapsed: 387.0121085969986\n",
      "Number of layers: 112\n",
      "Max training accuracy: 0.9495833277702331\n",
      "Max test accuracy: 0.9566666666666667\n",
      "Time elapsed: 387.39702782899985\n",
      "Number of layers: 113\n",
      "Max training accuracy: 0.9685416638851165\n",
      "Max test accuracy: 0.9791666666666666\n",
      "Time elapsed: 398.58878781399835\n",
      "Number of layers: 114\n",
      "Max training accuracy: 0.9610416650772095\n",
      "Max test accuracy: 0.9716666666666666\n",
      "Time elapsed: 392.67270083199764\n",
      "Number of layers: 115\n",
      "Max training accuracy: 0.9408333361148834\n",
      "Max test accuracy: 0.9416666666666667\n",
      "Time elapsed: 400.4344038390009\n",
      "Number of layers: 116\n",
      "Max training accuracy: 0.9539583325386047\n",
      "Max test accuracy: 0.965\n",
      "Time elapsed: 403.36959829600164\n",
      "Number of layers: 117\n",
      "Max training accuracy: 0.9335416555404663\n",
      "Max test accuracy: 0.9441666666666666\n",
      "Time elapsed: 409.7208005709981\n",
      "Number of layers: 118\n",
      "Max training accuracy: 0.9520833313465118\n",
      "Max test accuracy: 0.9600000000000001\n",
      "Time elapsed: 410.06581206499686\n",
      "Number of layers: 119\n",
      "Max training accuracy: 0.9791666626930237\n",
      "Max test accuracy: 0.9841666666666666\n",
      "Time elapsed: 414.80154957099876\n",
      "Number of layers: 120\n",
      "Max training accuracy: 0.9556249856948853\n",
      "Max test accuracy: 0.9683333333333334\n",
      "Time elapsed: 415.23337027699745\n",
      "Number of layers: 121\n",
      "Max training accuracy: 0.9281249940395355\n",
      "Max test accuracy: 0.9383333333333332\n",
      "Time elapsed: 421.3018950170008\n"
     ]
    }
   ],
   "source": [
    "from Networks.ResNet import ResAntiSymNet\n",
    "import numpy as np\n",
    "import timeit\n",
    "\n",
    "num_epochs = 100\n",
    "num_features = 2\n",
    "num_classes = 2\n",
    "size_hidden = 8\n",
    "gamma = 0.3\n",
    "h = 0.1\n",
    "lr = 0.1\n",
    "\n",
    "max_layers = 120\n",
    "train_results_per_layer = np.zeros(max_layers)\n",
    "test_results_per_layer = np.zeros(max_layers)\n",
    "basic_save_key = 'AntiSym_Comparison_results//AntiSymNet'\n",
    "\n",
    "for num_hidden_layers in range(1,max_layers+1):\n",
    "    tic = timeit.default_timer()\n",
    "    num_layers = num_hidden_layers + 1\n",
    "    print('Number of layers: '+str(num_layers))\n",
    "    train_results_per_epoch = np.zeros(num_epochs)\n",
    "    test_results_per_epoch = np.zeros(num_epochs)\n",
    "    for i in range(10):\n",
    "        net = ResAntiSymNet(features=num_features, classes=num_classes, num_layers=num_layers, gamma=gamma, h=h, bias=True, hidden_size=size_hidden)\n",
    "        net.set_test_tracking(True)\n",
    "        net.train(num_epochs, train_loader, test_loader, print_output=False)\n",
    "        train_results_per_epoch += net.avg_correct_pred.numpy()\n",
    "        test_results_per_epoch += net.test_results.numpy()\n",
    "        \n",
    "    train_results_per_epoch = train_results_per_epoch / 10\n",
    "    test_results_per_epoch = test_results_per_epoch / 10\n",
    "    train_save_key = basic_save_key+'_train_per_epoch_layers_'+str(num_layers)+'_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "    test_save_key = basic_save_key+'_test_per_epoch_layers_'+str(num_layers)+'_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "    \n",
    "    np.savetxt(train_save_key, train_results_per_epoch, delimiter=',')\n",
    "    np.savetxt(test_save_key, test_results_per_epoch, delimiter=',')\n",
    "    \n",
    "    train_results_per_layer[num_hidden_layers-1] = train_results_per_epoch.max()\n",
    "    print('Max training accuracy: '+str(train_results_per_epoch.max()))\n",
    "    test_results_per_layer[num_hidden_layers-1] = test_results_per_epoch.max()\n",
    "    print('Max test accuracy: '+str(test_results_per_epoch.max()/test_set_size))\n",
    "    \n",
    "    toc = timeit.default_timer()\n",
    "    print('Time elapsed: '+str(toc-tic))\n",
    "\n",
    "train_save_key = basic_save_key+'_train_per_layer_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "test_save_key = basic_save_key+'_test_per_layer_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "np.savetxt(train_save_key, train_results_per_layer, delimiter=',')\n",
    "np.savetxt(test_save_key, test_results_per_layer, delimiter=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x26cd75a5eb8>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd8FNUWwPHfSQ8Qeu+hg4gCEUFQAUEpKqCoNLti1+ez8uxdQbEiKBYU6VgoooDSlCIE6T300GsgpO+e98csS4CQLJjNkni+nw8fdmbvzpy7s5kzc+/MHVFVjDHGGICgQAdgjDHm/GFJwRhjjJclBWOMMV6WFIwxxnhZUjDGGONlScEYY4yXJQVjjDFelhSMMcZ4WVIwxhjjFRLoAM5W6dKltXr16oEOwxhj8pXFixfvV9UyOZXLd0mhevXqxMbGBjoMY4zJV0Rkqy/lrPnIGGOMlyUFY4wxXpYUjDHGeFlSMMYY42VJwRhjjJclBWOMMV6WFIwxxnhZUjDGGONlScEYY4yXJQVjjDFelhSMMcZ4WVIwxhjjZUnBGGOMlyUFY4wxXn5LCiLylYjsFZGVZ3hfROQjEYkTkeUi0sRfsRhjjPGNP88UhgEdsnm/I1Db868vMNiPsRhjjPGB35KCqs4BDmZTpAvwrToWAMVFpIK/4jHGGJOzQPYpVAK2Z5qO98wzxhgTIIFMCpLFPM2yoEhfEYkVkdh9+/b5OSxjjPn3CmRSiAeqZJquDOzMqqCqfq6qMaoaU6ZMjs+dNsYY/0pPhqWjIC0p0JHkukAmhYnAbZ6rkJoDCaq6K4DxGIC0Y2QsGweujEBH4kg9CruzvIAt/0hNBLc70FGcTtX5fs8k7Ri4XXkXz86lsHlO7i0vYQeM7AE7l+TeMsH52xh3J/x0P0x7PneXDeBKh2Vjst82fuTPS1JHAfOBuiISLyJ3i8j9InK/p8gUYBMQBwwFHvRXLMZ3O0Y8SMiP97Bz7vBAhwIJ8fBFOxjSCrb9Fehozk16MgxuAcM6nX9Hlb8+CwMvgAMbT3/P7UI/b4t+3hqSD2W/nC1/Ojuxs0l8i7+BDdNPXt+YPvDNdTB/kO/LyU7sl7D+FxjezfcDiyXfwUeNnUSeFVV00iOw/hdWSm1nHRtnnl7uwEZYMgIy0s467Iz5Q+DHvmSMuztvk7KHP68+6qmqFVQ1VFUrq+qXqjpEVYd43ldVfUhVa6rqhaoa669Y/tWSDjpHhD44GjuWSlt/wqVC+qJv/BxYDvauhS+vxnV4B8fCSqMTH4GM1HNf3uJhMKoXJO7N+n0fv6NsJe6FkbfAqp8yrfcbOLwNts2HcXc4R4Fn4+BmWDY65x3u3jUw7Fo4tv/k+a50SNwH++Mg+fCJ+TuXoH99BqkJ8EPf0+LStZOR/WuR3cvR4TdCypHT13loC4zpA8M6w4994euOsG99znXatgAmPYr+cO+Jo+F1U5CE7WwOqg5T/wfTX/xn28TtIjX2O5a4a3HUFQbfdnF+U9lJPYp7+ktwcBOsmXRifsoRmN0fJj+O69suyNKRfJBxA7ekPs82qYROePjk78ftgnG3w4QHnQOC9dN8jzslAdfsAezWEoTETYUZr59dvXOB3dEcKNNfhI9jsj7KyC0HN8PA+jC6V85HqQk7CJryX5a5azIyshfVji7BvS/Of7FlZ+8a+LoD6elp3JL+Ag8m3oXsXwdz3j235W1fiP78BKz7GYa2Pf2oMS0JPr8Sfn4y6x1R8mHnqDa7o7YjO+HrTrD+V5jwMBzeDunJpM95j/muBjyXfhdsmOrsgNf9AlvnO/U8shPSU7JepipHR90NP96Ha/yd2W/DGa/Dlj9g9YQT8w5ugrerwru14JOmpA28ELbOA7cb9+QnOCTFeDb9HtgRe9p3mzjrI7a5y3Bf2uO4dy2FEd1PitO1fxPpH19K2rrp/FTyLsZX6Yd771oY0hJmvOE0PWXFlY570n84TBSSfMhJTEDyn4OI19JcnfQqU8I7wdwPYf4nZ65vTjbNJDx5D59ndOa6o0+T7MJJYNltw3mfEJS0n4NahGOLMp0pz34HZr5B+soJ7Noax8cZXXFd/gxD7mzJYyl90SM7YcpTJ347S4bD7hVsr3e3c+XMyJtgwRCfwnb9+SHh6QncnfYko91XwZ8DYcX4c/0WzoklhUDISHOOXA9uhOFd4aeHsj4S+4f0ryG4Xenoul+c9SSd4bYRVfaN7AuuNFZc+i5lWvclQ4PYM+eL08tuW+DEnlOfg9sNe1Y5p9HHDpxe3pWR9dHv4W0w/AbSCeH65BfYU6g2IXXa84Prctx/DHSO4M5Uj+OSDsIRT/dU8iFcY+9kl7skPdOeIzklFb66BuJ+P1H9mW/ArmWwaKhTt+PiF8P396Lv1oUR3Umd80HW6zu0Fb7uiB7dxewL3sCtbpj4MCweRmjSXoZId1ZV7M6H7pth1Q8wqgd83QE+bQ4D6+N+pzpsmXvaYnXDdKL2LWaW6yJk9QT0606w6kfnQOLg5hMF96yGtZOd1+unnviKl42D9CT66238N+1+tqcWJuObLjDxYYJ2Lub11B4sKH4tk7gCndPfaQby1DtqbywjpTMlmt7Af1IfgO1/OTs7j+2/foC60unmHsCgjK78b/NFXJH0DutKtoU5/Z0DnjnvwtTnYOzt8Of7TjPU/EEE7VvDU2n38rurMRl/fgzbFhC5Yz4j3NfwRvcmPHK0D7ERzdEZb5xo2ko9CpMeg686wqctnLOi9OQz/gSO/fUth7QItS/vTtXajXgysQ8c2HDyGUBmiftwzf2YKa5mDMvoQOSOeU5iTzqIxn5NbNF21D78MbeEfcIFffrzxDX1uLx2GWo0bsPHGd1g+WinrikJ6O+vsTKkAZcvbUvP4IEcK38pzPvo5LOxuR+d3HwGcHQ3Ov9TJrma06PLdbzuvpNNkRfCz0/kadNjSJ6t6XygCpLVlbB+knoUkg5ASgIUqwKFSjrzN82ElARGR79O1dQ4mi8bTsaRvYT1GQNBuZSnUxJwLR7OxIwWzAttTv8dHxI0tA1c/iQ0uhlCwr1FMzb8Rpk9f/J54b7c1aE1qRlu5vzSmEvWjAXX6xDs+ZnEfg1TngR3hvO6yydQ/sKs17/4K+fHnElGSCFSJJJQVzLh7iRSI0oT3qQXXNQTwqPg2D74oS+u1ER6pL1AYqEqjO7bgqIRIfT+6F5aJa2i7Jg+zsKKVYUKjaB8IygZDRHFnD+6FeNg7c/gToeKTXAHBaNHd/GY6xUyKjfhqviKTC/3CYVH94JeYyG8CCz4lFEZbagQdJArpjxFUMkazs5j0RekhxbhR9cVVHDtpPmcAdC4BxTLdDvN9oUwuhfqSmNAuXf4dHEJHi12G//dNBjdMpeF7gZUanw1A9rVpssnKczUq+jbpAiXVQxhz749zFy6ga5HviNs8ouUfPhEokKVpKmvcMBdhi8qv8G3W2cxeM8gwsfd4bwtQdDtc6TRTfDHe2hoYTaVbk2Nzb8jaUkQVojEv8ez3l2HrfXu5Lbm1fhsbgd6bniSxktHsMhdh6KX3so3raK56cM7iQndSPnvbkS6DCJt1WRSNZLUC3vxWteG9NmXyNJdv3LBnx8TGnMXZKRQduN4ZgZfxsTnbyU4SNh+MIk3p6zhmpW3cXn4ZbzjGknFGa/hDg7HVbgcoat/gtkDUHUxQ2Nw1+nEiB3luCrtGdyjepKqYRxt0JObY6oQFhzEQ2P6MDNyORETHyWox3cw4ibY8TdUbQHFKsOGabBwKLR81Pm+9q5F532IXPYoRJUnPO4Xvne3ptdldbgvIoQeQ5LYemgsVf8YiDToctp+QOf0h4xkvgrvQ4MqpQnaPJ6Uv0cREaRI+jFeSGrPI21r8WDrWkSGBXs/91zn+rRb24NqIWl0XTAI4qZD0gGeSX2cOy6LZtKynTyW0oovQt9zEvcF3ZzkO/0FCC8GDy+CqHIAuGe8Aa40fih+J19dWpWdh5N5ZnY3xoW/CivGQtM7fPvb/4f+PUlh4wz4YyDc+AVElff/+pZ8BxMfBfWcrpa/EPrOgaAgdNUPJFKYV9dXI52a9ER5ddM3pM0aQFjbZ3Jl9br4G0IyjvFTeBf2FqlHj72F+CRiFGUnPgy/vwqd34UGXcDt5tjk50hwl6FO58cICQ4iJDiI9RW70XbX87hWTSC4VDQsHQGLvsBVsx2bynWgSuybhH7WmqQm9xLVvp+zU/auXNFFX5Jcoi7Tit/ClvgduJMSKJpxjKigFJKJ5LA7kgbHtnDVvEEEz/voxEdDIvhv6ItsCY7mp3uaU6l4JABv9m7NNZ++SyM20L7kXlqEb6fGvrXI8SNkD3dkSWYUuZ64pEg6HVxE1ZR1vJHeh9tuvpEr65bhxk8z6HDoCX4rMYDwUT3QqPIcoDgjit1LRJBS9+hjVPj2ehRhdvEbeGh3J2pWLk959x4uOXg/wVOfI+jmYc4BxvKxMPER3FEVeL7YW4zcUIjuTSvz0eJWdC37NzWO/MXA9Bt5pUU1ykZFMPS2GB4dvYQHZx0DXEBpyhetTGLwMZ7c/5VztlC9pVORNRMpfGAlb8qDfHZHCz6bXZaYGfWoIAcoxjGeCB1Hsx/uQ47sQFf9wPfhN/Dj1tqMCPsZNs+GUrUodnQ9CyLu5pOejRERGle9koe/GUi9Td+ypER7hnaqT0RoME9efwmdxr/AhDJDqPr93YQiDHN14sYW9QgNDuLj3k14Y8D1fHjEOVM7fHAvxfUYhy64jeAgZ+dapWQhBvdpyuKth/hmXkVar6xJYddRDlMEPRZE+5J7ea3sbIJ2/c3LSbfz7bUNmLexLL9PHs1VyUv40dWWHlc0AqBr40ocS7uC1yb25K2tX6KDLoWkgyy65H2WFmlFQnI6PQ8eo9If7yFNboOQcI4M70XRoxtxLxuLVmtJiKYRX+0GyheLAODR9vUZNKIz/XcPdQ7KarZ1vufUo87fw8KhjMloQ/dObWhYqRh/fVaPeou+JdidyGxXY65s1Zonrq572t9ZycJhfNCjMX2HZyDhaXTZP5VxrtbUvbgVL19/Af+9ug4PDi/EtvjvKDf3U8Lrd4Fpz5MSUYaw9CMETe0H3b+CpSMJWvItQzM6c0O7KxARHmhdkyv/asiWkJpUXzAEmtyeJwe1/56kkJIA8bHw2RVw0zAoXcfpwNu1FDq9C5HFc29d2xbApP9Atcuco+DDW512yZXjoUEX3Kt/5peMGF7scjE9mlVl5pqmTBgZx3Vz3sJVJYbg2ledWJYrHRK2Q8kavq/flUHqvMEsdden3VVXc1PTKrwwoRjNFtflhQb7uCvlW2TcnXDTMDQjhWJH1vFJ5OP0q3/iCDi6RTf2fv8uZX+4yztvZ7076bmlM1tXpVGMt+kXMpKbFw/BtWYcwde8ARf1cArGL0L2rubV9Hv4YW8DLq99BVfWLUPTaiWoWy6KkOAgMlxuBkxdx3Nz/ub20mtpVbsM1SpV5M3Fwfy8NZyR9zalSslC3nU3rFSM4Q+24+cVFzBq/T6e33aEp66py0OXlYOjuyElgfj9h7lrurL5QDpX1inD59u6kpqSwG2tG3LdRRUB+PrOS7jh03lcd+RpJke9SdjBjTyT9gT3dW1Mw0rFePTjp3hAvueT1E6sPVife9pG88hVtZmzfh+Dv7uOx1d/DzPrOn0Hu5ZypNyl3HHsEZZsDuLNbhfS69KqhIUE0XXhPbSKbI9Wb0m98kW9dfj9v1eyYW8i01fvoWhECDfFVGHEn+XZN2s8kb+9TZF7JkFKAhm/vc5WrUhYk54UDg/h8fZ1iC5TmMRUF/XLRzE5thUhyx8m5reXSJNw+h9pR6GiJTmWEknkul84GFKO0kCJpt0Rz44kLCSID29txSczy/FS40pEhDpHvDc1rczy+Ia0W/A4w8uP4YLDs1lQujt9KzmJvmxUBJVbdGfz/BGUn/0+GYmJrHZXo2Wbzqf99JpWK0HTaiXYe7Q+f289RLpLOZKSztA5m2i+tixwE3e1jCa6dGEqFY+k74xbqZq8l78q9KJXpRMHFr0vrcbX6Q8xf+p8miRu4L60/zBrTmlgLUECM+nMlPC5MPdDEhOPUPToRp50PUwr/qbrltmscVeldet23uW1qVuGlyOv4pD+QIk/BkK5hs6R+5z30CM7GB/ciTHF7uCHmCoEBwmfFu/ApUec5sLvCz3NwKtqn/HP7Yo6ZRh5bwvu/jqIKWkN2FA4hh+vvwCAohGhvN29Md+9fw3P7vwWnf4CsnMJ/dIe4IJCh7hn5WgoWx+d3Z8lwY0YV/gupl5YwfvZB9rUYtDUqxiQ/rlzuW6NK8/8d59LRHPjqos8FBMTo7Gx53ahUvrOFYSOv925akKCnCYGIL3pPYRe917uBJgQD5+3xhUWxYxWo0gMKkJGhosuf/UiLOMotH8Vxt3OPa5nef+5J4mKCAVgzNy1XDS1O9VCDxHReyRS40qnn2FMH+fI7+rX4bJHnHXs3wALBjt9Eoe2QvGqcOUzHCnfjKhQYNGXyK/P8GTIs7z+zNNEhAajqrzz6zqGzN5I74tL8nrii8jOJaSHF2PDscIs7vATt152IvGkpLt4/PX+XB65mdBKF7E+qCZDV7ioXqoQ/TrVp175KPYnpvH2l6N4OfQbLnCthV7joM7VHBl9L0FrJvJM9XG83bOFt45Z+XXlLp79YQWHk060tx7fuZ6JqvLIqCX8unI34x+4jIurFGfaqt08MXYZ4aFBDOnTlJjqJXG7lR2Hk6lcItK7YwSI25tIz6ELKOY+TKOgLawtcimTH2lFUJAwY+0eXp+8hi4XV+K2FtUoUTgMAJdbuWbAVEakPUY5127cJWsysVA3noprRNniUbzerSFt6pYFIDE1g2ven8OOw8l80qsx1zaqmO1P5nBSGp+/9R+eDh4BHfvDvE9wJ+zgnrTHee7xx6lZpshpn3G7lX6j/uSatc+z0F2PGl2fp0ThMNJG9aFt4S0kSDF2HoOqT8+ldJHwLNZ6+nf68Yw4Bk5fTwgZvNLtYnpfWu2kGD9+px8viNPP9EWJx7nnsZdzXO5xqRkuhs3dwp9x+/m4Z2OKF3K+17Gx23l6/HKG9GlKh4ann8GPnreW2NVx1Kpdn1a1SlO9dGEKhQbz2JiltFv9P64LXUSQO53h2pGWD3/B6EXb+fuPKRQuWY5hT/Y+absPmLqWtD8+4rmQETgDKihJJeryYMJtLHbXZtidzWharQQAv8Sup/WklizXGiT3nkRrz7bNTtzeRF6ZtIqH2tSieY1SJ7333azldJvZjsKSykp3dd6sNJjFW/Yyt+gLlE7dxsGwirQ98hIf3XUVV9Q5cYNuYmoGbd76lRlBDxJVuxX0HOnzd34qEVmsqjE5lvu3JIWxsdsZMnsjP919IUXnD0AliP9tuZh68WO5NeR3gu6b7bRR71kNs9+G4tUg+koIjYQtf6C7VyCX3AM125xYqNvldPot+Ra2/ImqG01LJk2DuDH9NValnxjf79pCq/nE/ToaWZKEpDRebzCBd285efsM+WkGbf9+mJrBuwlu97LTPr5nFVRp5lzS2OJhiCzhXB4XFAJl60GxKmRsmUdI0l6WuGtRI2g3xXCO5BZe/SN3tKrpXb6q8tHvcbz/23raVo9gsPsVwvcu40H6MaDfkxQOP/nEcdDMOEYt3MbBY2mkZri5tXk1nu5Ql0JhJ8rN27ifvl/PZVL4i1QJSyTorqmkD7qMSVzOFU+MoGxURI7bJsPlZvWuI/y16SCRYcH0aV4tx88kJKXT8cM5hIYE0eGC8nw2ZxMXVirGkFubepucsnM8Mew7msrXd1xCm3o5/9F/8ccmRkyZweuti/DyynLE7U/i7pbRPN6+zmnf3dLth/lpyQ6e61yf0OCc+4leGDOfJ1Z3p7gkklG0KvcnPUBqhaYMv/vSM34m3eXmtcmrqVW2CLe1qI7brbw34CWeSv4QgPFlHqT7Q2/luO7Mxi+OZ+KynXzauwlFTqnTx1OX02teJ0JwMff6P+nUtOYZluI7VWXVziNcULHoSTvwnCSlZfDAxz8w9Mj9bNYKLO/4Ize1qAPAgk0HKFk4jDrlok76zLYDSXQY8As/VhlDnfoXMTnjEv47O50qJQoz9PaYk5JvusvNY/0/o1yVGrzU55p/XE+XW/m1fx86p0zmw0rv8dBdd/HKpNWsWTiNQaXGc+v+27isxeW87DnDyGzgtHWEznmTh0MmII8ucfrQzoElhVMs3nqImz+bzzUXlGNQryZMXbWH+79bTHThdMZnPEJUpXqEXf0yjO7ptBenJ3vPJBQhgcJESQrB3YbAhd2dS/+mvwiHt5ISVoK/Qi8lPlFJc8EvtKL6xVdyc0wVShUJZ39iKrd+uYDvC73NBanLGJXRhui7vjztaEJVeXncfNqseJbWwctwh0QyouprTE5qwNuFRxK98TsA0upez8waT7L4QBgr4hNYunknd4XN4PZCc4kLiua7hItYU+gSfnnqGm8TQWZjFm3j9clrCHcnUt+9kQaXXUu/zg2y/f5cbvW2H5/q9zV7+HTMREZqP1KCC1HMncBvl4+l3VX//I8pO39tOkDPoQtwK/RsVpWXrmuQZX3PZNuBJBZuOciNTSr5tENKSE6nxVu/k5TmonSRMD7s0ZiWtUr/kyp4rdyRwJBBA+hUcicvJHTmGIUYdmez034jOZk8fzmdfr2CIFEWdfuDSy5qlCvxARxJSeeptz8gTDJ497mnCQ/x/bv2hy37j/G/T76hfNVavHdne5+2Ye8vFrBlfxINKhZl+uo9tKlbhg96NKZY5OlnsynpLkKDg874uz9bO3bvZsWC6Vx1XW9Cg4M4kpJO+4Gz2XMklZplCjP5kctP6sQ+7kBiKt3eGc/M4EcIvuZ1aHFu9/laUsjC4FkbeefXtTzfuT5fz91CkfAQBvVuzBcfvcrbIZ87TUola3LghtGsSQjj8Lo/+GPNdn45Ek25opG8mvIWLYJWQ7kLYc8KdkfWpn/SdUxKvZjKpYvRqlZpmkWX5PLapb2nx8eNXriNET9OYEzYazwZ8QqDnr0/yx+xy608OmIRpdd+R6y7DhuCalKuWDjbDybxWKlFZESUZMiu2rjcSlhwEHXLR9G6bhnubhXtXefRlHRcbj0thsx2J6Tw/E8rWbDpAFMfv8Kno+vsHEhMZf6I17h218dsCa1Ntf8tOqsjv3M1YekOQoKC6Nwob0ZdHzxrI39vO8QbXRtStmjOZ0Fn48bB81i89RAdG5bn+WsbnNM2SXe5WfHG5QTj5sLn5xGUSzu04/7csJ8Mt9un5pS8kJCcTpHwEJ933JOW7eSRUUsIDRae6VCPu1pG5/p3dDZmrN3DCz+tYkifplxYudgZy708cRWzFixixNO3nPPfqiWFLLjdyh3DFjFnvTPS6pi+zbm0RilembCCdovvp1apMF6O7Mcvm060bzeoUJTnO9fnoirF6Tl4Ng8cepfWYWv4wHUzXyZfzrUXVaZ382rEVCuR7U5QVXl41BJ+Xr6TJ9rX5ZFsOq5SM1y8NWUtZaLCueWSKpQoFMaEpTv4ZEYcItChYXmublCeBhWL+tQ0kZ10l/sfL8PL7ebwL68QWe9qwmu2zJ1l/ovsPJzMniMpNK5a4h8tZ3O8M65kdOXs+zL+jdIy3Hw6K4529cvRsNKZd8J5SVVzPIDacTiZK/vP5NmO9bjn8rO46CQTSwpnsD8xlS6fzKVlrVL0734R4Bzlth4wk6OpLsoXjaBHsypcGl2KeuWjvB2N4GyYLp/M5UBiMvUqFKf/jY2yze6nOpKSzvD5W+nTvFqWp6vGGHMmm/cfI7p04XP+vCWFbKRluAkNlpOy88LNBzmclEbbemUJyebIef2eoyzeeojuTSvn3hG2Mcb4ma9J4d9zn0ImYSGn78ybRZf06bN1ykWddlWDMcYUFHaoa4wxxsuSgjHGGC9LCsYYY7wsKRhjjPGypGCMMcbLkoIxxhgvSwrGGGO8LCkYY4zxsqRgjDHGy5KCMcYYL0sKxhhjvCwpGGOM8bKkYIwxxsuSgjHGGC9LCsYYY7wsKRhjjPGypGCMMcbLkoIxxhgvvyYFEekgIutEJE5Ens3i/aoiMlNElojIchHp5M94jDHGZM9vSUFEgoFBQEegAdBTRBqcUux5YKyqNgZ6AJ/6Kx5jjDE58+eZQjMgTlU3qWoaMBrockoZBYp6XhcDdvoxHmOMMTkI8eOyKwHbM03HA5eeUuZlYJqIPAIUBtr5MR5jjDE58OeZgmQxT0+Z7gkMU9XKQCdguIicFpOI9BWRWBGJ3bdvnx9CNcYYA/5NCvFAlUzTlTm9eehuYCyAqs4HIoDSpy5IVT9X1RhVjSlTpoyfwjXGGOPPpLAIqC0i0SIShtORPPGUMtuAqwBEpD5OUrBTAWOMCRC/JQVVzQAeBqYCa3CuMlolIq+KyPWeYk8A94rIMmAUcIeqntrEZIwxJo/4s6MZVZ0CTDll3ouZXq8GWvozBmOMMb7L8UxBRBrmRSDGGGMCz5fmoyEislBEHhSR4n6PyBhjTMDkmBRUtRXQG+dKolgRGSki7f0emTHGmDznU0ezqm7AGZLiGeBK4CMRWSsiN/gzOGOMMXnLlz6FRiLyPs4VRG2B61S1vuf1+36OzxhjTB7y5eqjT4ChwP9UNfn4TFXdKSLP+y0yY4wxec6XpNAJSFZVF4BnGIoIVU1S1eF+jc4YY0ye8qVP4TcgMtN0Ic88Y4wxBYwvSSFCVROPT3heF/JfSMYYYwLFl6RwTESaHJ8QkaZAcjbljTHG5FO+9Cn8BxgnIsdHOK0A3OK/kIwxxgRKjklBVReJSD2gLs4zEtaqarrfIzPGGJPnfB0Qry7Oc5YjgMYigqp+67+wjDHGBEKOSUFEXgJa4ySFKUBH4E/AkoIxxhQwvnQ0d8d5EM5uVb0TuAgI92tUxhhjAsKXpJCsqm4gQ0SKAnuBGv4NyxhjTCD40qcQ6xkyeyiwGEgEFvo1KmOMMQGRbVIQEQHeUtXDOM9V+BUoqqrL8yQ6Y4wxeSrb5iPP85J/yjS9xRKCMcYUXL70KSwQkUv8HokxxpiA86VPoQ1wn4hsBY7h3MCmqtrIr5F4P35sAAAWjElEQVQZY4zJc74khY5+j8IYY8x5wZekoH6PwhhjzHnBl6TwM05iEJxhLqKBdcAFfozLGGNMAPgyIN6Fmac9w2jf57eIjDHGBIwvVx+dRFX/BuxqJGOMKYB8GRDvv5kmg4AmwD6/RWSMMSZgfOlTiMr0OgOnj+F7/4RjjDEmkHzpU3glLwIxxhgTeDn2KYjIdM+AeMenS4jIVP+GZYwxJhB86Wgu4xkQDwBVPQSU9V9IxhhjAsWXpOASkarHJ0SkGj7e0CYiHURknYjEicizZyhzs4isFpFVIjLSt7CNMcb4gy8dzc8Bf4rIbM/0FUDfnD4kIsHAIKA9EA8sEpGJqro6U5naQD+gpaoeEhE7AzHGmADypaP5V88Na81x7mp+XFX3+7DsZkCcqm4CEJHRQBdgdaYy9wKDPE1SqOres4zfGGNMLvKlo7kbkK6qk1V1Es5jObv6sOxKwPZM0/GeeZnVAeqIyFwRWSAiHc4QQ18RiRWR2H377BYJY4zxF1/6FF5S1YTjE55O55d8+JxkMe/UvogQoDbQGugJfJH5SqdM6/xcVWNUNaZMmTI+rNoYY8y58CUpZFXGl76IeKBKpunKwM4sykxQ1XRV3Ywz0F5tH5ZtjDHGD3xJCrEiMlBEaopIDRF5H1jsw+cWAbVFJFpEwoAewMRTyvyE8xAfRKQ0TnPSJt/DN8YYk5t8SQqPAGnAGGAckAI8mNOHVDUDeBiYCqwBxqrqKhF5VUSu9xSbChwQkdXATOApVT1w9tUwxhiTG0T17J6hIyIRwHWqOs4/IWUvJiZGY2NjA7FqY4zJt0RksarG5FTOp6GzRSRYRDqKyLfAFuCWfxifMcaY81C2HcYicgXQC+gMLARaAjVUNSkPYjPGGJPHzpgURCQe2AYMxmnrPyoimy0hGGNMwZVd89H3ODeb3QJcJyKF8XHMI2OMMfnTGZOCqj4GVAcG4lw2uh4o4xnArkjehGeMMSYvZdvRrI4ZqnovToLoBXTF6Ww2xhhTwPhyZzIAqpoOTAImiUik/0IyxhgTKD5dknoqVU3O7UCMMcYE3jklBWOMMQWTz0nBc/WRMcaYAsyX5ylc5hmbaI1n+iIR+dTvkRljjMlzvpwpvA9cAxwAUNVlOI/kNMYYU8D41HykqttPmeXyQyzGGGMCzJdLUreLyGWAep6L8CiepiRjjDEFiy9nCvcDD+EMeREPXOyZNsYYU8DkeKagqvuB3nkQizHGmADLMSmIyEdZzE4AYlV1Qu6HZIwxJlB8aT6KwGky2uD51wgoCdwtIh/4MTZjjDF5zJeO5lpAW88zlxGRwcA0oD2wwo+xGWOMyWO+nClUAjLfzVwYqKiqLiDVL1EZY4wJCF/OFPoDS0VkFiA4N6696Rn24jc/xmaMMSaP+XL10ZciMgVohpMU/qeqOz1vP+XP4IwxxuQtXwfESwF2AQeBWiJiw1wYY0wB5MslqfcAjwGVgaVAc2A+0Na/oRljjMlrvpwpPAZcAmxV1TZAY2CfX6MyxhgTEL4khRRVTQEQkXBVXQvU9W9YxhhjAsGXq4/iRaQ48BMwXUQOATtz+Iwxxph8yJerj7p5Xr4sIjOBYsCvfo3KGGNMQGSbFEQkCFiuqg0BVHV2nkRljDEmILLtU1BVN7BMRKrmUTzGGGMCyJeO5grAKhH5XUQmHv/ny8JFpIOIrBOROBF5Npty3UVERSTG18CNMcbkPl86ml85lwWLSDAwCGfgvHhgkYhMVNXVp5SLwnma21/nsh5jjDG5J8czBU8/whYg1PN6EfC3D8tuBsSp6iZVTQNGA12yKPcazvhKKb4GbYwxxj9yTAoici8wHvjMM6sSzuWpOakEbM80He+Zl3nZjYEqqjrZp2iNMcb4lS99Cg8BLYEjAKq6ASjrw+cki3nqfdO5sul94IkcFyTSV0RiRSR23z67mdoYY/zFl6SQ6mn+AUBEQsi0c89GPFAl03RlTr7pLQpoCMwSkS04YypNzKqzWVU/V9UYVY0pU6aMD6s2xhhzLnxJCrNF5H9ApIi0B8YBk3z43CKgtohEi0gY0APwXrWkqgmqWlpVq6tqdWABcL2qxp51LYwxxuQKX5LCszgD4K0A7gOmAM/n9CHP4zsfBqYCa4CxqrpKRF4VkevPPWRjjDH+IqrZtwSJSDdgiqqeF4/ejImJ0dhYO5kwxpizISKLVTXHe8F8OVO4HlgvIsNFpLOnT8EYY0wB5Mt9CncCtXD6EnoBG0XkC38HZowxJu/5dNSvquki8gvOVUeRODeh3ePPwIwxxuQ9X25e6yAiw4A4oDvwBc54SMYYYwoYX84U7sAZouK+86Wz2RhjjH/48pCdHpmnRaQl0EtVH/JbVMYYYwLCpz4FEbkYp5P5ZmAz8IM/gzLGGBMYZ0wKIlIH5y7knsABYAzOfQ1t8ig2Y4wxeSy7M4W1wB/AdaoaByAij+dJVMYYYwIiu6uPbgR2AzNFZKiIXEXWI58aY4wpIM6YFFT1R1W9BagHzAIeB8qJyGARuTqP4jPGGJOHfLmj+ZiqjlDVa3GGv16KM0ieMcaYAsaXsY+8VPWgqn6mqm39FZAxxpjAOaukYIwxpmCzpGCMMcbLkoIxxhgvSwrGGGO8LCkYY4zxsqRgjDHGy5KCMcYYL0sKxhhjvCwpGGOM8bKkYIwxxsuSgjHGGC9LCsYYY7wsKRhjjPGypGCMMcbLkoIxxhgvSwrGGGO8LCkYY4zxsqRgjDHGy69JQUQ6iMg6EYkTkdOe6ywi/xWR1SKyXER+F5Fq/ozHGGNM9vyWFEQkGBgEdAQaAD1FpMEpxZYAMaraCBgP9PdXPMYYY3LmzzOFZkCcqm5S1TRgNNAlcwFVnamqSZ7JBUBlP8ZjjDEmB/5MCpWA7Zmm4z3zzuRu4Jes3hCRviISKyKx+/bty8UQjTHGZObPpCBZzNMsC4r0AWKAAVm9r6qfq2qMqsaUKVMmF0M0xhiTWYgflx0PVMk0XRnYeWohEWkHPAdcqaqp57Ki9PR04uPjSUlJOadA84uIiAgqV65MaGhooEMxxhRQ/kwKi4DaIhIN7AB6AL0yFxCRxsBnQAdV3XuuK4qPjycqKorq1asjktUJSv6nqhw4cID4+Hiio6MDHY4xpoDyW/ORqmYADwNTgTXAWFVdJSKvisj1nmIDgCLAOBFZKiITz2VdKSkplCpVqsAmBAARoVSpUgX+bMgYE1j+PFNAVacAU06Z92Km1+1ya10FOSEc92+oozEmsOyO5lxw+PBhPv3007P+XKdOnTh8+LAfIjLGmHNjSSEXnCkpuFyubD83ZcoUihcv7q+wjDHmrPm1+ejf4tlnn2Xjxo1cfPHFhIaGUqRIESpUqMDSpUtZvXo1Xbt2Zfv27aSkpPDYY4/Rt29fAKpXr05sbCyJiYl07NiRVq1aMW/ePCpVqsSECROIjIwMcM2MMf82BS4pvDJpFat3HsnVZTaoWJSXrrvgjO+//fbbrFy5kqVLlzJr1iw6d+7MypUrvVcJffXVV5QsWZLk5GQuueQSbrzxRkqVKnXSMjZs2MCoUaMYOnQoN998M99//z19+vTJ1XoYY0xOClxSOB80a9bspMtGP/roI3788UcAtm/fzoYNG05LCtHR0Vx88cUANG3alC1btuRZvMYYc1yBSwrZHdHnlcKFC3tfz5o1i99++4358+dTqFAhWrduneVlpeHh4d7XwcHBJCcn50msxhiTmXU054KoqCiOHj2a5XsJCQmUKFGCQoUKsXbtWhYsWJDH0RljjO8K3JlCIJQqVYqWLVvSsGFDIiMjKVeunPe9Dh06MGTIEBo1akTdunVp3rx5ACM1xpjsiWqWY9Sdt2JiYjQ2NvakeWvWrKF+/foBiihv/ZvqaozJPSKyWFVjcipnzUfGGGO8LCkYY4zxsqRgjDHGy5KCMcYYL0sKxhhjvCwpGGOM8bKkkAvOdehsgA8++ICkpKRcjsgYY86NJYVcYEnBGFNQ2B3NuSDz0Nnt27enbNmyjB07ltTUVLp168Yrr7zCsWPHuPnmm4mPj8flcvHCCy+wZ88edu7cSZs2bShdujQzZ84MdFWMMf9yBS8p/PIs7F6Ru8ssfyF0fPuMb2ceOnvatGmMHz+ehQsXoqpcf/31zJkzh3379lGxYkV+/vlnwBkTqVixYgwcOJCZM2dSunTp3I3ZGGPOgTUf5bJp06Yxbdo0GjduTJMmTVi7di0bNmzgwgsv5LfffuOZZ57hjz/+oFixYoEO1RhjTlPwzhSyOaLPC6pKv379uO+++057b/HixUyZMoV+/fpx9dVX8+KLLwYgQmOMOTM7U8gFmYfOvuaaa/jqq69ITEwEYMeOHezdu5edO3dSqFAh+vTpw5NPPsnff/992meNMSbQCt6ZQgBkHjq7Y8eO9OrVixYtWgBQpEgRvvvuO+Li4njqqacICgoiNDSUwYMHA9C3b186duxIhQoVrKPZGBNwNnR2PvNvqqsxJvfY0NnGGGPOmiUFY4wxXpYUjDHGeBWYpJDf+kbOxb+hjsaYwCoQSSEiIoIDBw4U6J2mqnLgwAEiIiICHYoxpgArEJekVq5cmfj4ePbt2xfoUPwqIiKCypUrBzoMY0wB5tekICIdgA+BYOALVX37lPfDgW+BpsAB4BZV3XK26wkNDSU6OvqfB2yMMf9yfms+EpFgYBDQEWgA9BSRBqcUuxs4pKq1gPeBd/wVjzHGmJz5s0+hGRCnqptUNQ0YDXQ5pUwX4BvP6/HAVSIifozJGGNMNvyZFCoB2zNNx3vmZVlGVTOABKCUH2MyxhiTDX/2KWR1xH/q5UG+lEFE+gJ9PZOJIrLOh/WXBvb7UC4/sLqcvwpSfawu56fcqks1Xwr5MynEA1UyTVcGdp6hTLyIhADFgIOnLkhVPwc+P5uVi0isL+N85AdWl/NXQaqP1eX8lNd18Wfz0SKgtohEi0gY0AOYeEqZicDtntfdgRlakG82MMaY85zfzhRUNUNEHgam4lyS+pWqrhKRV4FYVZ0IfAkMF5E4nDOEHv6KxxhjTM78ep+Cqk4Bppwy78VMr1OAm/y0+rNqbjrPWV3OXwWpPlaX81Oe1iXfPU/BGGOM/xSIsY+MMcbkjgKZFESkg4isE5E4EXk20PGcDRGpIiIzRWSNiKwSkcc880uKyHQR2eD5v0SgY/WViASLyBIRmeyZjhaRvzx1GeO5EOG8JyLFRWS8iKz1bJ8W+XW7iMjjnt/XShEZJSIR+Wm7iMhXIrJXRFZmmpflthDHR579wXIRaRK4yE93hroM8PzOlovIjyJSPNN7/Tx1WSci1+R2PAUuKfg4vMb5LAN4QlXrA82BhzzxPwv8rqq1gd890/nFY8CaTNPvAO976nIIZ7iT/OBD4FdVrQdchFOnfLddRKQS8CgQo6oNcS4E6UH+2i7DgA6nzDvTtugI1Pb86wsMzqMYfTWM0+syHWioqo2A9UA/AM++oAdwgeczn3r2ebmmwCUFfBte47ylqrtU9W/P66M4O55KnDwkyDdA18BEeHZEpDLQGfjCMy1AW5xhTSCf1EVEigJX4Fwxh6qmqeph8ul2wbnIJNJzf1AhYBf5aLuo6hxOv6fpTNuiC/CtOhYAxUWkQt5EmrOs6qKq0zyjPAAswLnPC5y6jFbVVFXdDMTh7PNyTUFMCr4Mr5EviEh1oDHwF1BOVXeBkziAsoGL7Kx8ADwNuD3TpYDDmX7w+WX71AD2AV97msK+EJHC5MPtoqo7gHeBbTjJIAFYTP7cLpmdaVvk933CXcAvntd+r0tBTAo+DZ1xvhORIsD3wH9U9Uig4zkXInItsFdVF2eenUXR/LB9QoAmwGBVbQwcIx80FWXF09beBYgGKgKFcZpYTpUftosv8utvDhF5DqdJecTxWVkUy9W6FMSk4MvwGuc1EQnFSQgjVPUHz+w9x095Pf/vDVR8Z6ElcL2IbMFpxmuLc+ZQ3NNsAfln+8QD8ar6l2d6PE6SyI/bpR2wWVX3qWo68ANwGflzu2R2pm2RL/cJInI7cC3QO9NID36vS0FMCr4Mr3He8rS5fwmsUdWBmd7KPCTI7cCEvI7tbKlqP1WtrKrVcbbDDFXtDczEGdYE8k9ddgPbRaSuZ9ZVwGry4XbBaTZqLiKFPL+343XJd9vlFGfaFhOB2zxXITUHEo43M52vxHlA2TPA9aqalOmtiUAPEQkXkWiczvOFubpyVS1w/4BOOD32G4HnAh3PWcbeCud0cDmw1POvE05b/O/ABs//JQMd61nWqzUw2fO6hueHHAeMA8IDHZ+PdbgYiPVsm5+AEvl1uwCvAGuBlcBwIDw/bRdgFE5/SDrO0fPdZ9oWOE0ugzz7gxU4V10FvA451CUOp+/g+D5gSKbyz3nqsg7omNvx2B3NxhhjvApi85ExxphzZEnBGGOMlyUFY4wxXpYUjDHGeFlSMMYY42VJwZzXRERF5L1M00+KyMu5tOxhItI955L/eD03eUZVnXnK/OqZR8Y05nxgScGc71KBG0SkdKADyewsR6a8G3hQVdv4K56cZLpT2ZhsWVIw57sMnMcRPn7qG6ce6YtIouf/1iIyW0TGish6EXlbRHqLyEIRWSEiNTMtpp2I/OEpd63n88Ge8ewXecazvy/TcmeKyEicm6BOjaenZ/krReQdz7wXcW5IHCIiA3ypsIjc61n3MhH53nPncZSIbPYMgYKIFBWRLSISKiI1ReRXEVnsqUu9TN/PQM8ZyjsicqWILPX8WyIiUb7EY/5d7OjB5AeDgOUi0v8sPnMRUB9nSOJNwBeq2kychxY9AvzHU646cCVQE5gpIrWA23CGQrhERMKBuSIyzVO+Gc4495szr0xEKuI8j6ApzrMIpolIV1V9VUTaAk+qaqyPsf+gqkM9y30duFtVPxaRWTjDkP+EM2zI96qaLiKfA/er6gYRuRT4FGecKYA6QDtVdYnIJOAhVZ3rGXAxxcd4zL+InSmY8546o8R+i/NgGF8tUufZFKk4QwIc36mvwEkEx41VVbeqbsBJHvWAq3HGylmKM2x5KZwxZgAWnpoQPC4BZqkzyNzxUS2vOIt4M2voOeJfAfTGeaAKOM+kuNPz+k6cYbyL4AxmN84T72dA5mcFjFNVl+f1XGCgiDwKFNcTw2Qb42VnCia/+AD4G/g607wMPAc2noHdMj8+MjXTa3emaTcn/+5PHedFccbKeURVp2Z+Q0Ra4wyZnZWshjQ+V8OArqq6TETuwBk3Cs8RfnURuRIIVtWV4jz857CqXnyGZXnjVdW3ReRnnLG0FohIO1Vdm4txmwLAzhRMvqCqB4GxnPyIyC04zTXgPB8g9BwWfZOIBHn6GWrgDDI2FXggU/t9HXEeqJOdv4ArRaS0pxO6JzD7HOIBiAJ2edbf+5T3vsUZQO1r8J5FbRaRmzyxiohclNVCRaSmqq5Q1XdwBvard47xmQLMkoLJT94DMl+FNBRnR7wQuJQzH8VnZx3OzvsXnHb5FJxmmtXA355LRj8jh7NqdYZi7ocz/PQy4G9V9WXo6boiEp/p303ACzhJZjrOSKaZjcAZnXVUpnm9gbtFZBmwijM/fvY/nk7wZUAyJ57mZYyXjZJqTD7iudqqi6reGuhYTMFkfQrG5BMi8jHOYzM7BToWU3DZmYIxxhgv61MwxhjjZUnBGGOMlyUFY4wxXpYUjDHGeFlSMMYY42VJwRhjjNf/AcFgO1TJXLTrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "layers = np.arange(2,max_layers+2,1)\n",
    "ax.plot(layers, train_results_per_layer, label='train')\n",
    "ax.plot(layers, test_results_per_layer/test_set_size, label='test')\n",
    "ax.set_ylim(0,1.1)\n",
    "ax.set_xlabel('Number of Layers')\n",
    "ax.set_ylabel('Average Accuracy')\n",
    "ax.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning rate 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### h = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of layers: 2\n",
      "Max training accuracy: 0.9662499904632569\n",
      "Max test accuracy: 0.9600000000000001\n",
      "Time elapsed: 12.112743869000042\n",
      "Number of layers: 3\n",
      "Max training accuracy: 0.9985416471958161\n",
      "Max test accuracy: 0.9975\n",
      "Time elapsed: 16.315213189000133\n",
      "Number of layers: 4\n",
      "Max training accuracy: 0.9985416531562805\n",
      "Max test accuracy: 0.9975\n",
      "Time elapsed: 20.84354746200006\n",
      "Number of layers: 5\n",
      "Max training accuracy: 0.9981249868869781\n",
      "Max test accuracy: 0.9975\n",
      "Time elapsed: 24.1426120849992\n",
      "Number of layers: 6\n",
      "Max training accuracy: 0.9983333230018616\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 30.043436734000352\n",
      "Number of layers: 7\n",
      "Max training accuracy: 0.9991666555404664\n",
      "Max test accuracy: 0.9958333333333333\n",
      "Time elapsed: 33.38494030799848\n",
      "Number of layers: 8\n",
      "Max training accuracy: 0.9891666531562805\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 38.8770654789987\n",
      "Number of layers: 9\n",
      "Max training accuracy: 0.9931249916553497\n",
      "Max test accuracy: 0.9933333333333334\n",
      "Time elapsed: 44.380669526999554\n",
      "Number of layers: 10\n",
      "Max training accuracy: 0.9879166483879089\n",
      "Max test accuracy: 0.9858333333333333\n",
      "Time elapsed: 45.3374343029991\n",
      "Number of layers: 11\n",
      "Max training accuracy: 0.9568749904632569\n",
      "Max test accuracy: 0.9333333333333333\n",
      "Time elapsed: 52.599229841000124\n",
      "Number of layers: 12\n",
      "Max training accuracy: 0.989166659116745\n",
      "Max test accuracy: 0.985\n",
      "Time elapsed: 55.265721899999335\n",
      "Number of layers: 13\n",
      "Max training accuracy: 0.976666659116745\n",
      "Max test accuracy: 0.9725\n",
      "Time elapsed: 57.78252588800024\n",
      "Number of layers: 14\n",
      "Max training accuracy: 0.9639583230018616\n",
      "Max test accuracy: 0.9575\n",
      "Time elapsed: 64.43589614399934\n",
      "Number of layers: 15\n",
      "Max training accuracy: 0.956458330154419\n",
      "Max test accuracy: 0.9516666666666667\n",
      "Time elapsed: 64.75633420200029\n",
      "Number of layers: 16\n",
      "Max training accuracy: 0.9349999964237213\n",
      "Max test accuracy: 0.8983333333333333\n",
      "Time elapsed: 69.47652047500014\n",
      "Number of layers: 17\n",
      "Max training accuracy: 0.9535416603088379\n",
      "Max test accuracy: 0.9491666666666667\n",
      "Time elapsed: 71.95118058200023\n",
      "Number of layers: 18\n",
      "Max training accuracy: 0.9464583277702332\n",
      "Max test accuracy: 0.9291666666666667\n",
      "Time elapsed: 76.8870962609999\n",
      "Number of layers: 19\n",
      "Max training accuracy: 0.9185416579246521\n",
      "Max test accuracy: 0.9091666666666666\n",
      "Time elapsed: 85.31754751900007\n",
      "Number of layers: 20\n",
      "Max training accuracy: 0.8566666543483734\n",
      "Max test accuracy: 0.85\n",
      "Time elapsed: 82.93505771899981\n",
      "Number of layers: 21\n",
      "Max training accuracy: 0.7189583241939544\n",
      "Max test accuracy: 0.7033333333333334\n",
      "Time elapsed: 91.47647318300005\n"
     ]
    }
   ],
   "source": [
    "from Networks.ResNet import ResAntiSymNet\n",
    "import numpy as np\n",
    "import timeit\n",
    "\n",
    "num_epochs = 100\n",
    "num_features = 2\n",
    "num_classes = 2\n",
    "size_hidden = 8\n",
    "gamma = 0.3\n",
    "h = 1\n",
    "lr = 0.5\n",
    "\n",
    "max_layers = 20\n",
    "train_results_per_layer = np.zeros(max_layers)\n",
    "test_results_per_layer = np.zeros(max_layers)\n",
    "basic_save_key = 'AntiSym_Comparison_results//AntiSymNet'\n",
    "\n",
    "for num_hidden_layers in range(1,max_layers+1):\n",
    "    tic = timeit.default_timer()\n",
    "    num_layers = num_hidden_layers + 1\n",
    "    print('Number of layers: '+str(num_layers))\n",
    "    train_results_per_epoch = np.zeros(num_epochs)\n",
    "    test_results_per_epoch = np.zeros(num_epochs)\n",
    "    for i in range(10):\n",
    "        net = ResAntiSymNet(features=num_features, classes=num_classes, num_layers=num_layers, gamma=gamma, h=h, bias=True, hidden_size=size_hidden)\n",
    "        net.set_test_tracking(True)\n",
    "        net.train(num_epochs, train_loader, test_loader, print_output=False)\n",
    "        train_results_per_epoch += net.avg_correct_pred.numpy()\n",
    "        test_results_per_epoch += net.test_results.numpy()\n",
    "        del net\n",
    "        \n",
    "    train_results_per_epoch = train_results_per_epoch / 10\n",
    "    test_results_per_epoch = test_results_per_epoch / 10\n",
    "    train_save_key = basic_save_key+'_train_per_epoch_layers_'+str(num_layers)+'_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "    test_save_key = basic_save_key+'_test_per_epoch_layers_'+str(num_layers)+'_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "    \n",
    "    np.savetxt(train_save_key, train_results_per_epoch, delimiter=',')\n",
    "    np.savetxt(test_save_key, test_results_per_epoch, delimiter=',')\n",
    "    \n",
    "    train_results_per_layer[num_hidden_layers-1] = train_results_per_epoch.max()\n",
    "    print('Max training accuracy: '+str(train_results_per_epoch.max()))\n",
    "    test_results_per_layer[num_hidden_layers-1] = test_results_per_epoch.max()\n",
    "    print('Max test accuracy: '+str(test_results_per_epoch.max()/test_set_size))\n",
    "    \n",
    "    toc = timeit.default_timer()\n",
    "    print('Time elapsed: '+str(toc-tic))\n",
    "\n",
    "train_save_key = basic_save_key+'_train_per_layer_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "test_save_key = basic_save_key+'_test_per_layer_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "np.savetxt(train_save_key, train_results_per_layer, delimiter=',')\n",
    "np.savetxt(test_save_key, test_results_per_layer, delimiter=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x2050ad7f940>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VPW9//HXJyF7AiEL+xI2qTtgRCu4r2gFe7XW7ddd7bW2apdb7WLV9t6qvS7VWltrvV21aq0KihWtuOECgbILsggS1rAFQkjI8vn9cU7GMWYZIDMTwvv5cB4z55zvOeeTcZjPfM/3fL9fc3dEREQAUpIdgIiIdB5KCiIiEqGkICIiEUoKIiISoaQgIiIRSgoiIhKhpCAiIhFKCiIiEqGkICIiEd2SHcDeKioq8pKSkmSHISJyQJk9e/Zmdy9ur9wBlxRKSkooKytLdhgiIgcUM1sdSzldPhIRkQglBRERiVBSEBGRCCUFERGJUFIQEZEIJQUREYlQUhARkQglBRERiVBSEBGRCCUFERGJUFIQEZEIJQUREYlQUhARkQglBRERiYhbUjCzR8xsk5ktbGW7mdl9ZrbczOab2Zh4xSIiIrGJZ03hD8A5bWyfAIwIH1cBD8YxFhERiUHckoK7vw5sbaPIJOBPHngHyDezvvGKR0RE2pfMmdf6A2uilsvDdeuTE07bNu2oZuHixbg37tdxDMMMDILnptdN6y36tYXlwmV38AbMHfcGrLERaIDGBvDG4NHYgNEI3gCNjZg3hvs04A71lk59SvCoa3pt6dRZBnWWRj1pNFg3GtxpbHQa3WlodNyD2PrlZ9E/P4sBPbMoyEnHzDrg3RWRziKZSaGlbxNvsaDZVQSXmBg0aFA8Y/qYxkZnxtJ1rJz+Rz694a+cllKesHMnU52nsodu1JJGLenUehq1pLGbDCo8n/nek5e8J9tSC2nM7U1K9/5kFfansKgP/QuyGdAziwH5WRTlZpCSsm9Jo7a+gepd1VRXVrCnagt7dm6mobaK9NxeZBX0JaegL3m5uaTu4/FFpGXJTArlwMCo5QHAupYKuvtDwEMApaWlLSaOjrRpRw3PvLuU2ncf4cK6yZxoW9mUM4y1R99CalaP/T5+I074H+7Bn+NNy+GCh+vgo3IOYKnhIwVSUnBLDX6tp6TiTestFbcUzILtpKRglgpAqteR2riH1MZaUhv3kNJYS2rDHlIaaklprA2eG2pJadhDdkMtOQ21WLjOa3fSsGMjKVUrSN+zPQiuOnxsgFpPY6Pns5GezPSebLYCajN70Zjbh275/cjIK8L27MRqtpFas41utZWk120ns24HWfWV5DTuIKdxJ929inyq6Gm19Gzjfaz0bLbQk+0p+ezsVsCutEJqMgqpyyqmIbsX5PYiJa8XGT16k5edRVFeBkf177HPiWp/rKio4tWlFfTunsGw4lyGFOWQmZaa8DhE2pPMpDAZuNbM/gYcB1S6e9IuHTU0Om8sq+C5t+YydOVfuTxlGj2sms3FY6k74zv0Gnl2cP3kIJfW9KKuBqo2ws714WMDtm0t+VvLyatczyG7NpC5ewEZe6qDlqVWWpcaSKEqpTvVqd2pyehObdoAtqTnsykzn4aMfMguwLIKSM3piWXk0FC1lYYdG7CqTaTsriB9dwX5tZvpV7eS7rtnk11dDds+fo5GN7aRS7kX818Zl9F/7CQuOmYAAwuy4/hOQU1dA/9cuIHHZn7IzlVzuCL1ZVbSnZcb+7GSfuzuPpS+vYoZVpzD0OJchhXnMKw4l155GbosJ0ljTb9UO/zAZo8BpwBFwEbgJ4TfKe7+Gws+9b8iuEOpGviyu5e1d9zS0lIvK2u3WMw27qjhiVlrePPdd5lY/Q8u6vY66dRTPexcck79Ngwo7bBzHZRqd8LODbBjHY3VW0nJ7A7ZBZDVE7IKICOvY5PtnmrYtQmqNtGwYyM129dTt3099Ts3kVE+g7yqD3i6YTy31V3BYcOHcHHpQM4+vE+H/mpfvmknj81cw1NzyqmpruLm3MlcUj8Z0jKw+lrMGyJlt1gByxr7sqyhLyu9Lyu8HxvSBpJVNJihvboztCiHYb1yGVqcQ0mhahey78xstru3+4UWt6QQLx2RFBoandffr+DRmR9SsfRtrkyZzITUWXhKGhx9Kanjr4PCYR0UsXQa9bXwxl34G3exOzWPO+wr/HHHGPIy05g0qh8Xlw7kyP499ulXek1dAy8sXM9j765h5qqtpKUa3yxZy5WV95FV9SGM+QKceRuk5cC2D2Dz+7B5GWxehm9Zhle8T0ptZeR4tZbBh/RlaX0fVnhfVoS1i5oewxnYq4BhxbkM65XLsOJchvfKpSAnvSPfKemClBRasL5yN0/MKufxmasZUTWTazOe51hfSGN6d1LGfg2O+zrk9e7giKXT2bgInr0W1s1h64AzuC/z6zy2pJ7a+kY+1SePz5UO5IJR/SjMzWj3UO9v3MljMz/kH3PWUrm7jpLCbL44ujuXbvstmYseh4JhcP4vYciJbR/IHXZtDpLFlmWRhNG4+X1s++rgLjKCy23lKQNYUD+ABQ2Dec8HsbixhIbsoiBRFOcyrFdO5PXAgmw1xgugpPAJD7+xkjunLmCCvcN3cv7JoLqVeF4/7NPXwJgvQmb3OEQrnVZjA7zzILzyM0hNo/qUn/C0nc4Ts9cxb8120lKNMw7tzcWlAzlxRBHdUj/q0lNT18Dz89fz2MwPKVu9jbRU4+zD+3DZsQM5fverpPzzRqjZDuOug5P+C9Iy9y/W+lrYuhIqlgQJbcNCfMMCbMdHd8Pt7FbAypQS5tYNZHbtABb7YD7wvqSmpjGkKIdhvXIYWpRL7+4ZFOcFj6Lc4Dk7PZlNi5IoSgrNrHz9UQrfvI0ee9ZD0cjgH+yRn4NuqnYf1LauhCnXwQevQ8mJcP4vWVrXiyfL1vCPf69l66499O6ewYVjBjB+RBHTFm3kH3PK2VFTz9CiHC4dO4j/GNOfwvqN8Ny3YflL0P8YOP8+6HNEfGOv3homiQWwcWHwXLEEGvYAUJ+SQUXWEJbbEObuGcBbu/qwsbEH2z2XSnJoIGifyElPpSgvg+LcTyaM4tyMYFteBkW56WR0U5vGgUpJobnFk+HtB2D89TDibEjRWIAScod//xle/BE01MKpP4Djv8EeT+GVJZt4smwN05duotEhPTWFCUf24dKxgzhuSEFwWWfmQ/CvnwbHOv1mGHslpCTpy7OhLrgEtWEhbJj/UbKo3vKJonu65VHTrTs7U7qzg1y2eQ6b6rPZWJfFhrpstnsu28mNPG+hOyX9+zFueBHjhxdxzOCeCWn43lPfyKJ1lSxat4OstCCBFeakU5yXQUFOOmmp+rccCyWF5pq65Iq0Zsd6eP47sPR56DsKJv0K+hwJBHepla3axqeHFX7UqLthIUz5FqydDSPOgvPugvzEda6MmXtwB1jFe7BrC+zeFj62Bs/VW5ut204r/UiZlnUu36u8iMrGTDK6pXBsSQEnDC9k/PAiDu/Xo0PaL7bt2sPs1dsoW72NOau3Ma98O93rt3Bp6nTWehHv+qGUe3GkfH52GoU56RTmBjWbwtx0isLnwpwMivOanjPIyTh4L5UpKYjsC3dY/AxM/V7wJTn+Bjjpe9AtqtG5bje8die8dR9k5sOEO+CIC7vOj47GxqBNZPe2IEE0JY81M2HWwzR278/80bcxuepQ3lqxmSUbdgLQIyuNE4YVMm54EeOGF1FSmN3unVzuzoqKXcxevTWSCFZW7AIgLdU4om8eV2a/xpnrHiStviqy3+7sfqzPH8OK7NEsTD+C5XuKqdi1hy1VtWyu2kPl7roWzzd6UD4Tj+7HeUf2pVf3/WzrOcAoKYjsj+qt8OIPYN5jQRvUxPth0HFB28OU64K2iFGXw1k/C/pdHCzWzIRnvxFcohp9BZz132yqz+TtFVt4c9lmZizfzLrKGgD652cxbniQJE4YVkRxXgY1dQ3MW7Od2R9uY/aqbcz+cBvbq4Mv8PzsNI4Z1JNjSnpSOriAo9PXkvHCDVA+C4acBOfdHbSXrJoBq98Mnqs3B3Hl9YOScTB4HJScyJ4eQ9i2u46KnbVs2bWHzTtrWbOtmhcXbeS99Tswg+OHFHL+0f2YcEQfeh4Et/QqKYh0hGUvw3PXQ2V58IWz+k3oOQTOvxeGnpLs6JKjrgZeux1m3Ae5veAz98LIYJR8d2fVlmreXL6ZGcs28/bKLZFf7YMKsllfuZu6huA7Z2hxDqWDgwQwZnBPhhXnBDWLPdXw+p3w1v2Q2QPO/h846vOfrIm5Q8XSjxLE6hlBL3uA3N5hghgHg8dD8cjI/ss37WTKvPVMmbeOlZt30S3FOHFEEecf3Y8zD+tNXmYaXZGSgkhHqd0ZNCTP/gMc/59w8vchPb5DZBwQ1s4J+ntsWhR8aZ9z+ydqTQ2NzqJ1lby5fDPz11QypDiHYwb1ZMzgni13uFv+L3j+27BtVVATO/OnkFMYWzzusGXFR0li1ZuwMxxOLbvoowRRMg6KD8XNWLRuB1Pmr+O5eetZu3036d1SOG1kLyaO6sdpn+rVpXqQKymIdLTGhuTdVdRZ1e+BN+6CN/43GLbkvLvgsIl7f5yqTcHlugVPQuHwoPbRXoe/9rgHvcebahGr3oTKcLT+rAIYfAKUjIfB42jsdTj/Lt/BlHnreG7+ejZX1ZKTnsqZh/Xm/KP7ceKIYtK7Hdh3OSkpiEjibFgAz1wT3AZ72AVw7v9CbnH7+zU2BrcDv3Qz1FXD+G8Hjfv72+GvNdtWhwkibJfYtipYn9kDBp0AJeNoGDSOd6v7MWXhJqYu2EDl7jp6ZKUx4Yg+fPP0EfTPz4pPbHGmpCAiidVQF9yR9ertkJ4L5/6i7buyNi0J2ms+fDu4/v+Ze6H4kMTGXLn2o1rE6hmwZXmwPqM7DDqe+oEnMDf1cB5bU8DURZvJSEvhns+P4tSRvRIbZwdQUhCR5Ni0JLhDaW0ZjDwPPnM35PX5aHtdTXC56c17IT0nuINr1OWdo0Ppzg0fJYhVM2Dz0mB9Wg7Vfcfyw63n8vTm/lx76nBuOPOQA2pcKSUFEUmexgZ459fB2FLdMuDsn8Ooy4Jbep+7AbauCBqnz/rv2C4zJUvVpo8SxJLn8aqNvFj8Fa758GSOG1rMLy8dRa+8A6O/g5KCiCTf5uUw+drgElHxp4KxmXoOCWoPw05LdnR7p6YySGgLn2Jj4XFctPGL1GT15leXjua4oTHeIZVEsSaFTlBfE5Euq2g4fGkqTLgz6B194nfgmrcPvIQAQWP0hb+Hib+i944FTM/9EaenzuWyh9/lN6+toLHxwPqB3RrVFERE9lbFUvj7V2DjQv6VfxH/ueF8Tjq0P3d9bhQ9sjtn5zfVFERE4qV4JHztXzD2Kk7f/nfeKr6dVe/P57z732BBeWX7+3diSgoiIvsiLTO47faSRymq38CLWT/mrLrpXPjgW/z5ndUcaFdhmigpiIjsj0+dB1+fQWr/Udxcfx//1+Nhbn9mFtc/PpddtfXJjm6vKSmIiOyvHv3hi1PglB9wwu5XmJF/Cx/Mf5NJD8xg2cadyY5urygpiIh0hJRUOOX72JeeJz+9kWcyb+W8qqeY9Ks3eObfa5MdXcyUFEREOtLgE+Drb5JyyNnc0PhH/pJ9Nz99/DV+8PQCausbkh1du5QUREQ6WnYBfP4vcO7/Mrp+Pq/l/YhVs6Zyz0vLkh1Zu5QURETiwQzGXold+Qq5PYr4S/rP+XDG46ysqGp/3yRSUhARiac+R8BV02koPpSbUv/C/0yZ26lvV1VSEBGJt/Qc0s75bwbaJgaveJRXlmxKdkStUlIQEUmEYafROPR0rkt7lnumvEtNXedsdFZSEBFJkJSzf0qeVTNpx2P8/s0Pkh1Oi5QUREQSpffh2KjL+FK3l3jmlRmsr9yd7Ig+QUlBRCSRTv0hqd26cb09xv9MXZLsaD4hrknBzM4xs6VmttzMbmxh+yAzm25m/zaz+WZ2bjzjERFJuu79SDnhm5yX8jZr5r/Ouyu3JDuij4lbUjCzVOABYAJwGHCpmR3WrNiPgCfcfTRwCfDreMUjItJpjPsWntOLWzMf4yfPLqS+oTHZEUXEs6YwFlju7ivdfQ/wN2BSszIOdA9f9wDWxTEeEZHOISMPO/Umjvb3GFQxnUdnfpjsiCLimRT6A2uilsvDddFuAa4ws3JgKvDNOMYjItJ5jP4CXjSSW7Oe4JcvLmbrrj3JjgiIb1KwFtY178Z3KfAHdx8AnAv82cw+EZOZXWVmZWZWVlFREYdQRUQSLLUbduZt9G1Yy2fqp/G/05YmOyIgvkmhHBgYtTyAT14e+irwBIC7vw1kAkXND+TuD7l7qbuXFhcXxylcEZEEO+RsKDmR/8p8mudmLmHh2uRP5RnPpDALGGFmQ8wsnaAheXKzMh8CpwOY2aEESUFVARE5OJjBWT8lp34712c+zy2TFyV9XKS4JQV3rweuBV4E3iO4y2iRmd1mZhPDYt8BrjSzecBjwJc82e+IiEgi9RsNR17MF2wqa1cv55m5yZ2Qxw607+DS0lIvKytLdhgiIh1n+4f4/aVM7zaeGxuv4ZXvnkJuRrcOPYWZzXb30vbKtVtTMLMjOiYkERFpUf4g7Pivc2rtKxRXLeX+V5I3GU8sl49+Y2YzzewaM8uPe0QiIgej8d/GsnpyT8FTPPLmSlYkaTKedpOCu48HLie4k6jMzB41szPjHpmIyMEkKx9O/j6H7JrNGd0WcNuUxUlpdI6podndlxEMSfF94GTgPjNbYmb/Ec/gREQOKqVfgYKh/HfuE7z5/gb+9V7iJ+OJpU3hKDO7h+AOotOA89390PD1PXGOT0Tk4NEtHc64hYJdK7gm/11ue25xwifjiaWm8CtgDnC0u3/D3ecAuPs6gtqDiIh0lEMnwsDj+CaPs3nr1oRPxhNLUjgXeNTddwOYWYqZZQO4+5/jGZyIyEHHDM76Gek1FdzR9zV+9cpy1m1P3GQ8sSSFl4GsqOXscJ2IiMTDwLFw2CTOq3qSQt/Kz19I3GQ8sSSFTHeP3BsVvs6OX0giIsLpPyGloY4H+7/IlHnreCdBk/HEkhR2mdmYpgUzOwbofBOLioh0JYXD4NivccSmyYzvXsEtkxclZDKeWJLC9cCTZvaGmb0BPE4wppGIiMTTyf+Fpedxd8E/WLJhZ0Im42l3cA13n2VmnwJGEsyRsMTd6+IemYjIwS67AE76Dr1euplfHncJxx3WJ+6njHWU1JEE8yyPJphr+QvxC0lERCLGXg09BjFp44P0yUuP++li6bz2E+D+8HEqcCcwsc2dRESkY6Rlwuk3w4b5sOCJuJ8ulrFZLwKOBv7t7l82s97Aw/ENS0REIo64ENbNgb5Hx/1UsSSF3e7eaGb1ZtYd2AQMjXNcIiLSJCUFzvl5Qk4VS1IoC4fM/h0wG6gCZsY1KhERSYo2k4KZGfBzd99OMK/CP4Hu7j4/IdGJiEhCtdnQHM6X/EzU8iolBBGRriuWW1LfMbNj4x6JiIgkXSxtCqcCV5vZamAXQQc2d/ej4hqZiIgkXCxJYULcoxARkU4hlqSQ+ElCRUQkKWJJCs8TJAYDMoEhwFLg8DjGJSIiSRDLgHhHRi+Hw2hfHbeIREQkaWIdEC8inKNZdyOJiHRB7dYUzOzbUYspwBigIm4RiYhI0sTSppAX9bqeoI3hqfiEIyIiyRRLm8KtiQhERESSL5b5FF4KB8RrWu5pZi/GNywREUmGWBqai8MB8QBw921Ar/iFJCIiyRJLUmgws0FNC2Y2mBg7tJnZOWa21MyWm9mNrZS52MwWm9kiM3s0trBFRCQeYmlo/iHwppm9Fi6fBFzV3k5mlgo8AJwJlAOzzGyyuy+OKjMCuAkY5+7bzEw1EBGRJIqlofmfYYe14wl6Nd/g7ptjOPZYYLm7rwQws78Bk4DFUWWuBB4IL0nh7pv2Mn4REelAsTQ0fxaoc/fn3H0KUG9mF8Rw7P7Amqjl8nBdtEOAQ8xshpm9Y2bntBLDVWZWZmZlFRXqIiEiEi+xtCn8xN0rmxbCRuefxLCftbCueVtEN2AEcApwKfBw9J1OUed8yN1L3b20uLg4hlOLiMi+iCUptFQmlraIcmBg1PIAYF0LZZ519zp3/4BgoL0RMRxbRETiIJakUGZmd5vZMDMbamb3ALNj2G8WMMLMhphZOnAJMLlZmWcIJvHBzIoILietjD18ERHpSLEkhW8Ce4DHgSeBGuCa9nZy93rgWuBF4D3gCXdfZGa3mdnEsNiLwBYzWwxMB77n7lv2/s8QEZGOYO57N4eOmWUC57v7k/EJqW2lpaVeVlaWjFOLiBywzGy2u5e2Vy6mobPNLNXMJpjZn4BVwOf3Mz4REemE2mwwNrOTgMuA84CZwDhgqLtXJyA2ERFJsFaTgpmVAx8CDxJc699pZh8oIYiIdF1tXT56iqCz2eeB880shxjHPBIRkQNTq0nB3a8DSoC7CW4bfR8oDgewy01MeCIikkhtNjR74BV3v5IgQVwGXEDQ2CwiIl1MLD2TAXD3OmAKMMXMsuIXkoiIJEtMt6Q25+67OzoQERFJvn1KCiIi0jXFnBTCu49ERKQLi2U+hRPCsYneC5ePNrNfxz0yERFJuFhqCvcAZwNbANx9HsGUnCIi0sXEdPnI3dc0W9UQh1hERCTJYrkldY2ZnQB4OC/CtwgvJYmISNcSS03h68A3CIa8KAdGhcsiItLFtFtTcPfNwOUJiEVERJKs3aRgZve1sLoSKHP3Zzs+JBERSZZYLh9lElwyWhY+jgIKgK+a2b1xjE1ERBIslobm4cBp4ZzLmNmDwDTgTGBBHGMTEZEEi6Wm0B+I7s2cA/Rz9wagNi5RiYhIUsRSU7gTmGtmrwJG0HHtf8JhL16OY2wiIpJgsdx99HszmwqMJUgKP3D3deHm78UzOBERSaxYB8SrAdYDW4HhZqZhLkREuqBYbkn9GnAdMACYCxwPvA2cFt/QREQk0WKpKVwHHAusdvdTgdFARVyjEhGRpIglKdS4ew2AmWW4+xJgZHzDEhGRZIjl7qNyM8sHngFeMrNtwLp29hERkQNQLHcffTZ8eYuZTQd6AP+Ma1QiIpIUbSYFM0sB5rv7EQDu/lpCohIRkaRos03B3RuBeWY2KEHxiIhIEsXS0NwXWGRm/zKzyU2PWA5uZueY2VIzW25mN7ZR7iIzczMrjTVwERHpeLE0NN+6Lwc2s1TgAYKB88qBWWY22d0XNyuXRzCb27v7ch4REek47dYUwnaEVUBa+HoWMCeGY48Flrv7SnffA/wNmNRCuZ8SjK9UE2vQIiISH+0mBTO7Evg78NtwVX+C21Pb0x9YE7VcHq6LPvZoYKC7PxdTtCIiElextCl8AxgH7ABw92VArxj2sxbWeWRjcGfTPcB32j2Q2VVmVmZmZRUV6kwtIhIvsSSF2vDyDwBm1o2oL/c2lAMDo5YH8PFOb3nAEcCrZraKYEylyS01Nrv7Q+5e6u6lxcXFMZxaRET2RSxJ4TUz+wGQZWZnAk8CU2LYbxYwwsyGmFk6cAkQuWvJ3SvdvcjdS9y9BHgHmOjuZXv9V4iISIeIJSncSDAA3gLgamAq8KP2dgqn77wWeBF4D3jC3ReZ2W1mNnHfQxYRkXgx97avBJnZZ4Gp7t4ppt4sLS31sjJVJkRE9oaZzXb3dvuCxVJTmAi8b2Z/NrPzwjYFERHpgmLpp/BlYDhBW8JlwAozezjegYmISOLF9Kvf3evM7AWCu46yCDqhfS2egYmISOLF0nntHDP7A7AcuAh4mGA8JBER6WJiqSl8iWCIiqs7S2OziIjERyyT7FwSvWxm44DL3P0bcYtKRESSIqY2BTMbRdDIfDHwAfCPeAYlIiLJ0WpSMLNDCHohXwpsAR4n6NdwaoJiExGRBGurprAEeAM4392XA5jZDQmJSkREkqKtu48uBDYA083sd2Z2Oi2PfCoiIl1Eq0nB3Z92988DnwJeBW4AepvZg2Z2VoLiExGRBIqlR/Mud/+ru3+GYPjruQSD5ImISBcTy9hHEe6+1d1/6+6nxSsgERFJnr1KCiIi0rUpKYiISISSgoiIRCgpiIhIhJKCiIhEKCmIiEiEkoKIiEQoKYiISISSgoiIRCgpiIhIhJKCiIhEKCmIiEiEkoKIiEQoKYiISISSgoiIRCgpiIhIhJKCiIhEKCmIiEhEXJOCmZ1jZkvNbLmZfWJeZzP7tpktNrP5ZvYvMxscz3hERKRtcUsKZpYKPABMAA4DLjWzw5oV+zdQ6u5HAX8H7oxXPCIi0r541hTGAsvdfaW77wH+BkyKLuDu0929Olx8BxgQx3hERKQd8UwK/YE1Ucvl4brWfBV4oaUNZnaVmZWZWVlFRUUHhigiItHimRSshXXeYkGzK4BS4BctbXf3h9y91N1Li4uLOzBEERGJ1i2Oxy4HBkYtDwDWNS9kZmcAPwROdvfafTlRXV0d5eXl1NTU7FOgB4rMzEwGDBhAWlpaskMRkS4qnklhFjDCzIYAa4FLgMuiC5jZaOC3wDnuvmlfT1ReXk5eXh4lJSWYtVRBOfC5O1u2bKG8vJwhQ4YkOxwR6aLidvnI3euBa4EXgfeAJ9x9kZndZmYTw2K/AHKBJ81srplN3pdz1dTUUFhY2GUTAoCZUVhY2OVrQyKSXPGsKeDuU4GpzdbdHPX6jI46V1dOCE0Ohr9RRJJLPZo7wPbt2/n1r3+91/ude+65bN++PQ4RiYjsGyWFDtBaUmhoaGhzv6lTp5Kfnx+vsERE9lpcLx8dLG688UZWrFjBqFGjSEtLIzc3l759+zJ37lwWL17MBRdcwJo1a6ipqeG6667jqquuAqCkpISysjKqqqqYMGEC48eP56233qJ///48++yzZGVlJfkvE5GDTZdLCrdOWcTidTs69JiH9evOT84/vNXtt99+OwsXLmTu3Lm8+uqrnHfeeSxcuDByl9AjjzxCQUEBu3fv5thjj+XCCy+ksLDwY8dYtmwZjz32GL/73e+4+OKLeeqpp7jiiis69O8QEWlPl0sKncHYsWM/dtvofffdx9Nz/d1pAAAMTElEQVRPPw3AmjVrWLZs2SeSwpAhQxg1ahQAxxxzDKtWrUpYvCIiTbpcUmjrF32i5OTkRF6/+uqrvPzyy7z99ttkZ2dzyimntHhbaUZGRuR1amoqu3fvTkisIiLR1NDcAfLy8ti5c2eL2yorK+nZsyfZ2dksWbKEd955J8HRiYjErsvVFJKhsLCQcePGccQRR5CVlUXv3r0j28455xx+85vfcNRRRzFy5EiOP/74JEYqItI2c29xjLpOq7S01MvKyj627r333uPQQw9NUkSJdTD9rSLSccxstruXtldOl49ERCRCSUFERCKUFEREJEJJQUREIpQUREQkQklBREQilBQ6wL4OnQ1w7733Ul1d3cERiYjsGyWFDqCkICJdhXo0d4DoobPPPPNMevXqxRNPPEFtbS2f/exnufXWW9m1axcXX3wx5eXlNDQ08OMf/5iNGzeybt06Tj31VIqKipg+fXqy/xQROch1vaTwwo2wYUHHHrPPkTDh9lY3Rw+dPW3aNP7+978zc+ZM3J2JEyfy+uuvU1FRQb9+/Xj++eeBYEykHj16cPfddzN9+nSKioo6NmYRkX2gy0cdbNq0aUybNo3Ro0czZswYlixZwrJlyzjyyCN5+eWX+f73v88bb7xBjx49kh2qiMgndL2aQhu/6BPB3bnpppu4+uqrP7Ft9uzZTJ06lZtuuomzzjqLm2++OQkRioi0TjWFDhA9dPbZZ5/NI488QlVVFQBr165l06ZNrFu3juzsbK644gq++93vMmfOnE/sKyKSbF2vppAE0UNnT5gwgcsuu4xPf/rTAOTm5vKXv/yF5cuX873vfY+UlBTS0tJ48MEHAbjqqquYMGECffv2VUOziCSdhs4+wBxMf6uIdBwNnS0iIntNSUFERCKUFEREJKLLJIUDrW1kXxwMf6OIJFeXSAqZmZls2bKlS39pujtbtmwhMzMz2aGISBfWJW5JHTBgAOXl5VRUVCQ7lLjKzMxkwIAByQ5DRLqwuCYFMzsH+CWQCjzs7rc3254B/Ak4BtgCfN7dV+3tedLS0hgyZMj+BywicpCL2+UjM0sFHgAmAIcBl5rZYc2KfRXY5u7DgXuAO+IVj4iItC+ebQpjgeXuvtLd9wB/AyY1KzMJ+GP4+u/A6WZmcYxJRETaEM+k0B9YE7VcHq5rsYy71wOVQGEcYxIRkTbEs02hpV/8zW8PiqUMZnYVcFW4WGVmS/czttYUAZvjdOyOoPj2j+Lbf509RsXXusGxFIpnUigHBkYtDwDWtVKm3My6AT2Arc0P5O4PAQ/FKc4IMyuLZWyQZFF8+0fx7b/OHqPi23/xvHw0CxhhZkPMLB24BJjcrMxk4Ivh64uAV7wrdzYQEenk4lZTcPd6M7sWeJHgltRH3H2Rmd0GlLn7ZOD3wJ/NbDlBDeGSeMUjIiLti2s/BXefCkxttu7mqNc1wOfiGcNeivslqv2k+PaP4tt/nT1GxbefDrj5FEREJH66xNhHIiLSMQ66pGBmA81supm9Z2aLzOy6FsqcYmaVZjY3fNzc0rHiGOMqM1sQnrushe1mZveZ2XIzm29mYxIY28io92Wume0ws+ublUn4+2dmj5jZJjNbGLWuwMxeMrNl4XPPVvb9YlhmmZl9saUycYjtF2a2JPz/97SZ5beyb5ufhTjHeIuZrY36/3huK/ueY2ZLw8/jjQmM7/Go2FaZ2dxW9o3re9jad0pn+fztNXc/qB5AX2BM+DoPeB84rFmZU4DnkhjjKqCoje3nAi8Q9PM4Hng3SXGmAhuAwcl+/4CTgDHAwqh1dwI3hq9vBO5oYb8CYGX43DN83TMBsZ0FdAtf39FSbLF8FuIc4y3Ad2P4DKwAhgLpwLzm/57iFV+z7XcBNyfjPWztO6WzfP729nHQ1RTcfb27zwlf7wTe45M9rTu7ScCfPPAOkG9mfZMQx+nACndfnYRzf4y7v84n+7hED6PyR+CCFnY9G3jJ3be6+zbgJeCceMfm7tM86MUP8A5BP56kaeX9i0Usw9nst7biC4fGuRh4rKPPG4s2vlM6xedvbx10SSGamZUAo4F3W9j8aTObZ2YvmNnhCQ0s6NU9zcxmh725m4tlCJFEuITW/yEm8/1r0tvd10PwDxfo1UKZzvBefoWg5teS9j4L8XZteInrkVYuf3SG9+9EYKO7L2tle8Lew2bfKQfK5+9jDtqkYGa5wFPA9e6+o9nmOQSXRI4G7geeSXB449x9DMEIs98ws5OabY9peJB4CjskTgSebGFzst+/vZHU99LMfgjUA39tpUh7n4V4ehAYBowC1hNcomku6Z9F4FLariUk5D1s5zul1d1aWJfUW0IPyqRgZmkE//P+6u7/aL7d3Xe4e1X4eiqQZmZFiYrP3deFz5uApwmq6NFiGUIk3iYAc9x9Y/MNyX7/omxsuqwWPm9qoUzS3suwUfEzwOUeXmBuLobPQty4+0Z3b3D3RuB3rZw7qZ9FC4bH+Q/g8dbKJOI9bOU7pVN//lpz0CWF8Prj74H33P3uVsr0CcthZmMJ3qctCYovx8zyml4TNEgubFZsMvCF8C6k44HKpmpqArX66yyZ718z0cOofBF4toUyLwJnmVnP8PLIWeG6uLJgAqrvAxPdvbqVMrF8FuIZY3Q71WdbOXcsw9nE0xnAEncvb2ljIt7DNr5TOu3nr03JbOVOxgMYT1A9mw/MDR/nAl8Hvh6WuRZYRHAnxTvACQmMb2h43nlhDD8M10fHZwQTGK0AFgClCX4Pswm+5HtErUvq+0eQoNYDdQS/vr5KMAz7v4Bl4XNBWLaUYCbApn2/AiwPH19OUGzLCa4lN30GfxOW7QdMbeuzkMD378/h52s+wRdc3+YxhsvnEtxxsyJeMbYUX7j+D02fu6iyCX0P2/hO6RSfv719qEeziIhEHHSXj0REpHVKCiIiEqGkICIiEUoKIiISoaQgIiIRSgrSqZmZm9ldUcvfNbNbOujYfzCzizriWO2c53PhCJrTm60viR71U6QzUFKQzq4W+I8k9YhulZml7kXxrwLXuPup8YqnPWHPX5F2KSlIZ1dPMIXhDc03NP+lb2ZV4fMpZvaamT1hZu+b2e1mdrmZzQzH1R8WdZgzzOyNsNxnwv1TLZjvYFY4GNzVUcedbmaPEnTqah7PpeHxF5rZHeG6mwk6N/3GzH4Ryx9sZleG555nZk+ZWbaZ5ZnZB+FwCphZdwvmCUgzs2Fm9s9wwLc3zOxTUe/P3WEN5Q4zO9k+mn/g3009fUWi6deDHAgeAOab2Z17sc/RwKEEwy2vJOhBOtaCCVC+CTRNDFQCnEww8Nt0MxsOfIFg6JBjzSwDmGFm08LyY4Ej3P2D6JOZWT+CeRGOAbYRjMp5gbvfZmanEcxLEOsEL/9w99+Fx/0ZQe/d+83sVeA8ggEGLwGecvc6M3uIoFfvMjM7Dvg1cFp4rEOAM9y9wcymAN9w9xnh4G01McYjBxHVFKTT82DEyT8B39qL3WZ5MM59LcHwC01f6gsIEkGTJ9y90YNhl1cCnyIYf+YLFszk9S7BcAUjwvIzmyeE0LHAq+5e4cE8CX8lmBhmXxwR/uJfAFwONA09/jDw5fD1l4H/C7/cTwCeDOP9LcGkL02edPeG8PUM4G4z+xaQ7x/N5yASoZqCHCjuJRiS+/+i1tUT/rAJByVLj9pWG/W6MWq5kY9/7puP8+IEY0t9090/NjCZmZ0C7GolvpaGQN5XfwAucPd5ZvYlgpnsCH/hl5jZyUCquy80s+7Adncf1cqxIvG6++1m9jzBuDzvmNkZ7r6kA+OWLkA1BTkguPtW4AmCRtsmqwgu10Awy1XaPhz6c2aWErYzDAWWEoxS+Z9R1+8PCUfYbMu7wMlmVhQ2Ql8KvLYP8UAwpeP68PyXN9v2J4LB4f4PIrWoD8zsc2GsZmZHt3RQMxvm7gvc/Q6gjKBWJPIxSgpyILkLiL4L6XcEX8QzgeNo/Vd8W5YSfHm/QHBdvobgMs1iYE54y+hvaadW7cHQ5TcB0wlG5Jzj7i0NldzcSDMrj3p8DvgxQZJ5CWj+S/6vBHP5Rg9bfjnwVTNrGgm0tekwrw8bwecBu2l9tjc5iGmUVJEDSHi31SR3/3/JjkW6JrUpiBwgzOx+ghnvzk12LNJ1qaYgIiIRalMQEZEIJQUREYlQUhARkQglBRERiVBSEBGRCCUFERGJ+P9vNaPno0EmOwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "layers = np.arange(2,max_layers+2,1)\n",
    "ax.plot(layers, train_results_per_layer, label='train')\n",
    "ax.plot(layers, test_results_per_layer/test_set_size, label='test')\n",
    "ax.set_ylim(0,1.1)\n",
    "ax.set_xlabel('Number of Layers')\n",
    "ax.set_ylabel('Average Accuracy')\n",
    "ax.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### h = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of layers: 2\n",
      "Max training accuracy: 0.9899999976158143\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 11.560841654\n",
      "Number of layers: 3\n",
      "Max training accuracy: 0.9804166615009308\n",
      "Max test accuracy: 0.9774999999999999\n",
      "Time elapsed: 15.513677497999993\n",
      "Number of layers: 4\n",
      "Max training accuracy: 0.9893749952316284\n",
      "Max test accuracy: 0.9841666666666666\n",
      "Time elapsed: 19.044559433000018\n",
      "Number of layers: 5\n",
      "Max training accuracy: 0.9618749976158142\n",
      "Max test accuracy: 0.9516666666666667\n",
      "Time elapsed: 22.928651548000005\n",
      "Number of layers: 6\n",
      "Max training accuracy: 0.9993749916553497\n",
      "Max test accuracy: 0.9958333333333333\n",
      "Time elapsed: 26.78676890899999\n",
      "Number of layers: 7\n",
      "Max training accuracy: 0.9714583337306977\n",
      "Max test accuracy: 0.9674999999999999\n",
      "Time elapsed: 29.42680865600005\n",
      "Number of layers: 8\n",
      "Max training accuracy: 0.9977083265781402\n",
      "Max test accuracy: 0.995\n",
      "Time elapsed: 32.99152629499997\n",
      "Number of layers: 9\n",
      "Max training accuracy: 0.998749989271164\n",
      "Max test accuracy: 0.9975\n",
      "Time elapsed: 36.23287781099998\n",
      "Number of layers: 10\n",
      "Max training accuracy: 0.9947916567325592\n",
      "Max test accuracy: 0.9924999999999999\n",
      "Time elapsed: 41.42379268799999\n",
      "Number of layers: 11\n",
      "Max training accuracy: 0.9804166674613952\n",
      "Max test accuracy: 0.9766666666666667\n",
      "Time elapsed: 46.850839864999955\n",
      "Number of layers: 12\n",
      "Max training accuracy: 0.9991666615009308\n",
      "Max test accuracy: 0.9958333333333333\n",
      "Time elapsed: 51.28697093400001\n",
      "Number of layers: 13\n",
      "Max training accuracy: 0.9997916638851165\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 50.04483291100007\n",
      "Number of layers: 14\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 57.123639293\n",
      "Number of layers: 15\n",
      "Max training accuracy: 0.9993749976158142\n",
      "Max test accuracy: 0.995\n",
      "Time elapsed: 60.57194835600001\n",
      "Number of layers: 16\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 61.30757585399999\n",
      "Number of layers: 17\n",
      "Max training accuracy: 0.9995833337306976\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 68.96126790999995\n",
      "Number of layers: 18\n",
      "Max training accuracy: 0.9997916638851165\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 68.21732349400008\n",
      "Number of layers: 19\n",
      "Max training accuracy: 0.9997916638851165\n",
      "Max test accuracy: 0.9975\n",
      "Time elapsed: 74.73729201300011\n",
      "Number of layers: 20\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 77.83502132299986\n",
      "Number of layers: 21\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 81.43732316499995\n",
      "Number of layers: 22\n",
      "Max training accuracy: 0.9993749976158142\n",
      "Max test accuracy: 0.9975\n",
      "Time elapsed: 85.28123406400005\n",
      "Number of layers: 23\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 90.06223217700017\n",
      "Number of layers: 24\n",
      "Max training accuracy: 0.9960416674613952\n",
      "Max test accuracy: 0.9916666666666667\n",
      "Time elapsed: 92.66517453300003\n",
      "Number of layers: 25\n",
      "Max training accuracy: 0.9997916638851165\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 95.97931334400005\n",
      "Number of layers: 26\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 100.53597212099999\n",
      "Number of layers: 27\n",
      "Max training accuracy: 0.9912500023841858\n",
      "Max test accuracy: 0.9866666666666667\n",
      "Time elapsed: 101.51435039700004\n",
      "Number of layers: 28\n",
      "Max training accuracy: 0.9912500023841858\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 112.91596206700001\n",
      "Number of layers: 29\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 119.3728034369999\n",
      "Number of layers: 30\n",
      "Max training accuracy: 0.9914583325386047\n",
      "Max test accuracy: 0.9883333333333333\n",
      "Time elapsed: 116.05551962800018\n",
      "Number of layers: 31\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 120.55024232599999\n",
      "Number of layers: 32\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 117.96724895600028\n",
      "Number of layers: 33\n",
      "Max training accuracy: 0.9997916638851165\n",
      "Max test accuracy: 0.9966666666666666\n",
      "Time elapsed: 117.7910427730003\n",
      "Number of layers: 34\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 118.09478376700008\n",
      "Number of layers: 35\n",
      "Max training accuracy: 0.9902083277702332\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 123.688418619\n",
      "Number of layers: 36\n",
      "Max training accuracy: 0.9897916615009308\n",
      "Max test accuracy: 0.9891666666666666\n",
      "Time elapsed: 125.21602652899992\n",
      "Number of layers: 37\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 129.6828030669999\n",
      "Number of layers: 38\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9975\n",
      "Time elapsed: 132.25518220200001\n",
      "Number of layers: 39\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 133.74883000800037\n",
      "Number of layers: 40\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 136.43705071399972\n",
      "Number of layers: 41\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 141.61712311500014\n",
      "Number of layers: 42\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 143.65474330100005\n",
      "Number of layers: 43\n",
      "Max training accuracy: 0.9910416662693023\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 148.83649615100012\n",
      "Number of layers: 44\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 151.38992973899985\n",
      "Number of layers: 45\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 152.79740987500008\n",
      "Number of layers: 46\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 156.16016710899976\n",
      "Number of layers: 47\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 158.67996701799984\n",
      "Number of layers: 48\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 162.94197074800013\n",
      "Number of layers: 49\n",
      "Max training accuracy: 0.9906249940395355\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 169.43397408800047\n",
      "Number of layers: 50\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 170.2838490690001\n",
      "Number of layers: 51\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 172.13271752899982\n",
      "Number of layers: 52\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 177.28774047499974\n",
      "Number of layers: 53\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9966666666666666\n",
      "Time elapsed: 181.3304354259999\n",
      "Number of layers: 54\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 184.07334761699985\n",
      "Number of layers: 55\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 188.28717347200018\n",
      "Number of layers: 56\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 191.70016069400026\n",
      "Number of layers: 57\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 194.79652004499985\n",
      "Number of layers: 58\n",
      "Max training accuracy: 0.9908333361148834\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 197.01821301699965\n",
      "Number of layers: 59\n",
      "Max training accuracy: 0.9983333289623261\n",
      "Max test accuracy: 0.9941666666666666\n",
      "Time elapsed: 202.8740099280003\n",
      "Number of layers: 60\n",
      "Max training accuracy: 0.9964583337306976\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 205.2021295300001\n",
      "Number of layers: 61\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 207.44053961499958\n",
      "Number of layers: 62\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9966666666666666\n",
      "Time elapsed: 212.96824571100024\n",
      "Number of layers: 63\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 217.55050089699944\n",
      "Number of layers: 64\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 214.78338487600013\n",
      "Number of layers: 65\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 222.56396043400036\n",
      "Number of layers: 66\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 226.36905133999971\n",
      "Number of layers: 67\n",
      "Max training accuracy: 0.9908333361148834\n",
      "Max test accuracy: 0.9866666666666667\n",
      "Time elapsed: 224.8667936039992\n",
      "Number of layers: 68\n",
      "Max training accuracy: 0.9908333361148834\n",
      "Max test accuracy: 0.9891666666666666\n",
      "Time elapsed: 232.7888544360012\n",
      "Number of layers: 69\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 233.37196860900076\n",
      "Number of layers: 70\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 240.20815632599988\n",
      "Number of layers: 71\n",
      "Max training accuracy: 0.9912500023841858\n",
      "Max test accuracy: 0.9866666666666667\n",
      "Time elapsed: 238.12200973400104\n",
      "Number of layers: 72\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 242.04223795500002\n",
      "Number of layers: 73\n",
      "Max training accuracy: 0.9910416662693023\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 248.07327821999934\n",
      "Number of layers: 74\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 248.85591514399857\n",
      "Number of layers: 75\n",
      "Max training accuracy: 0.9887499928474426\n",
      "Max test accuracy: 0.9833333333333333\n",
      "Time elapsed: 254.85861445400042\n",
      "Number of layers: 76\n",
      "Max training accuracy: 0.9997916638851165\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 258.6869160320002\n",
      "Number of layers: 77\n",
      "Max training accuracy: 0.9995833337306976\n",
      "Max test accuracy: 0.9966666666666666\n",
      "Time elapsed: 254.4130401700004\n",
      "Number of layers: 78\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 266.318600306\n",
      "Number of layers: 79\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 266.6607109810011\n",
      "Number of layers: 80\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 268.9102174869986\n",
      "Number of layers: 81\n",
      "Max training accuracy: 0.9912500023841858\n",
      "Max test accuracy: 0.9883333333333333\n",
      "Time elapsed: 269.67084817199975\n",
      "Number of layers: 82\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 274.67955577299836\n",
      "Number of layers: 83\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 280.56312097999944\n",
      "Number of layers: 84\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9966666666666666\n",
      "Time elapsed: 281.2702959729995\n",
      "Number of layers: 85\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 286.8001328320006\n",
      "Number of layers: 86\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 290.2325940520004\n",
      "Number of layers: 87\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 292.168098999\n",
      "Number of layers: 88\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9975\n",
      "Time elapsed: 297.0931454700003\n",
      "Number of layers: 89\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 297.68941588800044\n",
      "Number of layers: 90\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 307.2638054979998\n",
      "Number of layers: 91\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 302.91096389700033\n",
      "Number of layers: 92\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 309.750666730999\n",
      "Number of layers: 93\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 312.21219423899856\n",
      "Number of layers: 94\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 316.1715179879993\n",
      "Number of layers: 95\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 318.5060129169997\n",
      "Number of layers: 96\n",
      "Max training accuracy: 0.9914583325386047\n",
      "Max test accuracy: 0.9866666666666667\n",
      "Time elapsed: 321.2589229900004\n",
      "Number of layers: 97\n",
      "Max training accuracy: 0.9997916638851165\n",
      "Max test accuracy: 0.9966666666666666\n",
      "Time elapsed: 327.7267574559992\n",
      "Number of layers: 98\n",
      "Max training accuracy: 0.9910416662693023\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 328.8148559200017\n",
      "Number of layers: 99\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 332.04138873400007\n",
      "Number of layers: 100\n",
      "Max training accuracy: 0.9975000023841858\n",
      "Max test accuracy: 0.9924999999999999\n",
      "Time elapsed: 335.42228341699956\n",
      "Number of layers: 101\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 338.5545850740018\n",
      "Number of layers: 102\n",
      "Max training accuracy: 0.9987500011920929\n",
      "Max test accuracy: 0.9916666666666667\n",
      "Time elapsed: 341.84425531700253\n",
      "Number of layers: 103\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9975\n",
      "Time elapsed: 347.24764150899864\n",
      "Number of layers: 104\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 347.2725166959972\n",
      "Number of layers: 105\n",
      "Max training accuracy: 0.9731249988079071\n",
      "Max test accuracy: 0.9641666666666667\n",
      "Time elapsed: 352.7111594429989\n",
      "Number of layers: 106\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 359.5338315910012\n",
      "Number of layers: 107\n",
      "Max training accuracy: 0.9835416674613953\n",
      "Max test accuracy: 0.9758333333333333\n",
      "Time elapsed: 359.90718969500085\n",
      "Number of layers: 108\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 359.0597965779998\n",
      "Number of layers: 109\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 379.68127323500084\n",
      "Number of layers: 110\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 367.1966676810007\n",
      "Number of layers: 111\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9983333333333333\n",
      "Time elapsed: 372.0439055530005\n",
      "Number of layers: 112\n",
      "Max training accuracy: 0.9908333361148834\n",
      "Max test accuracy: 0.9866666666666667\n",
      "Time elapsed: 377.9670397339978\n",
      "Number of layers: 113\n",
      "Max training accuracy: 0.9916666686534882\n",
      "Max test accuracy: 0.9883333333333333\n",
      "Time elapsed: 379.37563177500124\n",
      "Number of layers: 114\n",
      "Max training accuracy: 0.9902083337306976\n",
      "Max test accuracy: 0.9883333333333333\n",
      "Time elapsed: 384.4507297539967\n",
      "Number of layers: 115\n",
      "Max training accuracy: 0.9820833325386047\n",
      "Max test accuracy: 0.9758333333333333\n",
      "Time elapsed: 386.25086669899974\n",
      "Number of layers: 116\n",
      "Max training accuracy: 0.9616666674613953\n",
      "Max test accuracy: 0.9491666666666667\n",
      "Time elapsed: 388.45869968499755\n",
      "Number of layers: 117\n",
      "Max training accuracy: 0.9997916638851165\n",
      "Max test accuracy: 0.9975\n",
      "Time elapsed: 389.9558204540008\n",
      "Number of layers: 118\n",
      "Max training accuracy: 0.9716666638851166\n",
      "Max test accuracy: 0.9625\n",
      "Time elapsed: 393.06096502700166\n",
      "Number of layers: 119\n",
      "Max training accuracy: 1.0\n",
      "Max test accuracy: 0.9975\n",
      "Time elapsed: 401.4344285139996\n",
      "Number of layers: 120\n",
      "Max training accuracy: 0.9912499964237214\n",
      "Max test accuracy: 0.9883333333333333\n",
      "Time elapsed: 396.3832362110006\n",
      "Number of layers: 121\n",
      "Max training accuracy: 0.9897916555404663\n",
      "Max test accuracy: 0.9883333333333333\n",
      "Time elapsed: 406.2036418839998\n"
     ]
    }
   ],
   "source": [
    "from Networks.ResNet import ResAntiSymNet\n",
    "import numpy as np\n",
    "import timeit\n",
    "\n",
    "num_epochs = 100\n",
    "num_features = 2\n",
    "num_classes = 2\n",
    "size_hidden = 8\n",
    "gamma = 0.3\n",
    "h = 0.1\n",
    "lr = 0.5\n",
    "\n",
    "max_layers = 120\n",
    "train_results_per_layer = np.zeros(max_layers)\n",
    "test_results_per_layer = np.zeros(max_layers)\n",
    "basic_save_key = 'AntiSym_Comparison_results//AntiSymNet'\n",
    "\n",
    "for num_hidden_layers in range(1,max_layers+1):\n",
    "    tic = timeit.default_timer()\n",
    "    num_layers = num_hidden_layers + 1\n",
    "    print('Number of layers: '+str(num_layers))\n",
    "    train_results_per_epoch = np.zeros(num_epochs)\n",
    "    test_results_per_epoch = np.zeros(num_epochs)\n",
    "    for i in range(10):\n",
    "        net = ResAntiSymNet(features=num_features, classes=num_classes, num_layers=num_layers, gamma=gamma, h=h, bias=True, hidden_size=size_hidden)\n",
    "        net.set_test_tracking(True)\n",
    "        net.train(num_epochs, train_loader, test_loader, print_output=False)\n",
    "        train_results_per_epoch += net.avg_correct_pred.numpy()\n",
    "        test_results_per_epoch += net.test_results.numpy()\n",
    "        \n",
    "    train_results_per_epoch = train_results_per_epoch / 10\n",
    "    test_results_per_epoch = test_results_per_epoch / 10\n",
    "    train_save_key = basic_save_key+'_train_per_epoch_layers_'+str(num_layers)+'_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "    test_save_key = basic_save_key+'_test_per_epoch_layers_'+str(num_layers)+'_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "    \n",
    "    np.savetxt(train_save_key, train_results_per_epoch, delimiter=',')\n",
    "    np.savetxt(test_save_key, test_results_per_epoch, delimiter=',')\n",
    "    \n",
    "    train_results_per_layer[num_hidden_layers-1] = train_results_per_epoch.max()\n",
    "    print('Max training accuracy: '+str(train_results_per_epoch.max()))\n",
    "    test_results_per_layer[num_hidden_layers-1] = test_results_per_epoch.max()\n",
    "    print('Max test accuracy: '+str(test_results_per_epoch.max()/test_set_size))\n",
    "    \n",
    "    toc = timeit.default_timer()\n",
    "    print('Time elapsed: '+str(toc-tic))\n",
    "\n",
    "train_save_key = basic_save_key+'_train_per_layer_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "test_save_key = basic_save_key+'_test_per_layer_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "np.savetxt(train_save_key, train_results_per_layer, delimiter=',')\n",
    "np.savetxt(test_save_key, test_results_per_layer, delimiter=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x205c67c4e80>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl4FeX5//H3nZN9Iez7ElQULSpIxF1RsbIouItrrVT0a7Xqr9W6VK1+e7W2+tXW1n2pdamKYhUVNyy4VdSAyK5EQYiAIHv27f79MYfjIYTkkObkkPh5XVeunJl5zsz9ZE7mPs88M8+YuyMiIgKQlOgARERk16GkICIiEUoKIiISoaQgIiIRSgoiIhKhpCAiIhFKCiIiEqGkICIiEUoKIiISkZzoAHZW586dPS8vL9FhiIi0KrNmzfrO3bs0Vq7VJYW8vDwKCgoSHYaISKtiZl/HUk6nj0REJEJJQUREIpQUREQkQklBREQilBRERCRCSUFERCKUFEREJEJJQUREIpQUREQkQklBREQilBRERCRCSUFERCKUFEREJEJJQUREIuKWFMzsUTNbY2bzd7DczOxuMys0s7lmdkC8YhERkdjEs6XwGDCygeWjgAHhn4nAfXGMRUREYhC3pODu7wLrGygyDnjcAzOB9mbWI17xiIhI4xL55LVewIqo6aLwvFWJCSdQVVPLcwVFnHJAL9JTQpH5324uZ+l3JWzcvIU0q2HYwH5kpW3/5/tP4XfM/upb2q+fQ4ctn1NBKsWWw8aMPmxutyfpqSmUV1aTteVLOpUtJYdSsr2UqqQ0SpKyKU3KoTyUTVkoi4yqjeSWrySrej0WXn9lSjvKsnpRndaRnC1f0ql4MVU5fek2/GIG9enIsnWlzPvobZK++YSypCxKQrnse8RJDN1j+3z76dLVzJn1EWOOP56uOekAfFW4iNWv30F1LZSFctiU0pXVWXuxLr0fuRWr6V66mJyOPRg57ixSQtt+p6isquGtFx6leNN3rE/tQUkol9yqtXSsWkWS11AWyqEiKYO02lIyaopJ8UoAPJTKgBETGLhbPwA2lVbyzgv3kbQl+CjUkkRZKJuyUDYhryajppi02tLI3yR9z6M59pgfY2a4O29Pn0bxF+8Ef8ekbFJry8ioKcYtxDfpe7A6rT+dK79hj5LZdK9YiuFB/JYWjjETt6BuaXscyYgRozALtrZmczlfry+lX8dMOmensWj1ZmYuWkbFijlsSunChpRupNWU0rFqFTmR/eaRGDJqtpBRW0xGTTE1eUdw2NiLIuvean1JJS+/PJmcTZ/TsXIVSdldGDr+RrIy0gBYMGcmGz+ZxD6n30SH9u0j+23Vvx9g4Kk30KlT8MTFjVtKeO+lh9nsGazNGkhpcg6ZpSvJKltJZvWmcBxbIn/PtCFncMCRJwZ/i+pa/jVtBiz/iI5Vq8iq3sSKjL0ozDqAWktiQMls+pUuINmrtondMdal9uCb9D1Zn9qDtJpSMmqLaV+1ho6Vq2hXvS7y945+T3lSFmWhbKqS0gDDvJa02lIya7ZQ06E/+Wf+hozw/1vBsvV8tHQ9FxyaF/kfXPTlMr5491lKdh/Dnn17UlXjrPyigK5fTmZhz5NJ7jqQkMGaLRVUblhJp4rldKxcSW6X3ow8+fxILMtXLGfBzNc57pSfkRz+fH+zoYS3XnmWtPK1kb9XRs0WUmvL2ZLSifUpPbB+h3DKiCMj74nVxtJKnv3P51wwfB/SkoPjTWllNQ+9vYDRB+zGgG45O7W+5mDu3nippq7cLA94xd0H1bPsVeAP7v5+ePpt4Bp3n1VP2YkEp5jo27fv0K+/jumpck0ydeZcUl69gm8PvZVzRx0BwBcr1/P+/ZdxCPPYw74BYIYfwIJuYxkx7jwG9e4AwGfLN7DooQsZl/Q+GVa53bo3ejbzvD9723I626ZmibeaEMnU8FntbvzNzmZ07XRODn2wTZnZtXswY+jfuHT0MNJTQlRU1/DgqzM5fNYvGJJUyGSOJfmE2ylb9QVHF/wP7Smm2pLJomyH230m61yOuuh2erTPBODb79bx+UMXcmTFjCbVo9B78cHBD7Jb/93YPOlSxtROj/m9FZ7MP7rfwMgzLuH15x7gJ6t+R5pVx/Te7+hANcmAk04F2ZSQTG1keaWHeLTLtYw5+zImz1rB7HdfJrWmlLdrhxBKCpHnK3gk5Q76Ja2JOd4qkqkglWxKebTjlZxwwXV0bRck5ffnf0nx5F8wMvi3oJJkUqnm1dCxtDvzPhbNm8WpcyfSybYwy35E8SlP8O3K5Rz+nwn0tHW8EzqIXhMnEwol8dkDEzip6rVG4yknlRqSSPEqJvW+nr1G/JSPnruDi0oeJM2qqSGJctK2+zxsJotSMreZF6KGLjs4QVBNEuvpQG2dExRJ1JJF6XbrryaJUjJoRwnPp5xIl9Pu5N8Liug3+48MT5rD3zIu4bQzzuOLxQs44uNL2N1WssGzebD6BHrYOs4JTSNkTqUnc3f1yczyPbkoeSrHJH0a2Ualh/jy/FnsvXt/AKbdfg4jSl7h6b3u5qyzfkJZZQ3P3fkLzi9/cpvYismknDQ6sIkQtZR5Kne2v57zzr+Yvp22/Zs05MXnn2DUvCt564hJnDDiWACef+VVxn1yHlP8cDYP/x3nD9+XUJI1sqbGmdksd89vtFwCk8IDwAx3fzo8/Tkw3N0bbCnk5+d7U57R/OnS1bw+/R0uHX8KuZkpABRtKOWdL9ZyZn6fSIZ/8u4bOHf933gh6ThGX/cs6Skhnrz/95y7+o9s6HYINT2HUlFRRu4XL5BdvYGXOYp9Ln2Knu0zeeD/rufKivupHHQmqfueDL3yoaYSyjbAt/Nh6buwcg502wf6HwU99oeMDpCWA9XlUL4JyjYGv8s3QUZ7aN8PcrpDUgjcofQ72PB18LvzntBlIMVzXiD5jWtJr1xPTVIqxUP/h9wjL4Xqciq+/ICkV69keU0nLku6nuqUHHpULecP/he6JJVQvPsYOhZOZlFtX3rZd9SkZMI5k+nQfzDUVMOGZbD6M1j7BXToB933Zfmrd9B3xYs8x3Gs6nU83WtWM3T1s+R5EUt+dAUDR/wkHOM6aNcreF8oNahTxZagvuntITkNzNhSOJPk589jY00aS2p7cWRoHquHXEX3UVcHO6+m6vu/SSgV0nMhLRssidqyzax55Ey6bprLi7WHcVLSB6zJ3ZeuFzxBErVQvhlSs4LtVZfBqrmwZiG07wt5R0Bur20/KO5QVQY4teVbWPPoWXTfOJt7q8dyQNISDk5aBMCG7D34tMPxHLHqHySlZhAa/UeoLIVNKyCtXVDnnB6wtRWQkhnEkJ4LKRl4dQXfPHAaPde+z2/tYtbn7M0e1Us4rfQZeth61uVfRdcjL4Lsbnzz0m/p9dlfeLHmUA5NWkhGSoiNgy+iZ8GfmFfbnx62jsyQs2n3sfRe8gR3cS7FlsmN/iCr97mQ7gedAavmBH+LDv2Cumd1CWJJz4XkNCqLN/Dtg6fQZ/NsPqodyEFJi/mu+xF0Pu2u4DMYSoG1i4PPsNcGf7uu+0BSPd+MyzfD6nmweSWktwvqndM9+CyEGjg5UVMNNRXhCYOUDABWPnsVPRf/nX9UH8e+SUs5IKmQioyupJWt4Z/Vx3BsaDY5oWo4/vckLZ5C+tJpuIUoH3wBGYdMxGf8AVv4YrB7MztjB06AvodQUlZO1vPjebbL5Zz5898xd9m39P37ENpbCQtr+7HstKl8MncRVy85h5LeR9Dl1NuDuqS1+74e1ZWw/is2PvVTsjct5qbaiSzMzCe3fCXtfTO5Gcm0z85gz2GjGXPggO1ahS//fjwnVr7Gs8ljOe36x6mpdV7+/XjG1r5NiFpW1HbmsdSzyM7JpWNWCvvnH8kBgwc3eryrT2tICmOAy4DRwEHA3e4+rLF1NjUpLHjyGgYseZi/plzI0eddz9LvSnlkytv8qHo+h5xyOacM7cvG0ko+u20ERyV9RoWnMHXEm+y75x5wz0G0y86k69UF3/+T11Sx6bVbyS24m78mX8CmfsdzVeEFVPY8kA4TX/m+XEspXQ9znoK9T4QOedsuW/Y+1U+NJ7lqS2RWeWZ30s+bBD32p2bRVGomT6QyowvZE6ZA+z4Nb8udDS/fQIfZ90RmbbAOlJxwL72Hjm5S+L56HhWPnUxq+TqqR95B6sETYn9zZSmbnzyPdsunsanPseSe9ySkxv5trUFVZWz55wXkLH2dyowupA6/BjI7wozbYN0S6LYvnP0M5PZuwrrLKf3HaWQWvReZtT69D1njHyEt76Dvy7lT9eZNpHx4N5VpHUmd8Bp0HUjlvBcJvTCBytT2pF/4MtZ1b0qfOo+0wldxjMq+R5J5weTgC0WM8Wx86gJyl71O2WHXkHnstfUf9FuaO1VTriLl079Tk5JF6KR7YM+RVL3+G1JmPUxZejcyLnwJuu4dlF89L0jCnXb/fh2Fb0PJWthnXCTZAKz5Uz6rimvIuuxd3nr+Qf5nza1UDDqLtPlPc3X1JRxi8xmb/DHJv/hk+/+raBVbKH/ybNJXvFvv4kW1fflb79u55tQj6NcpKwhpzRZCfxtK/6RvWeu5zDrlA0orqjjm1cOpyDuGrsdcStmzE8gsXRlZz5z9bmLwKb9s0p8x4UnBzJ4GhgOdgW+Bm4EUAHe/34KU+TeCK5RKgZ+6e6NH+6YmBUrXs+npCeSu+Dcv1RxKqadxRvI7hKjl5szruflX1zB55ueMfeNwKvoeQfaK6TyVcjoVXfblZytvZMuY+8k58Kxt1+nOpsfPJvur11jqPeidvIn0Kz5u2gEi3tZ+AQtfCr5hp7eHAcdBVufvl5dtgOQMSEmPfZ1ffwi11eFvxT0b/hYYi+I1sGVV0ILaWTXVsPw/0PfQ/z6OumprYOk70Ofg75NNTTUsew/6DAtaIk1VWRok88yO0H1/6Lhb/Qdid5g7CXoNhc57fD//2wWQ0RHahfuMKrbgD40Ar8V+9lbQ2twZ7sF+yOnW9DrFQ20tzHky2L/R9f9mFuT2geyuTVptyYy/kDXjJq7sdD9j1jzIIRkryP71YiofGkHZ6iXkUowf/ktsxE2Nr6y6Emb/I/hC2D4viMmMmu8K8RcuYXlNJy703/CPK8bRr1MWj7w8gwmzxlHR53DSVrzPre1uwarLuLH0NvycydiAEUGrdV3h99vI6QlZnZpU14QnhXhpclIAqK2lfMYdpL77B9xC2IEXUjH3X3xY0pOasybx2bSn+dX6m/HzXuTbaX8lbeVHfOOd6ZVRRYdr5tZ/sKksoeS+Y8nasIjKMX8l9cDzty8j0pKqyoPfO5Pgf6iK11B7x0Cerz6ck0PvU3PQpaSP/h18/R/4+yg8uzt2+azgy9R/Y9kH1D51Ossr2/G/vR/g4Z8dyZ233cAvK+6BS96n/JETmFY+kDQqOTyriIxrFsfewotRrEkhkVcftbykJNKPuQb2GR2cy8/tRVpaO4a/dwfnvfYeYza+S2VKJqn9DqPLyAxCfz+eDlbMliP+tONvn6lZZF34Enz9Pqk/OqVl6yNSHyWD2GV3pWq3Yzjjq7cASMk/J5jf71AYfQfWbdB/nxAA8g4jafyT5D1xMn2WPc9tr3Vl39JZlGZ2JbPbIJL3P53jCh4jRC025OfNnhB2xi5wwjABug+KdDAmDT0fA4ZteIXhSZ9S2e9ISE4l1O9gSrrlU5HehZyDftLw+nK6waBTW74fQUT+a2lDg0TgPQZ/3y8BMOwi6HdI821o92PwfodxWdqrPPbuYg5Nmk/S7sPBjOQhZ5NGFcnUEBpydvNtswl+WC2F+rTvS+3ux/KzL18ji3LYd0xkUdb5z0Jlib55ibRle46CHoOxQy+P+6bsyKvp/MRJ3JzyBB2tGPYaESzoOQS6/ii4Ii86MSWAkgIQyv8pWV9OCyYG/Pj7BVmdt+2MFZG2JyUdLn6nZba123Dolc/Z37wdnj4q+G0G570AlviTN4mPYFew5/GQ3T246iWne6KjEZG2ygyOuiZ43XWfbY83Od2bfBVVc1JLAYIbc855LrgxSkQkngb8GAaeAHmHJzqSeikpbNVjv0RHICI/BGYw/qlER7FDOn0kIiIRSgoiIhKhpCAiIhFKCiIiEqGkICIiEUoKIiISoaQgIiIRSgoiIhKhpCAiIhFKCiIiEqGkICIiEUoKIiISoaQgIiIRSgoiIhKhpCAiIhFKCiIiEqGkICIiEUoKIiISoaQgIiIRSgoiIhKhpCAiIhFKCiIiEqGkICIiEUoKIiISEdekYGYjzexzMys0s2vrWd7XzKab2admNtfMRsczHhERaVjckoKZhYB7gFHAPsBZZrZPnWK/ASa5+xBgPHBvvOIREZHGxbOlMAwodPev3L0SeAYYV6eMA+3Cr3OBlXGMR0REGpEcx3X3AlZETRcBB9Up81vgTTO7HMgCRsQxHhERaUQ8WwpWzzyvM30W8Ji79wZGA0+Y2XYxmdlEMysws4K1a9fGIVQREYH4JoUioE/UdG+2Pz00AZgE4O4fAulA57orcvcH3T3f3fO7dOkSp3BFRCSeSeETYICZ9TezVIKO5Cl1yiwHjgUws70JkoKaAiIiCRK3pODu1cBlwBvAIoKrjBaY2a1mNjZc7JfARWb2GfA0cIG71z3FJCIiLSSeHc24+1Rgap15N0W9XggcFs8YREQkdo22FMxsUEsEIiIiiRfL6aP7zexjM7vUzNrHPSIREUmYRpOCux8OnENwJVGBmf3TzI6Le2QiItLiYupodvclBENS/Bo4CrjbzBab2SnxDE5ERFpWLH0K+5nZXQRXEB0DnOjue4df3xXn+EREpAXFcvXR34CHgOvdvWzrTHdfaWa/iVtkIiLS4mJJCqOBMnevAQgPQ5Hu7qXu/kRcoxMRkRYVS5/CNCAjajozPE9ERNqYWJJCursXb50Iv86MX0giIpIosSSFEjM7YOuEmQ0FyhooLyIirVQsfQpXAs+Z2dYRTnsAZ8YvJBERSZRGk4K7f2JmA4G9CJ6RsNjdq+IemYiItLhYB8Tbi+A5y+nAEDPD3R+PX1giIpIIjSYFM7sZGE6QFKYCo4D3ASUFEZE2JpaO5tMIHoSz2t1/CuwPpMU1KhERSYhYkkKZu9cC1WbWDlgD7BbfsEREJBFi6VMoCA+Z/RAwCygGPo5rVCIikhANJgUzM+AP7r6R4LkKrwPt3H1ui0QnIiItqsHTR+HnJb8YNb1MCUFEpO2KpU9hppkdGPdIREQk4WLpUzgauNjMvgZKCG5gc3ffL66RiYhIi4slKYyKexQiIrJLiCUpeNyjEBGRXUIsSeFVgsRgBMNc9Ac+B34Ux7hERCQBYhkQb9/o6fAw2hfHLSIREUmYWK4+2oa7zwZ0NZKISBsUy4B4/y9qMgk4AFgbt4hERCRhYulTyIl6XU3QxzA5PuGIiEgixdKncEtLBCIiIonXaJ+Cmb0VHhBv63QHM3sjvmGJiEgixNLR3CU8IB4A7r4B6Bq/kEREJFFiSQo1ZtZ364SZ9SPGG9rMbKSZfW5mhWZ27Q7KnGFmC81sgZn9M7awRUQkHmLpaL4BeN/M3glPHwlMbOxNZhYC7gGOA4qAT8xsirsvjCozALgOOMzdN5iZWiAiIgkUS0fz6+Eb1g4muKv5Knf/LoZ1DwMK3f0rADN7BhgHLIwqcxFwT/iUFO6+ZifjFxGRZhRLR/PJQJW7v+LuLxM8lvOkGNbdC1gRNV0UnhdtT2BPM/vAzGaa2cgdxDDRzArMrGDtWt0iISISL7H0Kdzs7pu2ToQ7nW+O4X1Wz7y6fRHJwABgOHAW8HD0lU5R23zQ3fPdPb9Lly4xbFpERJoilqRQX5lY+iKKgD5R072BlfWUecndq9x9KcFAewNiWLeIiMRBLEmhwMzuNLPdzWw3M7sLmBXD+z4BBphZfzNLBcYDU+qUeZHgIT6YWWeC00lfxR6+iIg0p1iSwuVAJfAs8BxQDlza2JvcvRq4DHgDWARMcvcFZnarmY0NF3sDWGdmC4HpwNXuvm7nqyEiIs3B3HfuGTpmlg6c6O7PxSekhuXn53tBQUEiNi0i0mqZ2Sx3z2+sXExDZ5tZyMxGmdnjwDLgzP8yPhER2QU12GFsZkcCZwNjgI+Bw4Dd3L20BWITEZEWtsOkYGZFwHLgPoJz/VvMbKkSgohI29XQ6aPJBDebnQmcaGZZxDjmkYiItE47TArufgWQB9xJcNnoF0CX8AB22S0TnoiItKQGO5o98G93v4ggQZwNnETQ2SwiIm1MLHcmA+DuVcDLwMtmlhG/kEREJFFiuiS1Lncva+5AREQk8ZqUFEREpG2KOSmErz4SEZE2LJbnKRwaHptoUXh6fzO7N+6RiYhIi4ulpXAXcDywDsDdPyN4JKeIiLQxMZ0+cvcVdWbVxCEWERFJsFguSV1hZocCHn4uwi8In0oSEZG2JZaWwiXAzwmGvCgCBoenRUSkjWm0peDu3wHntEAsIiKSYI0mBTO7u57Zm4ACd3+p+UMSEZFEieX0UTrBKaMl4Z/9gI7ABDP7cxxjExGRFhZLR/MewDHhZy5jZvcBbwLHAfPiGJuIiLSwWFoKvYDou5mzgJ7uXgNUxCUqERFJiFhaCn8C5pjZDMAIblz7fXjYi2lxjE1ERFpYLFcfPWJmU4FhBEnhendfGV58dTyDExGRlhXrgHjlwCpgPbCHmWmYCxGRNiiWS1J/BlwB9AbmAAcDHwLHxDc0ERFpabG0FK4ADgS+dvejgSHA2rhGJSIiCRFLUih393IAM0tz98XAXvENS0REEiGWq4+KzKw98CLwlpltAFY28h4REWmFYrn66OTwy9+a2XQgF3g9rlGJiEhCNJgUzCwJmOvugwDc/Z0WiUpERBKiwT4Fd68FPjOzvi0Uj4iIJFAsHc09gAVm9raZTdn6E8vKzWykmX1uZoVmdm0D5U4zMzez/FgDFxGR5hdLR/MtTVmxmYWAewgGzisCPjGzKe6+sE65HIKnuX3UlO2IiEjzabSlEO5HWAakhF9/AsyOYd3DgEJ3/8rdK4FngHH1lPtfgvGVymMNWkRE4qPRpGBmFwHPAw+EZ/UiuDy1Mb2AFVHTReF50eseAvRx91diilZEROIqlj6FnwOHAZsB3H0J0DWG91k98zyyMLiy6S7gl42uyGyimRWYWcHatbqZWkQkXmJJChXh0z8AmFkyUQf3BhQBfaKme7PtTW85wCBghpktIxhTaUp9nc3u/qC757t7fpcuXWLYtIiINEUsSeEdM7seyDCz44DngJdjeN8nwAAz629mqcB4IHLVkrtvcvfO7p7n7nnATGCsuxfsdC1ERKRZxJIUriUYAG8ecDEwFfhNY28KP77zMuANYBEwyd0XmNmtZja26SGLiEi8mHvDZ4LM7GRgqrvvEo/ezM/P94ICNSZERHaGmc1y90bvBYulpTAW+MLMnjCzMeE+BRERaYNiuU/hp8AeBH0JZwNfmtnD8Q5MRERaXkzf+t29ysxeI7jqKIPgJrSfxTMwERFpebHcvDbSzB4DCoHTgIcJxkMSEZE2JpaWwgUEQ1RcvKt0NouISHzE8pCd8dHTZnYYcLa7/zxuUYmISELE1KdgZoMJOpnPAJYCL8QzKBERSYwdJgUz25PgLuSzgHXAswT3NRzdQrGJiEgLa6ilsBh4DzjR3QsBzOyqFolKREQSoqGrj04FVgPTzewhMzuW+kc+FRGRNmKHScHd/+XuZwIDgRnAVUA3M7vPzH7cQvGJiEgLiuWO5hJ3f8rdTyAY/noOwSB5IiLSxsQy9lGEu6939wfc/Zh4BSQiIomzU0lBRETaNiUFERGJUFIQEZEIJQUREYlQUhARkQglBRERiVBSEBGRCCUFERGJUFIQEZEIJQUREYlQUhARkQglBRERiVBSEBGRCCUFERGJUFIQEZEIJQUREYlQUhARkQglBRERiYhrUjCzkWb2uZkVmtl2z3U2s/9nZgvNbK6ZvW1m/eIZj4iINCxuScHMQsA9wChgH+AsM9unTrFPgXx33w94HvhTvOIREZHGxbOlMAwodPev3L0SeAYYF13A3ae7e2l4cibQO47xiIhII+KZFHoBK6Kmi8LzdmQC8Fp9C8xsopkVmFnB2rVrmzFEERGJFs+kYPXM83oLmp0L5AO317fc3R9093x3z+/SpUszhigiItGS47juIqBP1HRvYGXdQmY2ArgBOMrdK5qyoaqqKoqKiigvL29SoK1Feno6vXv3JiUlJdGhiEgbFc+k8AkwwMz6A98A44GzowuY2RDgAWCku69p6oaKiorIyckhLy8Ps/oaKK2fu7Nu3TqKioro379/osMRkTYqbqeP3L0auAx4A1gETHL3BWZ2q5mNDRe7HcgGnjOzOWY2pSnbKi8vp1OnTm02IQCYGZ06dWrzrSERSax4thRw96nA1Drzbop6PaK5ttWWE8JWP4Q6ikhi6Y7mZrBx40buvffenX7f6NGj2bhxYxwiEhFpGiWFZrCjpFBTU9Pg+6ZOnUr79u3jFZaIyE6L6+mjH4prr72WL7/8ksGDB5OSkkJ2djY9evRgzpw5LFy4kJNOOokVK1ZQXl7OFVdcwcSJEwHIy8ujoKCA4uJiRo0axeGHH85//vMfevXqxUsvvURGRkaCayYiPzRtLinc8vICFq7c3Kzr3KdnO24+8Uc7XH7bbbcxf/585syZw4wZMxgzZgzz58+PXCX06KOP0rFjR8rKyjjwwAM59dRT6dSp0zbrWLJkCU8//TQPPfQQZ5xxBpMnT+bcc89t1nqIiDSmzSWFXcGwYcO2uWz07rvv5l//+hcAK1asYMmSJdslhf79+zN48GAAhg4dyrJly1osXhGRrdpcUmjoG31LycrKiryeMWMG06ZN48MPPyQzM5Phw4fXe1lpWlpa5HUoFKKsrKxFYhURiaaO5maQk5PDli1b6l22adMmOnToQGZmJosXL2bmzJktHJ2ISOzaXEshETp16sRhhx3GoEGDyMjIoFu3bpFlI0eO5P7772e//fZjr7324uCDD05gpCIiDTP3eseo22Xl5+d7QUHBNvMWLVrE3nvvnaCIWtYPqa4i0ny1tIFyAAALI0lEQVTMbJa75zdWTqePREQkQklBREQilBRERCRCSUFERCKUFEREJEJJQUREIpQUmkFTh84G+POf/0xpaWkzRyQi0jRKCs1ASUFE2grd0dwMoofOPu644+jatSuTJk2ioqKCk08+mVtuuYWSkhLOOOMMioqKqKmp4cYbb+Tbb79l5cqVHH300XTu3Jnp06cnuioi8gPX9pLCa9fC6nnNu87u+8Ko23a4OHro7DfffJPnn3+ejz/+GHdn7NixvPvuu6xdu5aePXvy6quvAsGYSLm5udx5551Mnz6dzp07N2/MIiJNoNNHzezNN9/kzTffZMiQIRxwwAEsXryYJUuWsO+++zJt2jR+/etf895775Gbm5voUEVEttP2WgoNfKNvCe7Oddddx8UXX7zdslmzZjF16lSuu+46fvzjH3PTTTclIEIRkR1TS6EZRA+dffzxx/Poo49SXFwMwDfffMOaNWtYuXIlmZmZnHvuufzqV79i9uzZ271XRCTR2l5LIQGih84eNWoUZ599NocccggA2dnZPPnkkxQWFnL11VeTlJRESkoK9913HwATJ05k1KhR9OjRQx3NIpJwGjq7lfkh1VVEmo+GzhYRkZ2mpCAiIhFKCiIiEtFmkkJr6xtpih9CHUUksdpEUkhPT2fdunVt+qDp7qxbt4709PREhyIibVibuCS1d+/eFBUVsXbt2kSHElfp6en07t070WGISBsW16RgZiOBvwAh4GF3v63O8jTgcWAosA44092X7ex2UlJS6N+//38fsIjID1zcTh+ZWQi4BxgF7AOcZWb71Ck2Adjg7nsAdwF/jFc8IiLSuHj2KQwDCt39K3evBJ4BxtUpMw74R/j188CxZmZxjElERBoQz6TQC1gRNV0UnldvGXevBjYBneIYk4iINCCefQr1feOve3lQLGUws4nAxPBksZl9HsP2OwPfxVCuNVBddl1tqT6qy66puerSL5ZC8UwKRUCfqOnewModlCkys2QgF1hfd0Xu/iDw4M5s3MwKYhnnozVQXXZdbak+qsuuqaXrEs/TR58AA8ysv5mlAuOBKXXKTAF+En59GvBvb8s3G4iI7OLi1lJw92ozuwx4g+CS1EfdfYGZ3QoUuPsU4BHgCTMrJGghjI9XPCIi0ri43qfg7lOBqXXm3RT1uhw4PU6b36nTTbs41WXX1Zbqo7rsmlq0Lq3ueQoiIhI/bWLsIxERaR5tMimY2Ugz+9zMCs3s2kTHszPMrI+ZTTezRWa2wMyuCM/vaGZvmdmS8O8OiY41VmYWMrNPzeyV8HR/M/soXJdnwxci7PLMrL2ZPW9mi8P755DWul/M7Krw52u+mT1tZumtab+Y2aNmtsbM5kfNq3dfWODu8PFgrpkdkLjIt7eDutwe/pzNNbN/mVn7qGXXhevyuZkd39zxtLmkEOPwGruyauCX7r43cDDw83D81wJvu/sA4O3wdGtxBbAoavqPwF3humwgGO6kNfgL8Lq7DwT2J6hTq9svZtYL+AWQ7+6DCC4EGU/r2i+PASPrzNvRvhgFDAj/TATua6EYY/UY29flLWCQu+8HfAFcBxA+FowHfhR+z73hY16zaXNJgdiG19hlufsqd58dfr2F4MDTi22HBPkHcFJiItw5ZtYbGAM8HJ424BiCYU2gldTFzNoBRxJcMYe7V7r7RlrpfiG4yCQjfH9QJrCKVrRf3P1dtr+naUf7YhzwuAdmAu3NrEfLRNq4+uri7m+GR3kAmElwnxcEdXnG3SvcfSlQSHDMazZtMSnEMrxGq2BmecAQ4COgm7uvgiBxAF0TF9lO+TNwDVAbnu4EbIz6wLeW/bMbsBb4e/hU2MNmlkUr3C/u/g1wB7CcIBlsAmbROvdLtB3ti9Z+TLgQeC38Ou51aYtJIaahM3Z1ZpYNTAaudPfNiY6nKczsBGCNu8+Knl1P0dawf5KBA4D73H0IUEIrOFVUn/C59nFAf6AnkEVwiqWu1rBfYtFaP3OY2Q0Ep5Sf2jqrnmLNWpe2mBRiGV5jl2ZmKQQJ4Sl3fyE8+9utTd7w7zWJim8nHAaMNbNlBKfxjiFoObQPn7aA1rN/ioAid/8oPP08QZJojftlBLDU3de6exXwAnAorXO/RNvRvmiVxwQz+wlwAnBO1EgPca9LW0wKsQyvscsKn3N/BFjk7ndGLYoeEuQnwEstHdvOcvfr3L23u+cR7Id/u/s5wHSCYU2g9dRlNbDCzPYKzzoWWEgr3C8Ep40ONrPM8Odta11a3X6pY0f7YgpwfvgqpIOBTVtPM+2qLHhA2a+Bse5eGrVoCjDezNLMrD9B5/nHzbpxd29zP8Bogh77L4EbEh3PTsZ+OEFzcC4wJ/wzmuBc/NvAkvDvjomOdSfrNRx4Jfx6t/AHuRB4DkhLdHwx1mEwUBDeNy8CHVrrfgFuARYD84EngLTWtF+Apwn6Q6oIvj1P2NG+IDjlck/4eDCP4KqrhNehkboUEvQdbD0G3B9V/oZwXT4HRjV3PLqjWUREItri6SMREWkiJQUREYlQUhARkQglBRERiVBSEBGRCCUF2aWZmZvZ/0VN/8rMfttM637MzE5rvOR/vZ3Tw6OqTq8zPy96ZEyRXYGSguzqKoBTzKxzogOJtpMjU04ALnX3o+MVT2Oi7lQWaZCSguzqqgkeR3hV3QV1v+mbWXH493Aze8fMJpnZF2Z2m5mdY2Yfm9k8M9s9ajUjzOy9cLkTwu8Phcez/yQ8nv3FUeudbmb/JLgJqm48Z4XXP9/M/hiedxPBDYn3m9ntsVTYzC4Kb/szM5scvvM4x8yWhodAwczamdkyM0sxs93N7HUzmxWuy8Cov8+d4RbKH83sKDObE/751MxyYolHflj07UFag3uAuWb2p514z/7A3gRDEn8FPOzuwyx4aNHlwJXhcnnAUcDuwHQz2wM4n2AohAPNLA34wMzeDJcfRjDO/dLojZlZT4LnEQwleBbBm2Z2krvfambHAL9y94IYY3/B3R8Kr/d3wAR3/6uZzSAYhvxFgmFDJrt7lZk9CFzi7kvM7CDgXoJxpgD2BEa4e42ZvQz83N0/CA+4WB5jPPIDopaC7PI8GCX2cYIHw8TqEw+eTVFBMCTA1oP6PIJEsNUkd6919yUEyWMg8GOCsXLmEAxb3olgjBmAj+smhLADgRkeDDK3dVTLI3ci3miDwt/45wHnEDxQBYJnUvw0/PqnBMN4ZxMMZvdcON4HgOhnBTzn7jXh1x8Ad5rZL4D2/v0w2SIRailIa/FnYDbw96h51YS/2IQHdot+fGRF1OvaqOlatv3c1x3nxQnGyrnc3d+IXmBmwwmGzK5PfUMaN9VjwEnu/pmZXUAwbhThb/h5ZnYUEHL3+RY8/Gejuw/ewboi8br7bWb2KsFYWjPNbIS7L27GuKUNUEtBWgV3Xw9MYttHRC4jOF0DwfMBUpqw6tPNLCncz7AbwSBjbwD/E3X+fk8LHqjTkI+Ao8ysc7gT+izgnSbEA5ADrApv/5w6yx4nGEDt7xBpRS01s9PDsZqZ7V/fSs1sd3ef5+5/JBjYb2AT45M2TElBWpP/A6KvQnqI4ED8MXAQO/4W35DPCQ7erxGcly8nOE2zEJgdvmT0ARppVXswFPN1BMNPfwbMdvdYhp7ey8yKon5OB24kSDJvEYxkGu0pgtFZn46adw4wwcw+Axaw48fPXhnuBP8MKOP7p3mJRGiUVJFWJHy11Th3Py/RsUjbpD4FkVbCzP5K8NjM0YmORdoutRRERCRCfQoiIhKhpCAiIhFKCiIiEqGkICIiEUoKIiISoaQgIiIR/x8DMNlFTbqB9wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "layers = np.arange(2,max_layers+2,1)\n",
    "ax.plot(layers, train_results_per_layer, label='train')\n",
    "ax.plot(layers, test_results_per_layer/test_set_size, label='test')\n",
    "ax.set_ylim(0,1.1)\n",
    "ax.set_xlabel('Number of Layers')\n",
    "ax.set_ylabel('Average Accuracy')\n",
    "ax.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning rate 1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### h = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of layers: 2\n",
      "Max training accuracy: 0.9993749916553497\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 11.913266355000815\n",
      "Number of layers: 3\n",
      "Max training accuracy: 0.9874999940395355\n",
      "Max test accuracy: 0.985\n",
      "Time elapsed: 16.18582828900071\n",
      "Number of layers: 4\n",
      "Max training accuracy: 0.9991666555404664\n",
      "Max test accuracy: 0.9991666666666668\n",
      "Time elapsed: 19.67259904799903\n",
      "Number of layers: 5\n",
      "Max training accuracy: 0.9845833241939544\n",
      "Max test accuracy: 0.9891666666666666\n",
      "Time elapsed: 23.482667047999712\n",
      "Number of layers: 6\n",
      "Max training accuracy: 0.9931249916553497\n",
      "Max test accuracy: 0.9941666666666666\n",
      "Time elapsed: 27.457997482999417\n",
      "Number of layers: 7\n",
      "Max training accuracy: 0.9629166603088379\n",
      "Max test accuracy: 0.9600000000000001\n",
      "Time elapsed: 31.615032961999532\n",
      "Number of layers: 8\n",
      "Max training accuracy: 0.9752083241939544\n",
      "Max test accuracy: 0.9791666666666666\n",
      "Time elapsed: 36.841819204000785\n",
      "Number of layers: 9\n",
      "Max training accuracy: 0.9118749976158143\n",
      "Max test accuracy: 0.9058333333333334\n",
      "Time elapsed: 42.41540071700001\n",
      "Number of layers: 10\n",
      "Max training accuracy: 0.9247916579246521\n",
      "Max test accuracy: 0.9191666666666667\n",
      "Time elapsed: 43.24260067199975\n",
      "Number of layers: 11\n",
      "Max training accuracy: 0.9285416662693023\n",
      "Max test accuracy: 0.9275\n",
      "Time elapsed: 45.68807677099903\n",
      "Number of layers: 12\n",
      "Max training accuracy: 0.8708333253860474\n",
      "Max test accuracy: 0.86\n",
      "Time elapsed: 51.82803838200016\n",
      "Number of layers: 13\n",
      "Max training accuracy: 0.893958330154419\n",
      "Max test accuracy: 0.88\n",
      "Time elapsed: 53.4914969929996\n",
      "Number of layers: 14\n",
      "Max training accuracy: 0.8108333349227905\n",
      "Max test accuracy: 0.8141666666666667\n",
      "Time elapsed: 58.42741832699903\n",
      "Number of layers: 15\n",
      "Max training accuracy: 0.7739583253860474\n",
      "Max test accuracy: 0.7658333333333334\n",
      "Time elapsed: 63.22256635500162\n",
      "Number of layers: 16\n",
      "Max training accuracy: 0.737291669845581\n",
      "Max test accuracy: 0.7341666666666666\n",
      "Time elapsed: 62.29595920900101\n",
      "Number of layers: 17\n",
      "Max training accuracy: 0.5416666626930237\n",
      "Max test accuracy: 0.5375\n",
      "Time elapsed: 70.66713925600016\n",
      "Number of layers: 18\n",
      "Max training accuracy: 0.6527083396911622\n",
      "Max test accuracy: 0.6458333333333334\n",
      "Time elapsed: 68.03598754400082\n",
      "Number of layers: 19\n",
      "Max training accuracy: 0.5441666662693023\n",
      "Max test accuracy: 0.5475\n",
      "Time elapsed: 77.41455920699991\n",
      "Number of layers: 20\n",
      "Max training accuracy: 0.5772916674613953\n",
      "Max test accuracy: 0.5733333333333334\n",
      "Time elapsed: 77.59057415999996\n",
      "Number of layers: 21\n",
      "Max training accuracy: 0.5041666626930237\n",
      "Max test accuracy: 0.5\n",
      "Time elapsed: 83.28275053099969\n"
     ]
    }
   ],
   "source": [
    "from Networks.ResNet import ResAntiSymNet\n",
    "import numpy as np\n",
    "import timeit\n",
    "\n",
    "num_epochs = 100\n",
    "num_features = 2\n",
    "num_classes = 2\n",
    "size_hidden = 8\n",
    "gamma = 0.3\n",
    "h = 1\n",
    "lr = 1.0\n",
    "\n",
    "max_layers = 20\n",
    "train_results_per_layer = np.zeros(max_layers)\n",
    "test_results_per_layer = np.zeros(max_layers)\n",
    "basic_save_key = 'AntiSym_Comparison_results//AntiSymNet'\n",
    "\n",
    "for num_hidden_layers in range(1,max_layers+1):\n",
    "    tic = timeit.default_timer()\n",
    "    num_layers = num_hidden_layers + 1\n",
    "    print('Number of layers: '+str(num_layers))\n",
    "    train_results_per_epoch = np.zeros(num_epochs)\n",
    "    test_results_per_epoch = np.zeros(num_epochs)\n",
    "    for i in range(10):\n",
    "        net = ResAntiSymNet(features=num_features, classes=num_classes, num_layers=num_layers, gamma=gamma, h=h, bias=True, hidden_size=size_hidden)\n",
    "        net.set_test_tracking(True)\n",
    "        net.train(num_epochs, train_loader, test_loader, print_output=False)\n",
    "        train_results_per_epoch += net.avg_correct_pred.numpy()\n",
    "        test_results_per_epoch += net.test_results.numpy()\n",
    "        del net\n",
    "        \n",
    "    train_results_per_epoch = train_results_per_epoch / 10\n",
    "    test_results_per_epoch = test_results_per_epoch / 10\n",
    "    train_save_key = basic_save_key+'_train_per_epoch_layers_'+str(num_layers)+'_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "    test_save_key = basic_save_key+'_test_per_epoch_layers_'+str(num_layers)+'_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "    \n",
    "    np.savetxt(train_save_key, train_results_per_epoch, delimiter=',')\n",
    "    np.savetxt(test_save_key, test_results_per_epoch, delimiter=',')\n",
    "    \n",
    "    train_results_per_layer[num_hidden_layers-1] = train_results_per_epoch.max()\n",
    "    print('Max training accuracy: '+str(train_results_per_epoch.max()))\n",
    "    test_results_per_layer[num_hidden_layers-1] = test_results_per_epoch.max()\n",
    "    print('Max test accuracy: '+str(test_results_per_epoch.max()/test_set_size))\n",
    "    \n",
    "    toc = timeit.default_timer()\n",
    "    print('Time elapsed: '+str(toc-tic))\n",
    "\n",
    "train_save_key = basic_save_key+'_train_per_layer_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "test_save_key = basic_save_key+'_test_per_layer_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "np.savetxt(train_save_key, train_results_per_layer, delimiter=',')\n",
    "np.savetxt(test_save_key, test_results_per_layer, delimiter=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x2400aa84400>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VOX1x/HPyZ5AEkjY14ACiqggCCooIC7ggkuVirjUjfbnrtUWtbXaVdtqW1t3pdZd1LqjohbcURZZZd/DGnYSyH5+f8wwxphlAplMSL7v12temTv3ufeeDMOcPPe59zzm7oiIiADERDsAERGpP5QUREQkRElBRERClBRERCRESUFEREKUFEREJERJQUREQpQUREQkRElBRERC4qIdQE21aNHCs7Kyoh2GiMgBZcaMGZvdvWV17Q64pJCVlcX06dOjHYaIyAHFzFaF006nj0REJERJQUREQpQUREQk5IAbUxAR2RdFRUVkZ2eTn58f7VAiKikpiQ4dOhAfH79P2yspiEijkJ2dTWpqKllZWZhZtMOJCHdny5YtZGdn06VLl33ah04fiUijkJ+fT2ZmZoNNCABmRmZm5n71hpQURKTRaMgJYa/9/R2VFEREJERJQUSkDmzfvp2HHnqoxtuddtppbN++PQIRVUxJQUSkDlSWFEpKSqrcbuLEiTRr1ixSYf2Arj4SEakD48aNY9myZfTu3Zv4+HiaNm1K27ZtmTVrFt9++y1nn302a9asIT8/nxtuuIGxY8cC35X2yc3NZcSIEQwaNIgvvviC9u3b88Ybb5CcnFyrcSopiEijc/db8/l23c5a3WfPdmn85szDKl1/zz33MG/ePGbNmsWUKVM4/fTTmTdvXujS0fHjx5ORkcGePXs4+uij+dGPfkRmZub39rFkyRJeeOEFHn/8cUaNGsWrr77KRRddVKu/h5KCiEgU9O/f/3v3EjzwwAO89tprAKxZs4YlS5b8ICl06dKF3r17A9C3b19WrlxZ63EpKYhIo1PVX/R1pUmTJqHnU6ZM4cMPP+TLL78kJSWFIUOGVHivQWJiYuh5bGwse/bsqfW4IjbQbGbjzWyTmc2rZL2Z2QNmttTM5pjZUZGKRUQk2lJTU9m1a1eF63bs2EHz5s1JSUlh4cKFTJ06tY6j+04kewpPAf8Cnq5k/QigW/AxAHg4+FNEpMHJzMxk4MCB9OrVi+TkZFq3bh1aN3z4cB555BGOOOIIevTowTHHHBO1OM3dI7dzsyzgbXfvVcG6R4Ep7v5CcHkRMMTd11e1z379+rkm2RGRmlqwYAGHHnpotMOoExX9rmY2w937VbdtNMcU2gNryixnB1+rMinsqwUzppDz7SfEtz2M5l1607FDZ5okRv7X35GXz/rl89m5eg6+cx3prTvTplN3mrU7GJKbQyO47V5EDhzRTAoVfRtW2G0xs7HAWIBOnTrt08F2zpvECSsehGXAZ5DjacyP6czmlIPY07wHMW0OI63j4XRu25KOGSkkxsWGve9d+UWszMlj3dqV5GfPwTZ9S+rOxbTJX0EXX8MhVvRd40XfPc23JHYltaMkrQMJmVmktT2IuOadoFlnaNYRmrSsWdJwh8JcyN8B+Tshfweev53dO7aya8cWikuh7XEXEJvaKvx9ikij0qhOH+3euo6cpd+Qmz0X2/QtTbcvolX+CpIoAKDUjTXekkXekXWJXdmV3h1reShN2x9CVqt0MpoksHrrbtZt2Ejh+vnEb15I89wldC5ZSXfLprnlho61LSaDnJSD2NP8EGLa9CSt85GkZHZk7ZoV5KxZwu6cFbB9NSm719OOTbS3zTSzvO/FWxKbhKd1IDajM9asUyBJFOYFv/S3U7pnB8W7t1Gav5OYgh3EFe4ihtIq34NC4ljS4iSaDb6G9r2OV09FGg2dPqr/p4/eBK41sxcJDDDvqC4h7K+UjHZ07t8O+p/+3YulpbBtBXnZ89i5ahZxG+bTd9sihu35htgtpbAFChbEsczbk+Pp9I5Zxxm2ObR5fkwK25sdTG7maRS0P5zmnXuT2L4XzVMyaF5BDC3bdoL+g0PLxSWlrNicxycbdrF8zTq2rFtG/qYVpOxZR/vizXTYnEPnbSvpGDOdtNId7LEUdtGEnZ7M1tJkdnoKO+nELk9mJ03YbU2w5GbEpjQjqWlzUtIyaNosk2bNWxKzezOl08dzTM77pL76Hkve6EbOoZdw2CmXkZ6aGsF3XkQOFBHrKZjZC8AQoAWwEfgNEA/g7o9YoL7rv4DhwG7gMnevtgtQZwPNRfmweTG+cT57sudSuH4+lpcDLbqR3KEXCW0Ph9Y9Ib1jRP7a3rG7iIUbdrJwwy4WbtjJgvW7WL8tj8zUZNqkJ9E6LYk2aUm0SU8MPE8PLKcnx1dbOjdn82YWTnqCjkufJat0Dds8lWkZZ9L0+LH0P/JI4mJr+UrlvM2weirkbYIjL4T4pNrdv0gY1FMIr6cQ0dNHkaCrj2qPl5ayYvr75H/xCD22fwwOn8b0Y333i+k79By6t0nbh506bFtJycov2b30U2Kyp9Jk5/LQ6vx2x5J08YuQXHcFvkRASeFAOH0kUWYxMXTtPwL6j6BwyyrWfPAQfRe/QOqi61m64B4eTR1J6oCLGdG3O82bJPxge3dnW24+6xfPoGD55ySt/4q2O2bRvGQLsUCpp/B1aQ+mlV7AwoTDaFu6kd+ue5hdj5xC6hVvQFrbuv+lRaJk+/btPP/881x99dU13vbvf/87Y8eOJSUlJQKRfZ96CvJ9Rfnsmvkyez5/hFY755HrSbxWegLLskbT88ij2bpjJyWrp5O+eQZZebM5wheRZoFb7dd7Bgvie7GhWW/2tBtAasfDOahVKl1bNKV5kwTWbN3No/9+nNt2/oHipAyaXvkWsS27RfkXlsYi2j2FlStXcsYZZzBvXoVFHqq0t1JqixYtwmqvnoLUnvgkUgdcTOqAiyF7BsUfP8TopW8Qt2YSS1a1p5NtJNGKAdiYmMXazNPI7nQsaT1OoG2nbrStYjyiY0YKv7r+Wh59oQ1jlt1C3sPD4MIJpB0cvbs3RepK2dLZJ598Mq1atWLChAkUFBRwzjnncPfdd5OXl8eoUaPIzs6mpKSEX//612zcuJF169YxdOhQWrRoweTJkyMap5KCVK5DX5qNeRLy7qVk+lO0WzIFa38OdBkIHQfQukkmravfy/ckxcdy/cWjeHtKO3pPuZzMZ89ixfDH6XLMyIj8CiIVenccbJhbu/tscziMuKfS1WVLZ0+aNIlXXnmFr7/+Gndn5MiRfPLJJ+Tk5NCuXTveeecdIFATKT09nfvvv5/JkyeH3VPYH5p5TarXpAWxg2+hyZVvkzDiD3DIadAks/rtKmFmnDn0eHZe+A7Z1oYO7/6Eqa/VfJpCkQPVpEmTmDRpEn369OGoo45i4cKFLFmyhMMPP5wPP/yQX/7yl3z66aekp6fXeWzqKUjUHNajO9uu+4glj57LMbNv451N2Qy7/HckxYd/N7nIPqniL/q64O7cdttt/PSnP/3BuhkzZjBx4kRuu+02TjnlFO688846jU09BYmq5hkt6PHzSSzMOJHT1z/IxPuvIntrbvUbihxgypbOPvXUUxk/fjy5uYHP+tq1a9m0aRPr1q0jJSWFiy66iFtuuYWZM2f+YNtIU09Boi42IYlDrn2F1c9dx7nLnuPtB3JYOfpRBvVoF+3QRGpN2dLZI0aM4MILL+TYY48FoGnTpjz77LMsXbqUW2+9lZiYGOLj43n44YcBGDt2LCNGjKBt27YRH2jWJalSf7iz9b0/kPHVX5hcciSLBz/IVSf2IiYmcvWZ3J38olLyCovZXVDC7qJi8gpK2F0Y/FlQQOmuHCx3PfFexOBhZ5Ce8sN7NqT+i/YlqXVJl6RKw2BGxohfUZDZnsETb6b5J5dy8+q/cPfoE0hPjq/RrkpKnTVbd7N0Uy7LcnJZuimXVVt3k1dQzO7CEvLyi4gr2klqUQ6t2EZr20brvT9tG61tKwfbdlqynTj7rsjgB98MovmFj9PvYPVipGFSUpB6J7H/ZXhqS3q9fDnXrbqWsQ/cxd2XjuCQCspu5BeVsDwnj6XBL/5lwSSwfHMeFBeQZRvoZms5MnEDZyVupJVvJaN0C+klm4m3Qij3R39hQjqFya0padKe0qZ9yU1rR2x6W+KbtWPr8m8Y9s3fmfv06TzS/59cMeI44mu7TpRIlCkpSL1kh55B3KVv0Pm5UfxzzziufHA75592KglxMSzdFEgAS3Nyyd62hwQvpKutp1vMWvonb+TyhPVkNV1D84JsYrwkuMcYSO4EaR0g7RBIbQOpbcs82kBqGxLik8vniZB2h5/Fnq5Hcch/x9Jq2sWMW/obrr/4fDpnNqlkC6lv3L3agpEHuv0dEtCYgtRvG7+l5Jlzyc/byU8Lrme7N+HQuHX0S9nEoXHr6Fiyhmb5a7G980hYLGQeBC17QMtDgo8ekHkwxCfXTkwb5rL7P+dje7Zwe+nVHDfySs7r26HBf9kc6FasWEFqaiqZmZkN9t/K3dmyZQu7du2iS5cu31unKqnScGxfgz9zDrZlyXevxcQHvuhDX/7Bn5kHQ1wdDATnbqLgudEkrp/O/UXnsazn1fzxnCNIT6nZ2IfUnaKiIrKzs8nPz492KBGVlJREhw4diI///mdRSUEalt1bYe4rkNoaWh4KGV0gNspfwEX5lL51PTFzXuLtkmP5a/L13HPBAI7puu93e4tEipKCSF1wh8//jn94N4tiDuLSPTfxo8H9uOnk7rU2CL1m626mLM5hRU4eV53QhbbptXQaTBoVJQWRurTwHfzVq9jpyVyUdyPWvg//uKAPXVrUfBC6oLiEr1dsZcqiHKYs2sSynMDc3WaQldmEl8YeQ6s0zV4nNaOkIFLXNsyDFy6gJDeH20r/j7eKj+GukT0Z1a9jtQObe3sDHy/axOdLt7CnqISE2BgGdM1gcPeWDOnRih17Crnkya9pk57Ei2OPpWVqYh39YtIQKCmIREPuJnjpIljzFa+kXswtOcMZflhb/nTu4d+bva6y3kDHjGSGdG/FkB4tObZrc1K2LoRVn8PKz2DbShYfeg1nfZRBp4wUXhh7DBkVzIgnUhElBZFoKS6At26A2S+wrNXJnL12DE2apHHXyJ7k7CpgyqIcvlj2/d7AkB6tGNItg67Fy7G9SWD1F5C/I7DPZp0hLhE2L2b1ETdwyswBdG2ZxvNXDaCZym5IGJQURKLJHT7/B3x4F7tbHM5le27kqy2BcYC9vYGh3ZpxXEo2SWunBnoDq6dCwc7A9hldIWsQdB4EWQMhvQMU5cPbN8Hs58npcAqnrBhNx7atePbKAaQl6VJYqZqSgkh9sHAivHolnpjGl/3up1NGMu23z8RWfQarv4KiwGkjMrsFkkDWIOg8ENLaVrw/d5j6MEy6g9y0gxm5+WrS23fnmSsG0DRRBQqkckoKIvVFcACaHWu+e63loYEeQOfgI7WGE5sumwwv/4TCUrgi7xryOw7iP5f3JyVBiUEqpqQgUp/k5sA3zwRKcHQeCE1qYa7drcvhhQsp3byY3xddyIKOYxh/WX+SEzRznfyQkoJIY1CwC177GSx8m5dLTuDdzr/goUuP05Sm8gPhJgXV/RU5kCWmwqhnYMhtnB/7CdetvpHb/zOJwuLS6rcVqYCSgsiBLiYGhoyDHz9Lr/i1/HLNz7hv/LMUlSgxSM0pKYg0FIeeSfzYj0hJacLNa2/ixcfuoViJQWpISUGkIWl9GKnXfcbmjKO4eOO9fP6vKykpLop2VHIAUVIQaWhSMmh/7bvMan8hg7e9yrK/nUpp7pZoRyUHCCUFkYYoNo7eVz3MpG530jl3NtseGIRvnB/tqOQAENGkYGbDzWyRmS01s3EVrO9kZpPN7Bszm2Nmp0UyHpHG5uQLb2ZCr0coLthN4aPDKP3yYSgpjnZYUo9FLCmYWSzwIDAC6AmMNrOe5Zr9Cpjg7n2AC4CHIhWPSGNkZlx03nm82Ptpvio6mJj3x7Htb8dQsOzTaIcm9VQkewr9gaXuvtzdC4EXgbPKtXEgLfg8HVgXwXhEGiUz4/qzTyD3/An8KfUO8nZtI/GZM1jw4I/ZumF1tMOTeiaSSaE9UKbYC9nB18q6C7jIzLKBicB1EYxHpNEyM047oh3jbr6VdWM+5q30MXTd9CGJDx/Nu4/dzoqN26MdotQTkUwKFU01Vb6mxmjgKXfvAJwGPGNmP4jJzMaa2XQzm56TkxOBUEUaBzOjf/cOnHnTQ6wfM5nVaX0Yse5Bih48jvsfe4IZq7ZFO0SJskgmhWygY5nlDvzw9NAVwAQAd/8SSAJ+UCnM3R9z937u3q9ly5YRClekccnqfgSH/vw9tp/1NK2SnZvX/Zz1T1zA2H+9wfvzN1BaemDVRZPaEcmkMA3oZmZdzCyBwEDym+XarAaGAZjZoQSSgroCInWoWZ+zaHbLTAqPH8fw+G/4x+armPX8bzj1vg957qtV5BeVRDtEqUMRrZIavMT070AsMN7d/2BmvwWmu/ubwauRHgeaEji19At3n1TVPlUlVSSCtq2k9L3biVn0Dtkx7bkt/2K+Te7HJcdmcfGxnTUn9AFMpbNFZN8t+RB/91Zs63Kmpwzihq3nsyW+Nef37cgVg7qQ1aJJtCOUGqq1pGBmvdx9Xq1Ftp+UFETqSHEBfPFP+OSvlOJ80HwMt60byLbSJE7t2YarTuhC384Z0Y5SwlSbSeEzIAF4Cnje3aN67ZqSgkgd274GJt0B376BxyUzv/mJ/DVnAFPyD+KoTs0Ze0JXTu7ZhtiYii44lPqiVk8fmVk34HLgfOBr4N/u/sF+R7kPlBREomTtTJj5NMx9BQp3sSMli2eLBjN+1zE0zWzLlYO6cF7fjpoOtJ6q9TGFYNmKs4EHgJ0E7kO43d3/uz+B1pSSgkiUFebB/NcDCWLNVEotjqnxA3g0dxDzEvsw5tiuXHxsFi1TE6MdqZRRm6ePjgAuA04HPgCedPeZZtYO+NLdO9dGwOFSUhCpR3IWwcyn8dkvYLu3sDW2FU8XDOJ1hnJMn95ceXwXDm6VGu0ohdpNCp8QuGz0FXffU27dxe7+zH5FWkNKCiL1UHEhLJoYSBDL/ocDn5cewfPFQyjpNpzLB/dgQJcMzDTuEC21mRSaAnvcvSS4HAMkufvuWom0hpQUROq57avhm2cpmfkMsbvWsY00Xi4+ntktzuDXl51Lm/SkaEfYKIWbFMK5o/lDILnMckrwNRGRH2rWCYbeTuxN82DMK6T1OIEr49/nn9uu5sN3Xox2dFKNcJJCkrvn7l0IPk+JXEgi0iDExEK3k4kd/RwxP1/ArrgMOi9+ij2FKptRn4WTFPLM7Ki9C2bWF9hTRXsRke9r2opdh13IQJ/F/778KtrRSBXCSQo3Ai+b2adm9inwEnBtZMMSkYam/bCf4Wbs+fIJDrTyOo1JXHUN3H2amR0C9CBwb8JCdy+KeGQi0qBYegfWthrC0I2TmLNyI0d2aRPtkKQC4ZbO7kFgnuU+BOZaviRyIYlIQ9XixGvItF3M+6BOr2SXGqg2KZjZb4B/Bh9DgT8DIyMcl4g0QCndT2RzYgcOWTuBbXmF0Q5HKhBOT+E8AhPhbHD3y4AjAd2/LiI1FxNDyVGX09cW89GUj6IdjVQgnKSwx91LgWIzSwM2AV0jG5aINFStj7+MQhKI/+bfmvKzHgonKUw3s2YESl3MAGYSqJQqIlJzKRls6HQGJxVN4fNvl0c7GimnyqRggUIlf3L37e7+CHAycGnwNJKIyD5pe9I1NLECVk/+d7RDkXKqTAoeuJj49TLLK919TsSjEpEGLb5TPzY0OZSjc/7Lmi150Q5Hygjn9NFUMzs64pGISKOSeNxVdI9ZyycfvRntUKSMcJLCUOBLM1tmZnPMbK6ZqbcgIvul+dGjyYtpSosFT1NQrHpI9UW1dzQDIyIehYg0PgkpbO9+HkMXPMNH0+Yy4tje0Y5ICK+n4JU8RET2S9sTryHBStj22ZPRDkWCwkkK7wBvB39+BCwH3o1kUCLSOMS06s7a5v0ZnPsO87O3Ruw4izfu4sT7pvD+/A0RO0ZDUW1ScPfD3f2I4M9uQH/gs8iHJiKNQbPBV9PetjDjg5cisv/8ohJ+/tyXnL71Gf464SNW6WqnKoVbEC/E3WcCuhpJRGpFk8PPZEdcC7JWvsjO/NovwPy7t79lzNYH+Xn8Kzxkf+LW5z7XwHYVwimId3OZxy1m9jyQUwexiUhjEBtH/hEXc4LN4oNPv6zVXb87dz15057ngrgp0ON0DrJ1XJlzD/e8822tHqchCaenkFrmkUhgbOGsSAYlIo1L6yFjKSGG4q/H19oEPNnbdvPwq+/yp4TxlHY8FkY9Tcypf+SU2BlkTvsL783T+EJFwplk5+66CEREGrG0dqxvO4yT133A1CXrOLZ7+/3aXXFJKb94YSp/9ftJSE4h5vzxEBsHA35KyYb5XDvraX75SmcOa3cbHTM05XxZ4Zw++iBYEG/vcnMzez+yYYlIY9Nq6NVkWC6LPnp6v/f1wEdLOHPdA3S3NcT+6HFIaxdYYUbsGfeR324Ad/vD/P0/L1JYXLrfx2tIwjl91NLdt+9dcPdtQKvIhSQijVFCt6FsSerMEetfZePO/H3ez5fLtrD646cYHTcZBt0M3U76foO4BJLGPE9pk5bcuv23PPjmp/sZecMSTlIoMbNOexfMrDNh3rxmZsPNbJGZLTWzcZW0GWVm35rZ/OAgtog0RmbY0ZdzVMwSPvzfB/u0i615hdz/4tv8KX48JR2OgaF3VNywSQtSLnmZ5rH5DJ11Ex/NWbUfgTcs4SSFO4DPzOwZM3sG+AS4rbqNzCwWeJBAmYyeBOZ27lmuTbfgvga6+2HAjTWMX0QakIzjLqXAEmky5ymKSmp2WsfduWPC1/yu8K/EJyYTu3ccoTJtehFz7qP0jllG/n+vIXur7l+A8G5eew84CngJmAD0dfdwxhT6A0vdfbm7FwIv8sOrlq4CHgyeksLdN9UkeBFpYJKbsznrTE4p+ZQps5fWaNOnvljJ8cvu4xBbTdx5j0N69YPV8b3OYtuAX3A6n/K/J++ocSJqiMIZaD4HKHL3t939LQLTcp4dxr7bA2vKLGcHXyurO9DdzD43s6lmNrySGMaa2XQzm56To1skRBqyNsOuJcUKWPtx+BPwzFu7gznvPsmFcf/DB94I3U4Oe9vmw29nbYcRXJT7FK+/+MS+hNyghHP66DfuvmPvQnDQ+TdhbGcVvFZ+LCIO6AYMAUYDT5S90qnMMR9z937u3q9ly5ZhHFpEDlSxHfqwMfUwBm57g6Ubd1bbPq+gmD8/9za/j3ucovb9sRN/VbMDmtH+kvGsT+nOiMW/5quvGncVn3CSQkVtwim5nQ10LLPcAVhXQZs33L3I3VcAiwgkCRFpxFIG/pRuMWv57MM3qm37+9dnMi73XhISkogf9W+Ija/5ARNSyLzyFQpjkmn37uVs2LB2H6JuGMJJCtPN7H4zO8jMuprZ34AZYWw3DehmZl3MLAG4ACg/xdLrBCbxwcxaEDidpJm8RRq51L6jyItJpc3i59hdWFxpuzdmreWwuffQM2YV8ec9Bukd9vmYSZmd2HPuf2jlW9n85AUUFxbs874qcqCMV4STFK4DCgkMNL8M5ANXV7eRuxcD1wLvAwuACe4+38x+a2Yjg83eB7aY2bfAZOBWd99S819DRBqU+GR2HTqKYXzN+1NnV9hk1ZY8Pn3tMS6K+4jSY6+D7qfu92HbHz6YeX1/R6+iOcx58v/2e3/uzhfLNnP9c19x7p2P8qe35lBaWr+no7Ga1hkxsyTgTHd/OTIhVa1fv34+ffr0aBxaROqQb16K/asvTyddxMW//Bdm3w1TFhaXcv2DL/PXbTcQ3/YwEq98b99OG1Xi43/9jMGbX2Bxv7vofsZNNd5+x+4iXp+2hOVfvkHvvE8ZFvsNaezm85LDePOQP/O7Hx9HQlyNi1TvFzOb4e79qm0XTlII3nNwCoHB4FOAz9z9vP2Och8oKYg0Huv/ORzfvIj1P/mavl2+u8jkz+/M4vSvLqFb4nYSrvkcmnWsYi81l19QyOy/jOCo4m/Ydd7LZPQaVu027s68ZauZN/klWmRPYhCzSbZCCuKbEdvzdGIzulA65U8sLOnAwx3u5d5LT6ZJYjjDs7Uj3KRQZURmdgJwIXA68DUwEOjq7rtrJUoRkSo0H/wzkv57Ke9+9BJ9r7wWgE8W59Dmy99zWNwqOO/FWk8IAEmJCbT4ybOsfnwYrV69lOI2nxDXomuFbXdvzWb+Ry8Qu/gdDi+cw+FWwo6EluzufiHJ/X5EYqfjQjfRxbbrQ48XL+IXa6/n5kf+wB+uOIsWTRNrPf79UWlPwcyygdXAw8Dr7r7LzFa4e5e6DLA89RREGpGSYnbecwizCtrR8xcf4A73/e0e7im9n+IB1xA34o8RPfz7n3zOgI/OpzilNS1u+BiS0gIrti4nZ9qr7JnzBh1y5xFjTnZMe7Z3Hk6X439Mk6yjIaaS00PZ0yl8+kfsKijltuQ7+dVVY+iUGflKrft9+sjM/gGcDcwFngfeAOa6e8Xpso4oKYg0Llve+S2Z0+7j2f6v883qbdy9/mpi2xxK8thJtTqOUJlH/j2eK1f+nJ3tjyft4GPIm/U66TsXATDfs1jVahidB/2YnocfjVWWCMrbvISCp86iOHcLt8b8gquvuJJe7dMj+FvU0piCBUZ2hhIYSzgNSAOuACa6e24txVojSgoijczO9RTffxjPl5xEH1tE98StJF7zOTTrVP22tWB3YTFP3n871+U/SinG9NLufJ00kIy+5zJ80AAymiTs2453rqfgqbOJ2bqU2/0azrn4eo47uEXtBl9GrQ40B3cYDwwnONjs7pGLvgpKCiKNz/rHR9F2baDkml/wPHbI6XV6/CUbdvLXJ58mtV13zhrUh4EHtSAmpqKiDTW0ZzsFz/6YxLVT+V3JpfQ5fxxnHNFu//dbgVpPCuV2nuzue/Ypsv2kpCDS+JQu/5SYp8+g8Oj/I+H0e6IdTu0qyqdowuWTDUXsAAAQ60lEQVTEL3mHh4pHkjL8bn4yqPbP0oebFPbpQtloJQQRaZxiuh4P10wjIcIDy1ERn0T86Gco7nMpV8e9Scp7N/KXd+fV2lzVNVW3d0+IiOyrlt0rv6LnQBcTS9zIf1B6wi8ZFfcxfb64ljsmfE1xFEpjhP0Om1mTSAYiItKomRFz4u346fczLHYWP5p/DTc9NZk9hSV1GkY48ykcF6xNtCC4fKSZPRTxyEREGiE7+gps1H/oHbuS61ddx3WPvsW2vMI6O344PYW/AacCWwDcfTZwQiSDEhFp1HqeReylr5OVsIPfbb6Jnz/4Imu3181Qblinj9x9TbmX6rY/IyLS2GQNIv7K98hMieVvu8fxm389ycIN1U86tL/CSQprzOw4wM0swcxuIXgqSUREIqjN4SSM/ZDk9FY8WHw3O2a/HfFDhpMUfgZcQ2B+5Wygd3BZREQirXkWCWM/JK7T0QzoeXDED1dt3VZ33wyMiXgkIiJSsSYtiL3sHbBauIu6GtUmBTN7oIKXdwDT3b36CVRFRGT/1UFCgPBOHyUROGW0JPg4AsgArjCzv0cwNhERqWPhTPtzMHBicM5lzOxhYBJwMoGy2iIi0kCE01NoD5S9m7kJ0M7dS4CCiEQlIiJREU5P4c/ALDObAhiBG9f+GCx78WEEYxMRkToWztVHT5rZRKA/gaRwu7uvC66+NZLBiYhI3Qq3IF4+sB7YChxsZipzISLSAIVzSeqVwA1AB2AWcAzwJXBiZEMTEZG6Fk5P4QbgaGCVuw8F+gA5EY1KRESiIpykkO/u+QBmlujuC4EekQ1LRESiIZyrj7LNrBnwOvCBmW0D1lWzjYiIHIDCufronODTu8xsMpAOvBfRqEREJCqqTApmFgPMcfdeAO7+cZ1EJSIiUVHlmIK7lwKzzaxTHcUjIiJRFM5Ac1tgvpl9ZGZv7n2Es3MzG25mi8xsqZmNq6LdeWbmZtYv3MBFRKT2hTPQfPe+7NjMYoEHCRTOywammdmb7v5tuXapwPXAV/tyHBERqT3V9hSC4wgrgfjg82nAzDD23R9Y6u7L3b0QeBE4q4J2vyNQXyk/3KBFRCQyqk0KZnYV8ArwaPCl9gQuT61Oe2BNmeXs4Gtl990H6OjukZ94VEREqhXOmMI1wEBgJ4C7LwFahbFdRdMEeWhl4MqmvwE/r3ZHZmPNbLqZTc/J0c3UIiKREk5SKAie/gHAzOIo8+VehWygY5nlDnz/prdUoBcwxcxWEqip9GZFg83u/pi793P3fi1btgzj0CIisi/CSQofm9ntQLKZnQy8DLwVxnbTgG5m1sXMEoALgNBVS+6+w91buHuWu2cBU4GR7j69xr+FiIjUinCSwjgCBfDmAj8FJgK/qm6j4PSd1wLvAwuACe4+38x+a2Yj9z1kERGJFHOv+kyQmZ0DTHT3ejH1Zr9+/Xz6dHUmRERqwsxmuHu194KF01MYCSw2s2fM7PTgmIKIiDRA4dyncBlwMIGxhAuBZWb2RKQDExGRuhfWX/3uXmRm7xK46iiZwE1oV0YyMBERqXvh3Lw23MyeApYC5wFPEKiHJCIiDUw4PYWfEChR8dP6MtgsIiKREc4kOxeUXTazgcCF7n5NxKISEZGoCGtMwcx6ExhkHgWsAP4byaBERCQ6Kk0KZtadwF3Io4EtwEsE7msYWkexiYhIHauqp7AQ+BQ4092XApjZTXUSlYiIREVVVx/9CNgATDazx81sGBVXPhURkQai0qTg7q+5+4+BQ4ApwE1AazN72MxOqaP4RESkDoVzR3Oeuz/n7mcQKH89i0CRPBERaWDCqX0U4u5b3f1Rdz8xUgGJiEj01CgpiIhIw6akICIiIUoKIiISoqQgIiIhSgoiIhKipCAiIiFKCiIiEqKkICIiIUoKIiISoqQgIiIhSgoiIhKipCAiIiFKCiIiEqKkICIiIUoKIiISoqQgIiIhSgoiIhKipCAiIiERTQpmNtzMFpnZUjP7wbzOZnazmX1rZnPM7CMz6xzJeEREpGoRSwpmFgs8CIwAegKjzaxnuWbfAP3c/QjgFeDPkYpHRESqF8meQn9gqbsvd/dC4EXgrLIN3H2yu+8OLk4FOkQwHhERqUYkk0J7YE2Z5ezga5W5Ani3ohVmNtbMppvZ9JycnFoMUUREyopkUrAKXvMKG5pdBPQD/lLRend/zN37uXu/li1b1mKIIiJSVlwE950NdCyz3AFYV76RmZ0E3AEMdveCCMYjIiLViGRPYRrQzcy6mFkCcAHwZtkGZtYHeBQY6e6bIhiLiIiEIWJJwd2LgWuB94EFwAR3n29mvzWzkcFmfwGaAi+b2Swze7OS3YmISB2I5Okj3H0iMLHca3eWeX5SJI8vIiI1ozuaRUQkRElBRERClBRERCRESUFEREKUFEREJERJQUREQpQUREQkRElBRERClBRERCRESUFEREKUFEREJERJQUREQpQUREQkRElBRERClBRERCRESUFEREKUFEREJERJQUREQpQUREQkRElBRERClBRERCRESUFEREKUFEREJERJQUREQpQUREQkRElBRERClBRERCRESUFEREKUFEREJERJQUREQiKaFMxsuJktMrOlZjaugvWJZvZScP1XZpYVyXhERKRqEUsKZhYLPAiMAHoCo82sZ7lmVwDb3P1g4G/AvZGKR0REqhfJnkJ/YKm7L3f3QuBF4Kxybc4C/hN8/gowzMwsgjGJiEgVIpkU2gNryixnB1+rsI27FwM7gMwIxiQiIlWIi+C+K/qL3/ehDWY2FhgbXMw1s0X7GVtlWgCbI7Tv2qD49o/i23/1PUbFV7nO4TSKZFLIBjqWWe4ArKukTbaZxQHpwNbyO3L3x4DHIhRniJlNd/d+kT7OvlJ8+0fx7b/6HqPi23+RPH00DehmZl3MLAG4AHizXJs3gUuDz88D/ufuP+gpiIhI3YhYT8Hdi83sWuB9IBYY7+7zzey3wHR3fxN4EnjGzJYS6CFcEKl4RESkepE8fYS7TwQmlnvtzjLP84HzIxlDDUX8FNV+Unz7R/Htv/oeo+LbT6azNSIispfKXIiISEijSwpm1tHMJpvZAjObb2Y3VNBmiJntMLNZwcedFe0rgjGuNLO5wWNPr2C9mdkDwfIgc8zsqDqMrUeZ92WWme00sxvLtanz98/MxpvZJjObV+a1DDP7wMyWBH82r2TbS4NtlpjZpRW1iUBsfzGzhcF/v9fMrFkl21b5WYhwjHeZ2doy/46nVbJtleVsIhjfS2ViW2lmsyrZNqLvYWXfKfXl81dj7t6oHkBb4Kjg81RgMdCzXJshwNtRjHEl0KKK9acB7xK4z+MY4KsoxRkLbAA6R/v9A04AjgLmlXntz8C44PNxwL0VbJcBLA/+bB583rwOYjsFiAs+v7ei2ML5LEQ4xruAW8L4DCwDugIJwOzy/58iFV+59fcBd0bjPazsO6W+fP5q+mh0PQV3X+/uM4PPdwEL+OGd1vXdWcDTHjAVaGZmbaMQxzBgmbuvisKxv8fdP+GH97iULaPyH+DsCjY9FfjA3be6+zbgA2B4pGNz90keuIsfYCqB+3iippL3LxzhlLPZb1XFFyyNMwp4obaPG44qvlPqxeevphpdUigrWJW1D/BVBauPNbPZZvaumR1Wp4EF7uqeZGYzgndzlxdOCZG6cAGV/0eM5vu3V2t3Xw+B/7hAqwra1If38nICPb+KVPdZiLRrg6e4xldy+qM+vH/HAxvdfUkl6+vsPSz3nXKgfP6+p9EmBTNrCrwK3OjuO8utnknglMiRwD+B1+s4vIHufhSBCrPXmNkJ5daHVR4kkoI3JI4EXq5gdbTfv5qI6ntpZncAxcBzlTSp7rMQSQ8DBwG9gfUETtGUF/XPIjCaqnsJdfIeVvOdUulmFbwW1UtCG2VSMLN4Av94z7n7f8uvd/ed7p4bfD4RiDezFnUVn7uvC/7cBLxGoIteVjglRCJtBDDT3TeWXxHt96+MjXtPqwV/bqqgTdTey+Cg4hnAGA+eYC4vjM9CxLj7RncvcfdS4PFKjh3Vz6IFyuOcC7xUWZu6eA8r+U6p15+/yjS6pBA8//gksMDd76+kTZtgO8ysP4H3aUsdxdfEzFL3PicwIDmvXLM3gUuCVyEdA+zY202tQ5X+dRbN96+csmVULgXeqKDN+8ApZtY8eHrklOBrEWVmw4FfAiPdfXclbcL5LEQyxrLjVOdUcuxwytlE0knAQnfPrmhlXbyHVXyn1NvPX5WiOcodjQcwiED3bA4wK/g4DfgZ8LNgm2uB+QSupJgKHFeH8XUNHnd2MIY7gq+Xjc8ITGC0DJgL9Kvj9zCFwJd8epnXovr+EUhQ64EiAn99XUGgDPtHwJLgz4xg237AE2W2vRxYGnxcVkexLSVwLnnvZ/CRYNt2wMSqPgt1+P49E/x8zSHwBde2fIzB5dMIXHGzLFIxVhRf8PWn9n7uyrSt0/ewiu+UevH5q+lDdzSLiEhIozt9JCIilVNSEBGRECUFEREJUVIQEZEQJQUREQlRUpB6zczczO4rs3yLmd1VS/t+yszOq419VXOc84MVNCeXez2rbNVPkfpASUHquwLg3CjdEV0pM4utQfMrgKvdfWik4qlO8M5fkWopKUh9V0xgCsObyq8o/5e+meUGfw4xs4/NbIKZLTaze8xsjJl9Hayrf1CZ3ZxkZp8G250R3D7WAvMdTAsWg/tpmf1ONrPnCdzUVT6e0cH9zzOze4Ov3Ung5qZHzOwv4fzCZnZV8NizzexVM0sxs1QzWxEsp4CZpVlgnoB4MzvIzN4LFnz71MwOKfP+3B/sodxrZoPtu/kHvtl7p69IWfrrQQ4EDwJzzOzPNdjmSOBQAuWWlxO4g7S/BSZAuQ7YOzFQFjCYQOG3yWZ2MHAJgdIhR5tZIvC5mU0Ktu8P9HL3FWUPZmbtCMyL0BfYRqAq59nu/lszO5HAvAThTvDyX3d/PLjf3xO4e/efZjYFOJ1AgcELgFfdvcjMHiNwV+8SMxsAPAScGNxXd+Akdy8xs7eAa9z982Dxtvww45FGRD0Fqfc8UHHyaeD6Gmw2zQN17gsIlF/Y+6U+l0Ai2GuCu5d6oOzycuAQAvVnLrHATF5fEShX0C3Y/uvyCSHoaGCKu+d4YJ6E5whMDLMvegX/4p8LjAH2lh5/Args+Pwy4N/BL/fjgJeD8T5KYNKXvV5295Lg88+B+83seqCZfzefg0iIegpyoPg7gZLc/y7zWjHBP2yCRckSyqwrKPO8tMxyKd//3Jev8+IEaktd5+7fK0xmZkOAvEriq6gE8r56Cjjb3Web2U8IzGRH8C/8LDMbDMS6+zwzSwO2u3vvSvYVitfd7zGzdwjU5ZlqZie5+8JajFsaAPUU5IDg7luBCQQGbfdaSeB0DQRmuYrfh12fb2YxwXGGrsAiAlUq/6/M+fvuwQqbVfkKGGxmLYKD0KOBj/chHghM6bg+ePwx5dY9TaA43L8h1ItaYWbnB2M1Mzuyop2a2UHuPtfd7wWmE+gViXyPkoIcSO4Dyl6F9DiBL+KvgQFU/ld8VRYR+PJ+l8B5+XwCp2m+BWYGLxl9lGp61R4oXX4bMJlARc6Z7l5RqeTyephZdpnH+cCvCSSZD4Dyf8k/R2Au37Jly8cAV5jZ3kqglU2HeWNwEHw2sIfKZ3uTRkxVUkUOIMGrrc5y94ujHYs0TBpTEDlAmNk/Ccx4d1q0Y5GGSz0FEREJ0ZiCiIiEKCmIiEiIkoKIiIQoKYiISIiSgoiIhCgpiIhIyP8Dgaqw3czZ/yIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "layers = np.arange(2,max_layers+2,1)\n",
    "ax.plot(layers, train_results_per_layer, label='train')\n",
    "ax.plot(layers, test_results_per_layer/test_set_size, label='test')\n",
    "ax.set_ylim(0,1.1)\n",
    "ax.set_xlabel('Number of Layers')\n",
    "ax.set_ylabel('Average Accuracy')\n",
    "ax.legend(loc='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### h = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of layers: 2\n",
      "Max training accuracy: 0.9883333265781402\n",
      "Max test accuracy: 0.9866666666666667\n",
      "Time elapsed: 12.166836477001198\n",
      "Number of layers: 3\n",
      "Max training accuracy: 0.9893749833106995\n",
      "Max test accuracy: 0.9908333333333333\n",
      "Time elapsed: 16.030359653999767\n",
      "Number of layers: 4\n",
      "Max training accuracy: 0.9895833194255829\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 19.698025818001042\n",
      "Number of layers: 5\n",
      "Max training accuracy: 0.9879166603088378\n",
      "Max test accuracy: 0.985\n",
      "Time elapsed: 23.436378328999126\n",
      "Number of layers: 6\n",
      "Max training accuracy: 0.9897916555404663\n",
      "Max test accuracy: 0.9883333333333333\n",
      "Time elapsed: 26.99687556600111\n",
      "Number of layers: 7\n",
      "Max training accuracy: 0.9899999976158143\n",
      "Max test accuracy: 0.9883333333333333\n",
      "Time elapsed: 32.47332869999809\n",
      "Number of layers: 8\n",
      "Max training accuracy: 0.9991666615009308\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 36.93995924599949\n",
      "Number of layers: 9\n",
      "Max training accuracy: 0.9891666531562805\n",
      "Max test accuracy: 0.9883333333333333\n",
      "Time elapsed: 40.42694333899999\n",
      "Number of layers: 10\n",
      "Max training accuracy: 0.9893749952316284\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 41.471210787996824\n",
      "Number of layers: 11\n",
      "Max training accuracy: 0.9995833277702332\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 46.31709463600055\n",
      "Number of layers: 12\n",
      "Max training accuracy: 0.9997916638851165\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 53.46160461699765\n",
      "Number of layers: 13\n",
      "Max training accuracy: 0.9791666626930237\n",
      "Max test accuracy: 0.9791666666666666\n",
      "Time elapsed: 54.27618243199686\n",
      "Number of layers: 14\n",
      "Max training accuracy: 0.9995833277702332\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 59.093861036002636\n",
      "Number of layers: 15\n",
      "Max training accuracy: 0.9899999916553497\n",
      "Max test accuracy: 0.9883333333333333\n",
      "Time elapsed: 61.7459826489976\n",
      "Number of layers: 16\n",
      "Max training accuracy: 0.9995833277702332\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 65.98572979600067\n",
      "Number of layers: 17\n",
      "Max training accuracy: 0.9895833253860473\n",
      "Max test accuracy: 0.9883333333333333\n",
      "Time elapsed: 66.84099722799874\n",
      "Number of layers: 18\n",
      "Max training accuracy: 0.9897916615009308\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 71.30708595800024\n",
      "Number of layers: 19\n",
      "Max training accuracy: 0.9997916638851165\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 78.03823345499768\n",
      "Number of layers: 20\n",
      "Max training accuracy: 0.9991666555404664\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 77.86284462299955\n",
      "Number of layers: 21\n",
      "Max training accuracy: 0.9993749916553497\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 83.77908993500023\n",
      "Number of layers: 22\n",
      "Max training accuracy: 0.9991666555404664\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 88.38591136000105\n",
      "Number of layers: 23\n",
      "Max training accuracy: 0.9995833277702332\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 86.97699597799874\n",
      "Number of layers: 24\n",
      "Max training accuracy: 0.9997916638851165\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 97.15477722799915\n",
      "Number of layers: 25\n",
      "Max training accuracy: 0.9899999976158143\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 95.56329730200014\n",
      "Number of layers: 26\n",
      "Max training accuracy: 0.9897916555404663\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 101.5924042000006\n",
      "Number of layers: 27\n",
      "Max training accuracy: 0.9989583253860473\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 104.57973219700216\n",
      "Number of layers: 28\n",
      "Max training accuracy: 0.9995833277702332\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 108.52340961200025\n",
      "Number of layers: 29\n",
      "Max training accuracy: 0.9991666555404664\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 111.57090379400324\n",
      "Number of layers: 30\n",
      "Max training accuracy: 0.9889583170413971\n",
      "Max test accuracy: 0.9916666666666667\n",
      "Time elapsed: 116.13184054800149\n",
      "Number of layers: 31\n",
      "Max training accuracy: 0.9995833277702332\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 119.0444977469997\n",
      "Number of layers: 32\n",
      "Max training accuracy: 0.9991666555404664\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 124.67867879299956\n",
      "Number of layers: 33\n",
      "Max training accuracy: 0.9899999856948852\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 128.1330583480012\n",
      "Number of layers: 34\n",
      "Max training accuracy: 0.9997916638851165\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 131.05249594999987\n",
      "Number of layers: 35\n",
      "Max training accuracy: 0.9993749916553497\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 132.53978436399848\n",
      "Number of layers: 36\n",
      "Max training accuracy: 0.9995833277702332\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 134.49119729900121\n",
      "Number of layers: 37\n",
      "Max training accuracy: 0.9993749916553497\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 144.33094536999852\n",
      "Number of layers: 38\n",
      "Max training accuracy: 0.989166647195816\n",
      "Max test accuracy: 0.9883333333333333\n",
      "Time elapsed: 145.0137988150018\n",
      "Number of layers: 39\n",
      "Max training accuracy: 0.9993749916553497\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 146.49002062400177\n",
      "Number of layers: 40\n",
      "Max training accuracy: 0.998541659116745\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 153.75731296600134\n",
      "Number of layers: 41\n",
      "Max training accuracy: 0.9995833277702332\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 153.71234999900116\n",
      "Number of layers: 42\n",
      "Max training accuracy: 0.9993749916553497\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 157.5118772729984\n",
      "Number of layers: 43\n",
      "Max training accuracy: 0.9995833277702332\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 162.3531382120018\n",
      "Number of layers: 44\n",
      "Max training accuracy: 0.9989583253860473\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 165.96891089999917\n",
      "Number of layers: 45\n",
      "Max training accuracy: 0.9991666615009308\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 170.66576466900005\n",
      "Number of layers: 46\n",
      "Max training accuracy: 0.9993749916553497\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 172.78796100600084\n",
      "Number of layers: 47\n",
      "Max training accuracy: 0.9991666615009308\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 176.7939045250023\n",
      "Number of layers: 48\n",
      "Max training accuracy: 0.9991666555404664\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 181.63993487900007\n",
      "Number of layers: 49\n",
      "Max training accuracy: 0.9987499833106994\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 182.24443636799697\n",
      "Number of layers: 50\n",
      "Max training accuracy: 0.978124988079071\n",
      "Max test accuracy: 0.9816666666666667\n",
      "Time elapsed: 187.86190338599772\n",
      "Number of layers: 51\n",
      "Max training accuracy: 0.9991666555404664\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 192.44087066800057\n",
      "Number of layers: 52\n",
      "Max training accuracy: 0.9993749916553497\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 194.75358451499778\n",
      "Number of layers: 53\n",
      "Max training accuracy: 0.9995833277702332\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 199.64679136300037\n",
      "Number of layers: 54\n",
      "Max training accuracy: 0.9902083277702332\n",
      "Max test accuracy: 0.9916666666666667\n",
      "Time elapsed: 203.15508977500212\n",
      "Number of layers: 55\n",
      "Max training accuracy: 0.9981249868869781\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 205.1350770470017\n",
      "Number of layers: 56\n",
      "Max training accuracy: 0.9881249904632569\n",
      "Max test accuracy: 0.9908333333333333\n",
      "Time elapsed: 208.88490586600165\n",
      "Number of layers: 57\n",
      "Max training accuracy: 0.9983333230018616\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 209.09287507599947\n",
      "Number of layers: 58\n",
      "Max training accuracy: 0.9989583253860473\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 215.7646480359981\n",
      "Number of layers: 59\n",
      "Max training accuracy: 0.9993749916553497\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 220.26574349000293\n",
      "Number of layers: 60\n",
      "Max training accuracy: 0.9887499928474426\n",
      "Max test accuracy: 0.9908333333333333\n",
      "Time elapsed: 225.63594528799877\n",
      "Number of layers: 61\n",
      "Max training accuracy: 0.9889583230018616\n",
      "Max test accuracy: 0.9908333333333333\n",
      "Time elapsed: 222.94882877400232\n",
      "Number of layers: 62\n",
      "Max training accuracy: 0.9893749833106995\n",
      "Max test accuracy: 0.9908333333333333\n",
      "Time elapsed: 230.68885505199432\n",
      "Number of layers: 63\n",
      "Max training accuracy: 0.9885416567325592\n",
      "Max test accuracy: 0.9891666666666666\n",
      "Time elapsed: 234.24218737099727\n",
      "Number of layers: 64\n",
      "Max training accuracy: 0.9981249868869781\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 236.78685681299976\n",
      "Number of layers: 65\n",
      "Max training accuracy: 0.9785416603088379\n",
      "Max test accuracy: 0.9816666666666667\n",
      "Time elapsed: 238.47216898899933\n",
      "Number of layers: 66\n",
      "Max training accuracy: 0.9887499868869781\n",
      "Max test accuracy: 0.9916666666666667\n",
      "Time elapsed: 245.192352692\n",
      "Number of layers: 67\n",
      "Max training accuracy: 0.9989583253860473\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 249.0005654619963\n",
      "Number of layers: 68\n",
      "Max training accuracy: 0.9887499868869781\n",
      "Max test accuracy: 0.9916666666666667\n",
      "Time elapsed: 244.81696869200096\n",
      "Number of layers: 69\n",
      "Max training accuracy: 0.9989583194255829\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 257.045027224005\n",
      "Number of layers: 70\n",
      "Max training accuracy: 0.9889583230018616\n",
      "Max test accuracy: 0.9916666666666667\n",
      "Time elapsed: 255.38245895800355\n",
      "Number of layers: 71\n",
      "Max training accuracy: 0.9897916555404663\n",
      "Max test accuracy: 0.9875\n",
      "Time elapsed: 261.4667071399963\n",
      "Number of layers: 72\n",
      "Max training accuracy: 0.9989583253860473\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 266.4783916410015\n",
      "Number of layers: 73\n",
      "Max training accuracy: 0.9797916531562805\n",
      "Max test accuracy: 0.9791666666666666\n",
      "Time elapsed: 265.0321763299944\n",
      "Number of layers: 74\n",
      "Max training accuracy: 0.9885416507720948\n",
      "Max test accuracy: 0.9908333333333333\n",
      "Time elapsed: 272.6227623130035\n",
      "Number of layers: 75\n",
      "Max training accuracy: 0.9289583265781403\n",
      "Max test accuracy: 0.9275\n",
      "Time elapsed: 274.2271409420064\n",
      "Number of layers: 76\n",
      "Max training accuracy: 0.9791666626930237\n",
      "Max test accuracy: 0.9833333333333333\n",
      "Time elapsed: 280.49872154199693\n",
      "Number of layers: 77\n",
      "Max training accuracy: 0.9891666531562805\n",
      "Max test accuracy: 0.9908333333333333\n",
      "Time elapsed: 282.3340276529998\n",
      "Number of layers: 78\n",
      "Max training accuracy: 0.9768749892711639\n",
      "Max test accuracy: 0.9766666666666667\n",
      "Time elapsed: 285.1393123570015\n",
      "Number of layers: 79\n",
      "Max training accuracy: 0.9783333241939545\n",
      "Max test accuracy: 0.98\n",
      "Time elapsed: 290.0236106089942\n",
      "Number of layers: 80\n",
      "Max training accuracy: 0.978124988079071\n",
      "Max test accuracy: 0.9816666666666667\n",
      "Time elapsed: 293.1460506069998\n",
      "Number of layers: 81\n",
      "Max training accuracy: 0.9862499892711639\n",
      "Max test accuracy: 0.9858333333333333\n",
      "Time elapsed: 295.68936294000014\n",
      "Number of layers: 82\n",
      "Max training accuracy: 0.956249988079071\n",
      "Max test accuracy: 0.9575\n",
      "Time elapsed: 300.9839290559976\n",
      "Number of layers: 83\n",
      "Max training accuracy: 0.9541666507720947\n",
      "Max test accuracy: 0.9516666666666667\n",
      "Time elapsed: 302.18984255499527\n",
      "Number of layers: 84\n",
      "Max training accuracy: 0.9985416531562805\n",
      "Max test accuracy: 1.0\n",
      "Time elapsed: 306.0389464520049\n",
      "Number of layers: 85\n",
      "Max training accuracy: 0.9791666507720947\n",
      "Max test accuracy: 0.9808333333333333\n",
      "Time elapsed: 312.5699106359971\n",
      "Number of layers: 86\n",
      "Max training accuracy: 0.9787499904632568\n",
      "Max test accuracy: 0.9825\n",
      "Time elapsed: 309.0978336639964\n",
      "Number of layers: 87\n",
      "Max training accuracy: 0.967083317041397\n",
      "Max test accuracy: 0.9691666666666666\n",
      "Time elapsed: 319.170153168001\n",
      "Number of layers: 88\n",
      "Max training accuracy: 0.9691666603088379\n",
      "Max test accuracy: 0.9700000000000001\n",
      "Time elapsed: 319.04545646199404\n",
      "Number of layers: 89\n",
      "Max training accuracy: 0.9570833206176758\n",
      "Max test accuracy: 0.9616666666666667\n",
      "Time elapsed: 322.9099918159991\n",
      "Number of layers: 90\n",
      "Max training accuracy: 0.9568749964237213\n",
      "Max test accuracy: 0.9516666666666667\n",
      "Time elapsed: 327.6823260230012\n",
      "Number of layers: 91\n",
      "Max training accuracy: 0.9589583158493042\n",
      "Max test accuracy: 0.9641666666666667\n",
      "Time elapsed: 327.8792826029967\n",
      "Number of layers: 92\n",
      "Max training accuracy: 0.9433333277702332\n",
      "Max test accuracy: 0.9391666666666667\n",
      "Time elapsed: 335.70265181099967\n",
      "Number of layers: 93\n",
      "Max training accuracy: 0.9466666579246521\n",
      "Max test accuracy: 0.9441666666666666\n",
      "Time elapsed: 337.1386116979993\n",
      "Number of layers: 94\n",
      "Max training accuracy: 0.9685416579246521\n",
      "Max test accuracy: 0.9691666666666666\n",
      "Time elapsed: 342.9985966259992\n",
      "Number of layers: 95\n",
      "Max training accuracy: 0.9387499868869782\n",
      "Max test accuracy: 0.9425\n",
      "Time elapsed: 343.3169296240012\n",
      "Number of layers: 96\n",
      "Max training accuracy: 0.965833330154419\n",
      "Max test accuracy: 0.9691666666666666\n",
      "Time elapsed: 356.40229330899456\n",
      "Number of layers: 97\n",
      "Max training accuracy: 0.9316666722297668\n",
      "Max test accuracy: 0.9216666666666666\n",
      "Time elapsed: 351.15332707600464\n",
      "Number of layers: 98\n",
      "Max training accuracy: 0.9349999904632569\n",
      "Max test accuracy: 0.9383333333333332\n",
      "Time elapsed: 349.2332181160018\n",
      "Number of layers: 99\n",
      "Max training accuracy: 0.9543749868869782\n",
      "Max test accuracy: 0.9475\n",
      "Time elapsed: 358.11501034200046\n",
      "Number of layers: 100\n",
      "Max training accuracy: 0.9024999976158142\n",
      "Max test accuracy: 0.9033333333333334\n",
      "Time elapsed: 360.7250728930012\n",
      "Number of layers: 101\n",
      "Max training accuracy: 0.9314583301544189\n",
      "Max test accuracy: 0.9291666666666667\n",
      "Time elapsed: 362.17166757799714\n",
      "Number of layers: 102\n",
      "Max training accuracy: 0.9020833373069763\n",
      "Max test accuracy: 0.8908333333333334\n",
      "Time elapsed: 368.4250983899983\n",
      "Number of layers: 103\n",
      "Max training accuracy: 0.9539583206176758\n",
      "Max test accuracy: 0.95\n",
      "Time elapsed: 371.1790710179994\n",
      "Number of layers: 104\n",
      "Max training accuracy: 0.9306249916553497\n",
      "Max test accuracy: 0.9191666666666667\n",
      "Time elapsed: 376.65511856200465\n",
      "Number of layers: 105\n",
      "Max training accuracy: 0.8887499928474426\n",
      "Max test accuracy: 0.8891666666666667\n",
      "Time elapsed: 378.567865721001\n",
      "Number of layers: 106\n",
      "Max training accuracy: 0.8856249928474427\n",
      "Max test accuracy: 0.8833333333333333\n",
      "Time elapsed: 381.49033059999783\n",
      "Number of layers: 107\n",
      "Max training accuracy: 0.932708328962326\n",
      "Max test accuracy: 0.935\n",
      "Time elapsed: 389.03956948799896\n",
      "Number of layers: 108\n",
      "Max training accuracy: 0.8739583373069764\n",
      "Max test accuracy: 0.8758333333333332\n",
      "Time elapsed: 391.13527958499617\n",
      "Number of layers: 109\n",
      "Max training accuracy: 0.920416659116745\n",
      "Max test accuracy: 0.9149999999999999\n",
      "Time elapsed: 394.36364295300154\n",
      "Number of layers: 110\n",
      "Max training accuracy: 0.896875\n",
      "Max test accuracy: 0.8991666666666667\n",
      "Time elapsed: 398.10207205900224\n",
      "Number of layers: 111\n",
      "Max training accuracy: 0.9077083170413971\n",
      "Max test accuracy: 0.9133333333333333\n",
      "Time elapsed: 401.35712934500043\n",
      "Number of layers: 112\n",
      "Max training accuracy: 0.921875\n",
      "Max test accuracy: 0.9183333333333333\n",
      "Time elapsed: 400.50612755100155\n",
      "Number of layers: 113\n",
      "Max training accuracy: 0.8758333206176758\n",
      "Max test accuracy: 0.8766666666666667\n",
      "Time elapsed: 410.8743564019969\n",
      "Number of layers: 114\n",
      "Max training accuracy: 0.8743750095367432\n",
      "Max test accuracy: 0.87\n",
      "Time elapsed: 410.57486408299883\n",
      "Number of layers: 115\n",
      "Max training accuracy: 0.9270833313465119\n",
      "Max test accuracy: 0.9316666666666666\n",
      "Time elapsed: 416.55313903399656\n",
      "Number of layers: 116\n",
      "Max training accuracy: 0.9531249940395355\n",
      "Max test accuracy: 0.9516666666666667\n",
      "Time elapsed: 420.19264313399617\n",
      "Number of layers: 117\n",
      "Max training accuracy: 0.8572916626930237\n",
      "Max test accuracy: 0.86\n",
      "Time elapsed: 420.8396190429994\n",
      "Number of layers: 118\n",
      "Max training accuracy: 0.8972916662693023\n",
      "Max test accuracy: 0.8916666666666667\n",
      "Time elapsed: 426.75833233999583\n",
      "Number of layers: 119\n",
      "Max training accuracy: 0.8625000029802322\n",
      "Max test accuracy: 0.86\n",
      "Time elapsed: 438.6337389580003\n",
      "Number of layers: 120\n",
      "Max training accuracy: 0.8464583218097687\n",
      "Max test accuracy: 0.85\n",
      "Time elapsed: 435.61391283300327\n",
      "Number of layers: 121\n",
      "Max training accuracy: 0.8241666704416275\n",
      "Max test accuracy: 0.8191666666666666\n",
      "Time elapsed: 437.6581864499967\n"
     ]
    }
   ],
   "source": [
    "from Networks.ResNet import ResAntiSymNet\n",
    "import numpy as np\n",
    "import timeit\n",
    "\n",
    "num_epochs = 100\n",
    "num_features = 2\n",
    "num_classes = 2\n",
    "size_hidden = 8\n",
    "gamma = 0.3\n",
    "h = 0.1\n",
    "lr = 1.0\n",
    "\n",
    "max_layers = 120\n",
    "train_results_per_layer = np.zeros(max_layers)\n",
    "test_results_per_layer = np.zeros(max_layers)\n",
    "basic_save_key = 'AntiSym_Comparison_results//AntiSymNet'\n",
    "\n",
    "for num_hidden_layers in range(1,max_layers+1):\n",
    "    tic = timeit.default_timer()\n",
    "    num_layers = num_hidden_layers + 1\n",
    "    print('Number of layers: '+str(num_layers))\n",
    "    train_results_per_epoch = np.zeros(num_epochs)\n",
    "    test_results_per_epoch = np.zeros(num_epochs)\n",
    "    for i in range(10):\n",
    "        net = ResAntiSymNet(features=num_features, classes=num_classes, num_layers=num_layers, gamma=gamma, h=h, bias=True, hidden_size=size_hidden)\n",
    "        net.set_test_tracking(True)\n",
    "        net.train(num_epochs, train_loader, test_loader, print_output=False)\n",
    "        train_results_per_epoch += net.avg_correct_pred.numpy()\n",
    "        test_results_per_epoch += net.test_results.numpy()\n",
    "        \n",
    "    train_results_per_epoch = train_results_per_epoch / 10\n",
    "    test_results_per_epoch = test_results_per_epoch / 10\n",
    "    train_save_key = basic_save_key+'_train_per_epoch_layers_'+str(num_layers)+'_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "    test_save_key = basic_save_key+'_test_per_epoch_layers_'+str(num_layers)+'_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "    \n",
    "    np.savetxt(train_save_key, train_results_per_epoch, delimiter=',')\n",
    "    np.savetxt(test_save_key, test_results_per_epoch, delimiter=',')\n",
    "    \n",
    "    train_results_per_layer[num_hidden_layers-1] = train_results_per_epoch.max()\n",
    "    print('Max training accuracy: '+str(train_results_per_epoch.max()))\n",
    "    test_results_per_layer[num_hidden_layers-1] = test_results_per_epoch.max()\n",
    "    print('Max test accuracy: '+str(test_results_per_epoch.max()/test_set_size))\n",
    "    \n",
    "    toc = timeit.default_timer()\n",
    "    print('Time elapsed: '+str(toc-tic))\n",
    "\n",
    "train_save_key = basic_save_key+'_train_per_layer_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "test_save_key = basic_save_key+'_test_per_layer_h_'+str(h)+'_lr_'+str(lr)+'.csv'\n",
    "np.savetxt(train_save_key, train_results_per_layer, delimiter=',')\n",
    "np.savetxt(test_save_key, test_results_per_layer, delimiter=',')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x2400a85aef0>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd8VfX9x/HX596EJIwQIOxhwhYRBAMOHEgdoBVxW0fVWrGu2mFbra12/bps1dpaXLWuqnUvUHGACxECArIJO8wQ9si8n98f53INEOBCc3NJfD995ME9+/PNjfdzzznf8/mauyMiIgIQSnYAIiJy6FBSEBGRGCUFERGJUVIQEZEYJQUREYlRUhARkRglBRERiVFSEBGRGCUFERGJSUl2AAcqOzvbc3Jykh2GiEidMmXKlHXu3nJ/69W5pJCTk0N+fn6ywxARqVPMbGk86+nykYiIxCgpiIhIjJKCiIjEKCmIiEiMkoKIiMQoKYiISIySgoiIxCgpiIhIjJKCiIjEKCmIiEiMkoKIiMQoKYiISIySgoiIxCgpiIhITMKSgpk9ZmZrzWzmXpabmd1vZgVmNsPM+icqFhERiU8izxQeB4buY/kwoFv0ZyQwKoGxiIhIHBKWFNz9I2D9PlY5B3jSAxOBLDNrm6h4RERk/5I58lp7YHmV6cLovFXJCWfvyrZtZNGr/0er4y+neW7fPZYvm/0568c9gEfKcTcyj7+arnmnxZbPee8JKsrLOXLYd2PzZn/wLOUbltHn3J9goV1zc1lZGdOevYu0zUtJCRvhkOEOEXcMwwzMIBKBSo8QiQTLKsPptDvrNtrl9NgjxqLZH7Hhk39RWVlJRSQCQDgUItywOV1G3E5K0yAfr188jcJ37idUWUo4ZKR0OYFup38vtp8vx79ExYwXCYWMsBkQHBuIxhWi8YDLyBkwLNjAnTlv3IutnEpqOERKyCivjFBSESES8WAbvtqPGaSEQoRDtnNzHK+2/RaNycwIhYyUjCw6jriLcOMWwf5Kt7PgxTuJbF4d2z74L0TmgIvJGXh29Be+nY3j7iOtTS8y+o4I5kUq2fDhKLYtyaci4lRURqiMOJUO4TZH0GP4TyAcx/8+kUo2f/h3UtsfRUb3wXssLl1bwIp3/0Hbk79DRoc+wSabV7Nk9F/I7n8OmT1O3GObDYu/YMPUV8kd8QssnLr/GGpQ8dTX8NKtZB93Wa0eV2qPefR/6ITs3CwHeNPde1ezbDTwB3f/JDr9PvBTd59SzbojCS4x0alTp6OXLo1rVLkasXjJIiqfuoCulQvZTCMKhz5Gr2O/uir22fgxHDHuGkJE2ExjmrCNcksl5ftTyGzeipULppH99BAaWCUftLyCvO/cw6Rnfsupy+8HYGLmUI783uM0apgBwMriTSx9+BKOK53AKm9BZeyjr3oW/cGguW9io2VSMPQZTjj2WADcnQljn+foCTdSRiqbaRjbowOt2EBZKAM//bfsKFpCs6kPUOZhNtKEBpTT0jbxfpvvMvDKP/DBi//krIJfs5WGbCO92ngasYNGlPBFv99w1Jkjmf7QNQwofp01nkX5bt9BqsZRtZUH+xfZhvUsa3kSnW98FcyY9/iN9FjyNCu8BR49gkVjzLJtLGhzFu2Pv5iSN2+nedkKANZ3u4Dmp93Kuv/eTHbxZFZ7MyoIx44RJkJbW09ho960uepJUlp22Ws8XrKZ5Y9eRqd1H7HNGpHx/YmEmnUKFlZWsHn830j7+E+kUUoFYbYMuJm0FrlExv6CxpEtABT1vIKWI34P6ZkATP5kLD3eu4pMtjGz90/pfcEdB/nbOjirftuTlpVr2H7FWDK7DKjVY8v/xsymuHveftdLYlJ4CBjv7s9Gp+cBg919n2cKeXl5fjBjNC9Zt435BQvIXPIOrVePoySUweLGR7Mssz8NWvegY4vGVFRGWL1qOaFVX+AVZXhFOaeuHEW2bWJq75/TcdbDtI6s5d1OP2BrVnd2bFjDhct/x+aU5oSvep3WHbuxYMZEcl8axhctziLvpieZ+6eTaVeykCUtTqTv+reZ5TkcYUuY2XQwO5p2ZcCyR5kYPpq5OVewzRrRp2AUJzKV2X1up9d5txGJOKUVkeBbe/Tbc3kkODtokPLVN2qA1fMmk/HceZRGQvy77Z14o1Y02TiL7667m1WpneDbr9KuXUcapARnJhWVEd775FNafHArA2wuAO+ETqLTpX+jZ5dcdpSWUfDIlfQpfosP/GgGM5UljfvS7vrXSG+cVe3veX3xOlY+fD69S6cx33Lp7ov5tO2V9Pn2X9heHmFbaQWZGalkZaSSEq7+6mVZRYTtZRUYRigUnDmkhIMzEwcqIsG39p0qIk5peYTx//4lF65/iOIhfyG1eUcyX7yYtxoOZ8iPHic1FCIU/V2t37SZCY/dxhkbnyPVKlkcac27uT8jpXAiV1a8QNicrZ7OE01v4IQLvk+zRmk0TAuTnhomJWS8+czfOX3Rn2gQirC98WG4O070rMadkrQWlDTqQMM1U2hTtpTn0i9mRMkrbGl+BG1vfhe2rGLLU5fTZN0XfOB5rO7/IzK/GMU3+RiAyZEezOt3Bw1mPc8F5aMpSc1kfvPBzKILw9eMYms4k6JQK7pWzGf7dyeQ3aErZQUfsvXjUTS/5EHI2O292b6edWN+R9n2TbS77EE4yLOL4sL5tHg0SAQrG+TQ7qeTICXtoPYlta8uJIWzgJuAM4FjgPvdfeD+9nmwSWHiU3dy7MK/AbAw0pYMK6OdFQOw1dOZ451ozA4ODy3fZbstoaaUX/wczXscz5YNayh6aASdS2bHlq9NO4ym3xtDWrMOsXkTRl3P8WueYXL2uQxY9woTet3J8Rf+iKUv/YLDZv6DpZ0v5bDL/wGhMAvfup+cz+8iTCS2fdHJf6DlKTcccBsBylbNovSxs2lSXhybt6bJEWR/703CjZpXu83Mwg288uR90KgV3/vOd2nZpMr/6JEIq565nrYFz7G21SBaffdFaNBwnzFUlO5g/kOX03P9+8zu+3N6n/vTg2rLgVq9cTuL7z2d/qEFlIYasrq8IaHrxtG1fas91nV33nz3XTbOGU+f4d+nb24bNmwr44Enn6bzitcp6ns9N5x3Gql7SVxvfPQ5pe/9nkzfssv8EE62baSjFREhRH6/P3LG8G/x2AO/5bvFf2Vrz4sILxxLRVkp96XfwCVX/4BubTJZvn47D/77X6RuX8PZV/yAo3Oy2bS9nH889RxHFv6HIaEvaGwlFKd3ovHIMazesJVWT57E3EZ5ZB1zGe3H3UIDKlh4yii6nHzpV+2c/BilY39FatlmwuYsaD2Mbtc9A7tdsqSynLKFHzHn/aep2LCcrje8QNOsZruskv/yfeTNuIsXs6/ngnWjWNRjJJ0v+TNsK4IGjYIfOWQlPSmY2bPAYCAbWAPcBaQCuPuDZmbAPwh6KG0Hrnb3/X7aH2xSWD9/ApGCDwj1OofMjkcE37o3LMaXfErJ8i+oXDkDUtJJ7TqYtM6DIK1xsGHTjrt+86osp3LVl4RKNmLl2yDnxD2+mW3bsoktf+1PG9YxO6UXPW7/hHA4egli61po1DK4KL7TxuWwaTns2AhNWkP7ow+4fbvYshqWfBK8DqVA11O/as9eVEackAXX5/fgDks/hQ4D4v9m6I5vK8Ia7/mBnEgvfPA5p314LhmU8kyff3P1+cMPaPtIxCncsINOLfad+AC2llZQUl5JSshICYdITwmREg5RWlHJ5h0VpIYgq1Hw+1q1cTtf3juC0+1z5kY68nSn3/Kzy79Jk/SvvrVHIk5FxGNncjuVlFeSRhm2Yiq0PiL29zbp6TsZWBB80ZlhPegZKWBWzrfpd/V9AFQsm0zKY6cyMXI4b3f8If1KJ3HOukeZ2vYSWh9zAZULPyR97TQabi8kY9sKwpEydngDMqyMCd1+wvGX/WKXOKb85Rw6bp1B1h0FjPvLJZxW+i6VoTRSIyWUpzUj9eo3oc0e3//kEJH0pJAoB5sUatuM8S/R/MM7KLvwSTr32u8JkNSQyojz078/ReWOjfzhRzeS0SC8/41qySufzSZ/9L/oNPgqrh3SO3Y562BFystYdM+pbE/LJveax1l5z8l4ehY9fzYOgIWv/I4u0+9m1IB3uG7YMcH9pX9ey4nFLwJQ6cY878RSb8Vyb8XSRkdy1ograPbyRWRVrKXVHXMIpwRJq7yigq2/y2Fh1vHk/eB55i9bwYx/38LG8hRWejbXpowmq4GTMfIdaLlnRwdJPiUF+doqraikvNJpnJbMznXVKymvJD01MYnqk3svp++m92ly10owY/593yRlQwFt75gVS46Rykq+fPMfbG/QnJTcE8jMyg56rkWcrq0ak54aZso7/+Hoz25g+jF/pW+0x9yMyR/SZ/Rwpg/4M33Pug4ILsGVVUbYUVbJ355/ixsW30x6g1S25g5lx+ZiIg2a0Pn8XxNq2u6A2uGV5cx782/knHIV6ZnZ0ZkO0/4DnU+Bpu1r7pf2NRJvUlCZC6l30lLCh2RCABKWEABo148mbKdo2Vxwp82m6SxteOQuZ0uhcJi+59zCccOuYEDPXHq0acLhbTPp3b5pLLa+37iYJdaezKmjgg9jYN2MdwDocsyZsX2ZGWkpYbIaNuCX3x7O870eYF1pmAbzXsNWTqXj0lco/dsAKib/O7afmLmjoXCPjoYATH3tAXp+8VvmPvdVz6qygvHw2o0Ujv5DTfymZB+UFETqiezuxwCwctYnbCmcQ6ZvpqL9gV+6TElJYUn375BbXsCyKWMAyFr1CctScmic3bHabUIh44aLvsmKKyYw94ppNLx1Bi8MfJ5p5Z1IGf0Dtj9+PqtWr2TJum0UvPI7eO5Syp+9FMpLdtnPlq1baT/j7wD0XPkyZRtXA7D+rf8DwBaOP+D2yIFRUhCpJ3J75VHiqZQsm8LS6e8D0Lr34IPa11FnjWStN6Plm1cz9u7L6VU+m01tB+1zGzPjhG7ZDOqaTevMdC4/awgFw/7DneVXkrJkPBX/PIkP77uSrtPvZnKkO6nbVrN23K7VbT79719pwzo+6HobqV7Okjf/TMWSCbRZP5kCb0f7yuWsWLJgj2Ov2rSDv723gIrKyB7LqopEnKcnLmXDtrID/6XUoPmrNlK0pTSpMeyNkoJIPZGWls6S1C40KZ5J+aIJrPcm9Dzi4OpMZmVmsujs55mRNYRTtr1FupXTtt+wA97PFcd3ZsR1v+bjE5+mecMwV6a8y7qu55N1/btMtiNJmXAv6zcE1XAKCtdy9LJ/sbBRPwZ/62eMSz2RjgX/YePrd1DsTSg4/s8AzPz41V2OEYk4P3p2Cs+/9ymTl2zYZzzTFq8m7c2beOjldw64LTVl/YYNhB88nmlP3Jq0GPbl0LzwKiIHZXOz3hyx9g02bihicUZvjv4f7mEcmzcQ8p4Lujgvm0j24Wcd1H76d2oGnc6E446BJR+TffhwskMh5p31K5q/eT5PPPIrJjUfzqDlD3GpbaLB8N8QCofYdswtZHx6Phnrp/JYxre5+rSz2DCpOeHF44lEbo313nrlg0+5deUP6Ju2kKdntuO4LqfvNZZl08dxYcpHvDI/wswVJ9G7fdODatP/YtF/byPPVrBt0/RaP3Y8dKYgUo+kdDyaRlZKe19NWbsa6grdpA0cMWLPB94OVKMWu+ynR96prG59Mhdte5Z7Cy/mUhvL+m7n07THSQCcetJg3uFYNnhj2p56MxYKsbX9SfSvnMaEgiIAiiY8xRkfX8Dh4RWkWITUgrf3GULZ0kkAnBWeyD/e+Iza7n25ecEE+q/6L+UepmX5ilo9dryUFETqkbaHHxd7nd3rpCRGEp825/2RtNZdaXDMtXD9BJpf9lhsWaO0FBYPupuftRzF6f27AdC63zCa21be/WAso19+ihZjb2ae5bDlOx+xOqMb3TZPoKS8MraPquVQyisjZG+cwbZwUxpQQeflLzF+XtFeY9v4xi/YOv21mmtsRSllL9/AaprxcYvzae3FbNu2teb2X0OUFETqkTadj2Q76ZSSSu6R+74xfEho3YvQDZ/BsD8GT2vv5nun9eHhm4bHanw16DYEgO6FL3P89NtYEurEpvOeo3XHbuzI+Qb9mceXBUHBzC8WLOXhX13FFwuC6RnLN9KHBWzoMIRIzslc2eAD7n5rVrVnCxVbi8mc8g9Wjr2/xpq6fdJTZO9YzKvtbyWrywBC5qxeOq/G9l9TlBRE6hELp7Aqqx8rM/uRkpaR7HBqXuNWVLY6kstS3qdpmtH5xlcY0icHgJb9h5NiEYqmBd1oV43+A9eHXmXO248AMHv2DLJtM1ndjyd0zLW09nV0LPqQgrV7fltfMvVdQjjtt82CSOUey/dmc0n5XpetmPEBq70ZQ4ZfQZO23QHYtGJu3PuuLUoKIvVMl+tfIPeGl5IdRsKEu58KQOi8h6DFV6XLG3c5lk2WSeNlH7B0+TJO2vAKAF2L3mVp8Ta2FHwWXe846D6Myibt+XZ4LGNnr9njGJvnjAeCMuubls+Kzf/dm7P589vVf5B/8PkX/Ol3tzF6cvXLm6ybTkFKd3q2yaT1Yb0AKFtTcICtTzwlBZH6Jq1JbPyFeumEH8J3xkLP3XpDhcIsa3YcvbdPYtFrv6chpWzqMpw8m8d/359EZvE0SkMZ0PJwCKcQHngNg8KzmDNj0h6HyCr6nBXeEoCVMz8CoiVKPr8f++SvzF65+auVl09m/cPDOXnMKfxfyqNEJvxzj/1VbttAm4pCtrUMBlLKbNGKTTTCNiyuoV9KzVFSEJG6Jb0pdDqm+mXdz6C5beGkoueY3nQITYf+kpA5O2a8RG+fz9YWR341Yl7/q6gIpXFc0Yus2fzVk9Wbi9eSU76YgvbnsNEbUR7tsTRl/jJuDr3IT1KeZ+ILdwcrr5hC5RPDqVgxg+fSL2JZWje6rP+I8t0eols2M6ha3Dj3q7jXhNvTcFvtDRgWLyUFEak3DhtwNpUeDPCaNfQOaNmdHc0P57zQx/SypWTkHvvVyo1asLX7uZwX/piPZnz1lPSC/LGEzGnV9zQWNOhJs/XB8wRr8l8jzcpZn5HLFesfYP5bD1D2xHmsLG/Md9P/wjduuJ/t3UfQyxYza/bMXeIqnj8RgM59vxpedVPDDrQoPfS6pSopiEi9kdmiFZ9lnMz4zLPJ7RWMS5Jx1AUcGVpCA6ukYedjd1m/6eCbyLAyIvlPxOaVLBhPiafS9aiT2dziKNqXL6VyxyZaLn+bDaHmNLrhPVaG2tD985+zuTTCb5v9Hw/f8E3aNE2nw3HnA7Buyq5PXaesmspSa0/bNm1i88oyc2kdWUtleVDuYuXGHazetGstqGRQUhCReiXvxy8z6JbHv5pxxLlfvW6/a+Voa3MkS5r054QNr7B1RwnuTqviySzO6E1qWgZpOccQMmf2hNEMKM9nZbvTSGuSzcJvPMK7lf15uNPd/O2G82nTNBizvHG7wykMd6Jl4buxY0QiTvvtc1iXuWuX23B2F8LmrF0WdEu96d/juenxD2v2l3EQlBREpF5JTw2TllKlvEeLLtC2L2R1CkY23E3p0dfS3tYx5fGf8N5n+XSJLKW0w/EAdDwyuNzTeMLdpFs5TfpfAMCQEwbR5ftvcNvVF+8xkNOadkM4onwm64qCXk2LFy2gJRsIddh1RMWGbYIH8jYun0fBmi3cteHn/LT4l7vexE4CJQURqf/OfRgufLzaRV0GXcgX6cdw8ponGfzOGYTMadM36PbaqV1bFtGe3MpFFJNFx75DYtt1btm42tHzsvqfS4pFWDzhZQCWzfwYgHZHnLDLetmdegKwfc0Cpn32Ln1DixgYmscnn3zwPzf3f6GkICL1X6ueex37PCU1lX63jWXj5WNZ12ko65v1pc3hwdPgZsaKxsG40/OaD8bC+68hmtvnRIpoRuqC4CG6sqX5VBCmVbddj9+6TQe2eAasX0SzOU+xwzIoswa0mPP0Hr2XapOqpIqIAFldjyGr6zN7zC9vOwAWvIMdMSKu/YTCYeY2G8yJG15h7K/OoIOvZlVGVzqm7vqEeTgcYlW4HU2Kp9O7fDEFHUaQlVrJ0MWj+XTWYgb36bKXIySWzhRERPah86nXMKrtbzly0Dfj3qbbpX8h/7BrOTk0nSNsCek51Q+NvCG9A90r5pNm5TQ/6Xu0HnIDjayUVR8/Ue36tUFnCiIi+5DTujnXX/f9A9qmTcts2lz9F9h2O0z7Dy13f/o6akeTHNj+IbPDh9OrRx64syqjO/3WvsKGrXfQrHFaDbTgwOhMQUQkURq1gEHf36VGU1XWvDMAhV2/FZ1hMOAaetoy3nviN3ik9u8tKCmIiCRJk37n8vvIlXQbcmVsXtsTr2Jp80FcWPQP5j90GZRtq9WYrLZHHvpf5eXleX5+frLDEBGpEZURj40XsZNHKhn9wI85c93jbGo9kGY3jP2fj2NmU9y9+psbVehMQUQkiXZPCAAWCjPkur/wSMNraLb2c7Yu/LzW4lFSEBE5BDVskMKJF/2ILZ7BwtH31NpxlRRERA5RvXLb82X2mRxe/B6Lli6plWMqKYiIHMJ6Df8hDayC/Jfvq3Y86ZqmpCAicgjLOuxIVjYbyPEb3+CDWSsTfjwlBRGRQ1yrU2+mg62jU/HHCT9WQpOCmQ01s3lmVmBmt1WzvJOZjTOzL8xshpmdmch4RETqopSeZ0KvEXQ7rEPij5WoHZtZGHgAOA0oBCab2evuPrvKar8Annf3UWbWCxgD5CQqJhGROimcAhfVTj2kRJ4pDAQK3H2Ru5cBzwHn7LaOA5nR102BxF8wExGRvUpkQbz2wPIq04XAMbut8ytgrJndDDQCTk1gPCIish+JPFPY8zG94Mygqm8Bj7t7B+BM4Ckz2yMmMxtpZvlmll9UVJSAUEVEBBKbFAqBjlWmO7Dn5aFrgOcB3P0zIB3I3n1H7v6wu+e5e17Lli0TFK6IiCQyKUwGuplZrpk1AC4BXt9tnWXANwDM7HCCpKBTARGRJElYUnD3CuAm4B1gDkEvo1lm9hszGx5d7cfAtWY2HXgWuMrrWtlWEZF6JKEjr7n7GIJuplXn3Vnl9WxgUCJjEBGR+O33TMHMetdGICIiknzxXD560MwmmdkNZpaV8IhERCRp9psU3P0E4DKCnkT5ZvaMmZ2W8MhERKTWxXWj2d0XEJSk+BlwMnC/mc01s/MSGZyIiNSueO4p9DGzewl6EA0Bznb3w6Ov701wfCIiUovi6X30D+AR4OfuvmPnTHdfaWa/SFhkIiJS6+JJCmcCO9y9EiBahiLd3be7+1MJjU5ERGpVPPcU3gMyqkw3jM4TEZF6Jp6kkO7uW3dORF83TFxIIiKSLPEkhW1m1n/nhJkdDezYx/oiIlJHxXNP4QfAC2a2s8JpW+DixIUkIiLJst+k4O6Tzawn0INgjIS57l6e8MhERKTWxVsQrwfQi6C0dT8zw92fTFxYIiKSDPtNCmZ2FzCYICmMAYYBnwBKCiIi9Uw8N5ovIBgIZ7W7Xw30BdISGpWIiCRFPElhh7tHgAozywTWAp0TG5aIiCRDPPcU8qMlsx8BpgBbgUkJjUpERJJin0nBzAz4g7tvJBhX4W0g091n1Ep0IiJSq/Z5+Sg6XvKrVaaXKCGIiNRf8dxTmGhmAxIeiYiIJF089xROAa4zs6XANoIH2Nzd+yQ0MhERqXXxJIVhCY9CREQOCfEkBU94FCIickiIJymMJkgMRlDmIheYBxyRwLhERCQJ4imId2TV6WgZ7esSFpGIiCRNPL2PduHuUwH1RhIRqYfiKYj3oyqTIaA/UJSwiEREJGniuafQpMrrCoJ7DC8lJhwREUmmeO4p/Lo2AhERkeTb7z0FM3s3WhBv53QzM3snsWGJiEgyxHOjuWW0IB4A7r4BaJW4kEREJFniSQqVZtZp54SZHUacD7SZ2VAzm2dmBWZ2217WucjMZpvZLDN7Jr6wRUQkEeK50XwH8ImZfRidPgkYub+NzCwMPACcBhQCk83sdXefXWWdbsDtwCB332BmOgMREUmieG40vx19YO1Ygqeaf+ju6+LY90CgwN0XAZjZc8A5wOwq61wLPBC9JIW7rz3A+EVEpAbFc6P5XKDc3d909zcIhuUcEce+2wPLq0wXRudV1R3obmafmtlEMxu6lxhGmlm+meUXFekRCRGRRInnnsJd7r5p50T0pvNdcWxn1czb/V5ECtANGAx8C3i0ak+nKsd82N3z3D2vZcuWcRxaREQORjxJobp14rkXUQh0rDLdAVhZzTqvuXu5uy8mKLTXLY59i4hIAsSTFPLN7B4z62Jmnc3sXmBKHNtNBrqZWa6ZNQAuAV7fbZ1XCQbxwcyyCS4nLYo/fBERqUnxJIWbgTLgv8ALQAlww/42cvcK4CbgHWAO8Ly7zzKz35jZ8Ohq7wDFZjYbGAf8xN2LD7wZIiJSE8z9wMbQMbN04Gx3fyExIe1bXl6e5+fnJ+PQIiJ1lplNcfe8/a0XV+lsMwub2TAzexJYAlz8P8YnIiKHoH3eMDazk4BLgbOAScAgoLO7b6+F2EREpJbtNSmYWSGwDBhFcK1/i5ktVkIQEam/9nX56CWCh80uBs42s0bEWfNIRETqpr0mBXe/BcgB7iHoNjofaBktYNe4dsITEZHatM8bzR74wN2vJUgQlwIjCG42i4hIPRPPk8kAuHs58AbwhpllJC4kERFJlri6pO7O3XfUdCAiIpJ8B5UURESkfoo7KUR7H4mISD0Wz3gKx0drE82JTvc1s38mPDIREal18Zwp3AucARQDuPt0giE5RUSknonr8pG7L99tVmUCYhERkSSLp0vqcjM7HvDouAjfJ3opSURE6pd4zhS+B9xIUPKiEDgqOi0iIvXMfs8U3H0dcFktxCIiIkm236RgZvdXM3sTkO/ur9V8SCIikizxXD5KJ7hktCD60wdoDlxjZvclMDYREall8dxo7goMiY65jJmNAsYCpwFfJjA2ERGpZfGcKbQHqj7N3Aho5+6VQGlCohIRkaSI50zhz8A0MxsPGMGDa7+Plr14L4GxiYhILYun99G/zGwMMJAgKfzc3Vf1YKFEAAASQUlEQVRGF/8kkcGJiEjtircgXgmwClgPdDUzlbkQEamH4umS+l3gFqADMA04FvgMGJLY0EREpLbFc6ZwCzAAWOrupwD9gKKERiUiIkkRT1IocfcSADNLc/e5QI/EhiUiIskQT++jQjPLAl4F3jWzDcDK/WwjIiJ1UDy9j86NvvyVmY0DmgJvJzQqERFJin0mBTMLATPcvTeAu39YK1GJiEhS7POegrtHgOlm1qmW4hERkSSK50ZzW2CWmb1vZq/v/Iln52Y21MzmmVmBmd22j/UuMDM3s7x4AxcRkZoXz43mXx/Mjs0sDDxAUDivEJhsZq+7++zd1mtCMJrb5wdzHBERqTn7PVOI3kdYAqRGX08Gpsax74FAgbsvcvcy4DngnGrW+y1BfaWSeIMWEZHE2G9SMLNrgReBh6Kz2hN0T92f9sDyKtOF0XlV990P6Ojub8YVrYiIJFQ89xRuBAYBmwHcfQHQKo7trJp5HlsY9Gy6F/jxfndkNtLM8s0sv6hID1OLiCRKPEmhNHr5BwAzS6HKh/s+FAIdq0x3YNeH3poAvYHxZraEoKbS69XdbHb3h909z93zWrZsGcehRUTkYMSTFD40s58DGWZ2GvAC8EYc200GuplZrpk1AC4BYr2W3H2Tu2e7e4675wATgeHunn/ArRARkRoRT1K4jaAA3pfAdcAY4Bf72yg6fOdNwDvAHOB5d59lZr8xs+EHH7KIiCSKue/7SpCZnQuMcfdDYujNvLw8z8/XyYSIyIEwsynuvt9nweI5UxgOzDezp8zsrOg9BRERqYfieU7haqArwb2ES4GFZvZoogMTEZHaF9e3fncvN7O3CHodZRA8hPbdRAYmIiK1L56H14aa2eNAAXAB8ChBPSQREaln4jlTuIqgRMV1h8rNZhERSYx4Btm5pOq0mQ0CLnX3GxMWlYiIJEVc9xTM7CiCm8wXAYuBlxMZlIiIJMdek4KZdSd4CvlbQDHwX4LnGk6ppdhERKSW7etMYS7wMXC2uxcAmNkPayUqERFJin31PjofWA2MM7NHzOwbVF/5VERE6om9JgV3f8XdLwZ6AuOBHwKtzWyUmZ1eS/GJiEgtiueJ5m3u/h93/yZB+etpBEXyRESknomn9lGMu69394fcfUiiAhIRkeQ5oKQgIiL1m5KCiIjEKCmIiEiMkoKIiMQoKYiISIySgoiIxCgpiIhIjJKCiIjEKCmIiEiMkoKIiMQoKYiISIySgoiIxCgpiIhIjJKCiIjEKCmIiEiMkoKIiMQoKYiISIySgoiIxCQ0KZjZUDObZ2YFZrbHuM5m9iMzm21mM8zsfTM7LJHxiIjIviUsKZhZGHgAGAb0Ar5lZr12W+0LIM/d+wAvAn9OVDwiIrJ/iTxTGAgUuPsidy8DngPOqbqCu49z9+3RyYlAhwTGIyIi+5HIpNAeWF5lujA6b2+uAd6qboGZjTSzfDPLLyoqqsEQRUSkqkQmBatmnle7otnlQB5wd3XL3f1hd89z97yWLVvWYIgiIlJVSgL3XQh0rDLdAVi5+0pmdipwB3Cyu5cezIHKy8spLCykpKTkoAKtK9LT0+nQoQOpqanJDkVE6qlEJoXJQDczywVWAJcAl1Zdwcz6AQ8BQ9197cEeqLCwkCZNmpCTk4NZdScodZ+7U1xcTGFhIbm5uckOR0TqqYRdPnL3CuAm4B1gDvC8u88ys9+Y2fDoancDjYEXzGyamb1+MMcqKSmhRYsW9TYhAJgZLVq0qPdnQyKSXIk8U8DdxwBjdpt3Z5XXp9bUsepzQtjp69BGEUkuPdFcAzZu3Mg///nPA97uzDPPZOPGjQmISETk4Cgp1IC9JYXKysp9bjdmzBiysrISFZaIyAFL6OWjr4vbbruNhQsXctRRR5Gamkrjxo1p27Yt06ZNY/bs2YwYMYLly5dTUlLCLbfcwsiRIwHIyckhPz+frVu3MmzYME444QQmTJhA+/btee2118jIyEhyy0Tk66beJYVfvzGL2Ss31+g+e7XL5K6zj9jr8j/+8Y/MnDmTadOmMX78eM466yxmzpwZ6yX02GOP0bx5c3bs2MGAAQM4//zzadGixS77WLBgAc8++yyPPPIIF110ES+99BKXX355jbZDRGR/6l1SOBQMHDhwl26j999/P6+88goAy5cvZ8GCBXskhdzcXI466igAjj76aJYsWVJr8YqI7FTvksK+vtHXlkaNGsVejx8/nvfee4/PPvuMhg0bMnjw4Gq7laalpcVeh8NhduzYUSuxiohUpRvNNaBJkyZs2bKl2mWbNm2iWbNmNGzYkLlz5zJx4sRajk5EJH717kwhGVq0aMGgQYPo3bs3GRkZtG7dOrZs6NChPPjgg/Tp04cePXpw7LHHJjFSEZF9M/dqa9QdsvLy8jw/P3+XeXPmzOHwww9PUkS16+vUVhGpOWY2xd3z9reeLh+JiEiMkoKIiMQoKYiISIySgoiIxCgpiIhIjJKCiIjEKCnUgIMtnQ1w3333sX379hqOSETk4Cgp1AAlBRGpL/REcw2oWjr7tNNOo1WrVjz//POUlpZy7rnn8utf/5pt27Zx0UUXUVhYSGVlJb/85S9Zs2YNK1eu5JRTTiE7O5tx48Yluyki8jVX/5LCW7fB6i9rdp9tjoRhf9zr4qqls8eOHcuLL77IpEmTcHeGDx/ORx99RFFREe3atWP06NFAUBOpadOm3HPPPYwbN47s7OyajVlE5CDo8lENGzt2LGPHjqVfv37079+fuXPnsmDBAo488kjee+89fvazn/Hxxx/TtGnTZIcqIrKH+nemsI9v9LXB3bn99tu57rrr9lg2ZcoUxowZw+23387pp5/OnXfemYQIRUT2TmcKNaBq6ewzzjiDxx57jK1btwKwYsUK1q5dy8qVK2nYsCGXX345t956K1OnTt1jWxGRZKt/ZwpJULV09rBhw7j00ks57rjjAGjcuDFPP/00BQUF/OQnPyEUCpGamsqoUaMAGDlyJMOGDaNt27a60SwiSafS2XXM16mtIlJzVDpbREQOmJKCiIjEKCmIiEhMvUkKde3eyMH4OrRRRJKrXiSF9PR0iouL6/WHprtTXFxMenp6skMRkXqsXnRJ7dChA4WFhRQVFSU7lIRKT0+nQ4cOyQ5DROqxhCYFMxsK/A0IA4+6+x93W54GPAkcDRQDF7v7kgM9TmpqKrm5uf97wCIiX3MJu3xkZmHgAWAY0Av4lpn12m21a4AN7t4VuBf4U6LiERGR/UvkPYWBQIG7L3L3MuA54Jzd1jkHeCL6+kXgG2ZmCYxJRET2IZFJoT2wvMp0YXReteu4ewWwCWiRwJhERGQfEnlPobpv/Lt3D4pnHcxsJDAyOrnVzObFcfxsYF0c69UFasuhqz61R205NNVUWw6LZ6VEJoVCoGOV6Q7Ayr2sU2hmKUBTYP3uO3L3h4GHD+TgZpYfT52PukBtOXTVp/aoLYem2m5LIi8fTQa6mVmumTUALgFe322d14Ero68vAD7w+vywgYjIIS5hZwruXmFmNwHvEHRJfczdZ5nZb4B8d38d+BfwlJkVEJwhXJKoeEREZP8S+pyCu48Bxuw2784qr0uACxN0+AO63HSIU1sOXfWpPWrLoalW21LnxlMQEZHEqRe1j0REpGbUy6RgZkPNbJ6ZFZjZbcmO50CYWUczG2dmc8xslpndEp3f3MzeNbMF0X+bJTvWeJlZ2My+MLM3o9O5ZvZ5tC3/jXZEOOSZWZaZvWhmc6Pvz3F19X0xsx9G/75mmtmzZpZel94XM3vMzNaa2cwq86p9Lyxwf/TzYIaZ9U9e5HvaS1vujv6dzTCzV8wsq8qy26NtmWdmZ9R0PPUuKcRZXuNQVgH82N0PB44FbozGfxvwvrt3A96PTtcVtwBzqkz/Cbg32pYNBOVO6oK/AW+7e0+gL0Gb6tz7Ymbtge8Dee7em6AjyCXUrfflcWDobvP29l4MA7pFf0YCo2opxng9zp5teRfo7e59gPnA7QDRz4JLgCOi2/wz+plXY+pdUiC+8hqHLHdf5e5To6+3EHzwtGfXkiBPACOSE+GBMbMOwFnAo9FpA4YQlDWBOtIWM8sETiLoMYe7l7n7Ruro+0LQySQj+nxQQ2AVdeh9cfeP2POZpr29F+cAT3pgIpBlZm1rJ9L9q64t7j42WuUBYCLBc14QtOU5dy9198VAAcFnXo2pj0khnvIadYKZ5QD9gM+B1u6+CoLEAbRKXmQH5D7gp0AkOt0C2FjlD76uvD+dgSLg39FLYY+aWSPq4Pvi7iuAvwDLCJLBJmAKdfN9qWpv70Vd/0z4DvBW9HXC21Ifk0JcpTMOdWbWGHgJ+IG7b052PAfDzL4JrHX3KVVnV7NqXXh/UoD+wCh37wdsow5cKqpO9Fr7OUAu0A5oRHCJZXd14X2JR139m8PM7iC4pPyfnbOqWa1G21Ifk0I85TUOaWaWSpAQ/uPuL0dnr9l5yhv9d22y4jsAg4DhZraE4DLeEIIzh6zoZQuoO+9PIVDo7p9Hp18kSBJ18X05FVjs7kXuXg68DBxP3Xxfqtrbe1EnPxPM7Ergm8BlVSo9JLwt9TEpxFNe45AVveb+L2COu99TZVHVkiBXAq/VdmwHyt1vd/cO7p5D8D584O6XAeMIyppA3WnLamC5mfWIzvoGMJs6+L4QXDY61swaRv/edralzr0vu9nbe/E68O1oL6RjgU07LzMdqiwYoOxnwHB3315l0evAJWaWZma5BDfPJ9Xowd293v0AZxLcsV8I3JHseA4w9hMITgdnANOiP2cSXIt/H1gQ/bd5smM9wHYNBt6Mvu4c/UMuAF4A0pIdX5xtOArIj743rwLN6ur7AvwamAvMBJ4C0urS+wI8S3A/pJzg2/M1e3svCC65PBD9PPiSoNdV0tuwn7YUENw72PkZ8GCV9e+ItmUeMKym49ETzSIiElMfLx+JiMhBUlIQEZEYJQUREYlRUhARkRglBRERiVFSkEOambmZ/bXK9K1m9qsa2vfjZnbB/tf8n49zYbSq6rjd5udUrYwpcihQUpBDXSlwnpllJzuQqg6wMuU1wA3ufkqi4tmfKk8qi+yTkoIc6ioIhiP84e4Ldv+mb2Zbo/8ONrMPzex5M5tvZn80s8vMbJKZfWlmXars5lQz+zi63jej24ej9ewnR+vZX1dlv+PM7BmCh6B2j+db0f3PNLM/RefdSfBA4oNmdnc8DTaza6PHnm5mL0WfPG5iZoujJVAws0wzW2JmqWbWxczeNrMp0bb0rPL7uSd6hvInMzvZzKZFf74wsybxxCNfL/r2IHXBA8AMM/vzAWzTFzicoCTxIuBRdx9owaBFNwM/iK6XA5wMdAHGmVlX4NsEpRAGmFka8KmZjY2uP5Cgzv3iqgczs3YE4xEcTTAWwVgzG+HuvzGzIcCt7p4fZ+wvu/sj0f3+DrjG3f9uZuMJypC/SlA25CV3Lzezh4HvufsCMzsG+CdBnSmA7sCp7l5pZm8AN7r7p9GCiyVxxiNfIzpTkEOeB1VinyQYGCZekz0Ym6KUoCTAzg/1LwkSwU7Pu3vE3RcQJI+ewOkEtXKmEZQtb0FQYwZg0u4JIWoAMN6DInM7q1qedADxVtU7+o3/S+AyggFVIBiT4uro66sJyng3Jihm90I03oeAqmMFvODuldHXnwL3mNn3gSz/qky2SIzOFKSuuA+YCvy7yrwKol9sooXdqg4fWVrldaTKdIRd/+53r/PiBLVybnb3d6ouMLPBBCWzq1NdSeOD9Tgwwt2nm9lVBHWjiH7DzzGzk4Gwu8+0YPCfje5+1F72FYvX3f9oZqMJamlNNLNT3X1uDcYt9YDOFKROcPf1wPPsOkTkEoLLNRCMD5B6ELu+0MxC0fsMnQmKjL0DXF/l+n13CwbU2ZfPgZPNLDt6E/pbwIcHEQ9AE2BV9PiX7bbsSYICav+G2FnUYjO7MBqrmVnf6nZqZl3c/Ut3/xNBYb+eBxmf1GNKClKX/BWo2gvpEYIP4knAMez9W/y+zCP48H6L4Lp8CcFlmtnA1GiX0YfYz1m1B6WYbycoPz0dmOru8ZSe7mFmhVV+LgR+SZBk3iWoZFrVfwiqsz5bZd5lwDVmNh2Yxd6Hn/1B9Cb4dGAHX43mJRKjKqkidUi0t9U57n5FsmOR+kn3FETqCDP7O8GwmWcmOxapv3SmICIiMbqnICIiMUoKIiISo6QgIiIxSgoiIhKjpCAiIjFKCiIiEvP/wWer4CkVrRAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "layers = np.arange(2,max_layers+2,1)\n",
    "ax.plot(layers, train_results_per_layer, label='train')\n",
    "ax.plot(layers, test_results_per_layer/test_set_size, label='test')\n",
    "ax.set_ylim(0,1.1)\n",
    "ax.set_xlabel('Number of Layers')\n",
    "ax.set_ylabel('Average Accuracy')\n",
    "ax.legend(loc='best')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}