10: Classification_BERT_DistillBert_ALBERT_RoBERTa_XNNet_GPT2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "10: BERT, DistillBert, ALBERT, RoBERTa, XNNet, GPT2_Classification.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "E6BxJG3XtV84",
        "IC1tBYxRA6iU",
        "lri8KEomGkAh",
        "kFiArS_C4DlQ",
        "wwHNCEdHEaSz",
        "fZKbpevQ7a-h",
        "6QGPheuGXqIV",
        "wOz-aOhucHlf",
        "AHM3fSwmXs39",
        "VHbXqbckfqEo",
        "cLbK5JgBFkua",
        "KkM5QS7-BWrU",
        "vY0pHkn6mhYm",
        "kINbb8yIwFff",
        "-THS6Ok_XCSH",
        "kiSeT-4YwFfg",
        "XSybv8T4wFfg",
        "jGRK8fUfwFfg",
        "BRMtnpJ5wFfh",
        "t8fzWG5hw7rE",
        "c95d2LP5XGFm",
        "fYn3lsgYw7rL",
        "oxy_rKYUw7rN",
        "Q4cPsPMKw7rQ",
        "HnM7tMYDw7rT",
        "7afhBl6Lr2-V",
        "lhbg8DNEXKbl",
        "Sr-OL53Or2-Z",
        "Re320PdZr2-a",
        "bLOxFxOdr2-c",
        "es0jaUt_r2-d",
        "CmBcXmA_whrO",
        "yU3E8E2uXTWL",
        "vkOrqt-ywhrR",
        "sGXh1G4BwhrR",
        "gFaKlixUwhrS",
        "sACsGeJ8whrU",
        "Yr5JKDhBsAWS",
        "VCksRw4TL8xG",
        "nSyd4V2OL_Eo",
        "w0koK3CBLy19",
        "GevTfVV6Y7ZN",
        "DTXvFqcOLqMr",
        "hb1YZcAPLimH",
        "-SqL7dLuLaqN",
        "Z3XAE2JGLR3e",
        "doKtdR5qbiq0",
        "jYGBIz7qb5Py",
        "ruEMXcq5cKZv",
        "qjuCtVQtMKD3"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/gulabpatel/NLP_BERT/blob/main/10%3A%20Classification_BERT_DistillBert_ALBERT_RoBERTa_XNNet_GPT2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Loading, EDA & Preprocessing"
      ],
      "metadata": {
        "id": "E6BxJG3XtV84"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b50CCOvhAXfk"
      },
      "source": [
        "## Installing Simple Transformers\n",
        "!pip install simpletransformers"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_UkeC7SG2krJ"
      },
      "source": [
        "## Read Data\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import csv\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import accuracy_score\n",
        "from sklearn.metrics import classification_report\n",
        "from sklearn.metrics import confusion_matrix\n",
        "from sklearn.metrics import f1_score\n",
        "from sklearn.metrics import precision_score\n",
        "from sklearn.utils import class_weight"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F9YXiqtRUSxS",
        "outputId": "a8019b23-c204-4df2-a06b-d382995362d0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.read_csv(\"/content/drive/MyDrive/dataset/tripadvisor_hotel_reviews.csv\")\n",
        "df.sample(10)"
      ],
      "metadata": {
        "id": "Qby2C-zHUYg3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IC1tBYxRA6iU"
      },
      "source": [
        "## EDA and Prep Data for SimpleTransformers"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df[\"Review\"] = df[\"Review\"].apply(lambda x: x.replace(\"* \", \"\"))\n",
        "df[\"Review\"] = df[\"Review\"].apply(lambda x: x.replace(\"__Ç_é_\", \"s\"))\n",
        "df[\"Review\"] = df[\"Review\"].apply(lambda x: x.replace(\"__Ç\", \"s\"))\n",
        "df[\"Review\"] = df[\"Review\"].apply(lambda x: x.replace(\"///\", \" \"))"
      ],
      "metadata": {
        "id": "G8ZAtvP0WYEf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oeA6OK4H2eVL"
      },
      "source": [
        "# Rename 'sentence' column to 'text' (this is the name SimpleTransformers expects).\n",
        "df.columns = ['text', 'label']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Tramsformer assumes label from 0 to Num_labels."
      ],
      "metadata": {
        "id": "AvDPNybHarg-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Dict={1:0,2:1,3:2,4:3,5:4}\n",
        "# df['label']=df['label'].map(Dict)\n",
        "df['label'] = df['label'].apply(lambda x:x-1)"
      ],
      "metadata": {
        "id": "FtnrgzfnR6EO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## EDA"
      ],
      "metadata": {
        "id": "lri8KEomGkAh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check Missing values"
      ],
      "metadata": {
        "id": "D_9YgUabaKlE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.isnull().sum()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5sC9VFPGSLlr",
        "outputId": "a961ebaf-75ee-453f-8ac1-22bcf8d603db"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "text     0\n",
              "label    0\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check calss imbalancy"
      ],
      "metadata": {
        "id": "BQz_PWyDaNMa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.label.value_counts()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DJJv7pVUGnw_",
        "outputId": "00ddb829-818a-4198-d76a-49b109bd69bd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4    9054\n",
              "3    6039\n",
              "2    2184\n",
              "1    1793\n",
              "0    1421\n",
              "Name: label, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.groupby(\"label\").size().plot.bar()"
      ],
      "metadata": {
        "id": "QvfSB3fxDFtH",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "af32ab21-f627-49f5-a765-f0ed82e0439c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7f0cc701c550>"
            ]
          },
          "metadata": {},
          "execution_count": 10
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEDCAYAAADZUdTgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAPK0lEQVR4nO3df6zddX3H8efLFlRmwg+5Idiit4mdDtxUaABnYqZsUMVYlqDBGWkMS/9Ynbgsm7j90URlwWQb02WSNYIDR6ysM6ERAmkATdwi0gJTS2XcIdg2/LjaCnOoWH3vj/spPZB7e8/VyzkXP89HQu73+/l+v+d+vifyPMfv/Z5DqgpJUh9eNO4JSJJGx+hLUkeMviR1xOhLUkeMviR1xOhLUkeWj3sCR3LiiSfW5OTkuKchSS8oO3fu/H5VTcy2bUlHf3Jykh07dox7GpL0gpLk4bm2eXlHkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0v6w1mSNEqTl9007inw0BXnP6+P7zt9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0ZekjgwV/SR/lmRXkm8n+UKSlyRZleTOJFNJvpjk6Lbvi9v6VNs+OfA4H23j9yc57/k5JUnSXOaNfpIVwIeANVX1OmAZcBHwSeDKqno1cAC4pB1yCXCgjV/Z9iPJqe2404C1wGeSLFvc05EkHcmwl3eWAy9Nshw4BngEeBuwtW2/FrigLa9r67Tt5yRJG99SVT+tqu8CU8CZv/opSJKGNW/0q2of8LfA95iJ/RPATuCHVXWw7bYXWNGWVwB72rEH2/4vHxyf5ZhnJNmQZEeSHdPT07/MOUmS5jDM5Z3jmXmXvgp4BfAbzFyeeV5U1eaqWlNVayYmJp6vXyNJXRrm8s7vA9+tqumq+hnwJeDNwHHtcg/ASmBfW94HnALQth8L/GBwfJZjJEkjMEz0vwecneSYdm3+HOA+4A7gwrbPeuDGtrytrdO2315V1cYvanf3rAJWA99YnNOQJA1j+Xw7VNWdSbYCdwMHgXuAzcBNwJYkn2hjV7dDrgY+n2QK2M/MHTtU1a4kNzDzgnEQ2FhVP1/k85EkHcG80Qeoqk3ApucMP8gsd99U1U+Ad8/xOJcDly9wjpKkReInciWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0Zfkjpi9CWpI0ZfkjoyVPSTHJdka5LvJNmd5E1JTkiyPckD7efxbd8k+XSSqSTfTHL6wOOsb/s/kGT983VSkqTZDftO/1PALVX1WuD1wG7gMuC2qloN3NbWAd4OrG7/bACuAkhyArAJOAs4E9h06IVCkjQa80Y/ybHAW4CrAarq6ar6IbAOuLbtdi1wQVteB1xXM74OHJfkZOA8YHtV7a+qA8B2YO2ino0k6YiWD7HPKmAa+FyS1wM7gUuBk6rqkbbPo8BJbXkFsGfg+L1tbK5xSWM0edlN454CD11x/rin0I1hLu8sB04HrqqqNwL/x+FLOQBUVQG1GBNKsiHJjiQ7pqenF+MhJUnNMNHfC+ytqjvb+lZmXgQea5dtaD8fb9v3AacMHL+yjc01/ixVtbmq1lTVmomJiYWciyRpHvNGv6oeBfYkeU0bOge4D9gGHLoDZz1wY1veBlzc7uI5G3iiXQa6FTg3yfHtD7jntjFJ0ogMc00f4E+B65McDTwIfICZF4wbklwCPAy8p+17M/AOYAp4qu1LVe1P8nHgrrbfx6pq/6KchSRpKENFv6ruBdbMsumcWfYtYOMcj3MNcM1CJihJWjx+IleSOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOmL0JakjRl+SOjJ09JMsS3JPki+39VVJ7kwyleSLSY5u4y9u61Nt++TAY3y0jd+f5LzFPhlJ0pEt5J3+pcDugfVPAldW1auBA8AlbfwS4EAbv7LtR5JTgYuA04C1wGeSLPvVpi9JWoihop9kJXA+8Nm2HuBtwNa2y7XABW15XVunbT+n7b8O2FJVP62q7wJTwJmLcRKSpOEM+07/H4C/BH7R1l8O/LCqDrb1vcCKtrwC2APQtj/R9n9mfJZjJEkjMG/0k7wTeLyqdo5gPiTZkGRHkh3T09Oj+JWS1I1h3um/GXhXkoeALcxc1vkUcFyS5W2flcC+trwPOAWgbT8W+MHg+CzHPKOqNlfVmqpaMzExseATkiTNbd7oV9VHq2plVU0y84fY26vqfcAdwIVtt/XAjW15W1unbb+9qqqNX9Tu7lkFrAa+sWhnIkma1/L5d5nTR4AtST4B3ANc3cavBj6fZArYz8wLBVW1K8kNwH3AQWBjVf38V/j9kqQFWlD0q+orwFfa8oPMcvdNVf0EePccx18OXL7QSUqSFoefyJWkjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SeqI0Zekjhh9SerI8nFPQBqHyctuGvcUeOiK88c9BXVo3nf6SU5JckeS+5LsSnJpGz8hyfYkD7Sfx7fxJPl0kqkk30xy+sBjrW/7P5Bk/fN3WpKk2Qxzeecg8OdVdSpwNrAxyanAZcBtVbUauK2tA7wdWN3+2QBcBTMvEsAm4CzgTGDToRcKSdJozBv9qnqkqu5uy/8L7AZWAOuAa9tu1wIXtOV1wHU14+vAcUlOBs4DtlfV/qo6AGwH1i7q2UiSjmhBf8hNMgm8EbgTOKmqHmmbHgVOassrgD0Dh+1tY3ONS5JGZOjoJ3kZ8O/Ah6vqycFtVVVALcaEkmxIsiPJjunp6cV4SElSM1T0kxzFTPCvr6ovteHH2mUb2s/H2/g+4JSBw1e2sbnGn6WqNlfVmqpaMzExsZBzkSTNY5i7dwJcDeyuqr8f2LQNOHQHznrgxoHxi9tdPGcDT7TLQLcC5yY5vv0B99w2JkkakWHu038z8H7gW0nubWN/BVwB3JDkEuBh4D1t283AO4Ap4CngAwBVtT/Jx4G72n4fq6r9i3IWkqShzBv9qvoakDk2nzPL/gVsnOOxrgGuWcgEtXiWwgeSwA8lSePk1zBIUkeMviR1xOhLUkeMviR1xOhLUkeMviR1xOhLUkeMviR15Nf+v5zlB5Ik6TDf6UtSR4y+JHXE6EtSR4y+JHXE6EtSR4y+JHXE6EtSR4y+JHXE6EtSR4y+JHXE6EtSR4y+JHXE6EtSR4y+JHXE6EtSR4y+JHXE6EtSR4y+JHXE6EtSR4y+JHXE6EtSR4y+JHXE6EtSR4y+JHXE6EtSR4y+JHXE6EtSR4y+JHVk5NFPsjbJ/Ummklw26t8vST0bafSTLAP+CXg7cCrw3iSnjnIOktSzUb/TPxOYqqoHq+ppYAuwbsRzkKRupapG98uSC4G1VfXHbf39wFlV9cGBfTYAG9rqa4D7RzbBuZ0IfH/ck1gifC4O87k4zOfisKXwXLyqqiZm27B81DOZT1VtBjaPex6DkuyoqjXjnsdS4HNxmM/FYT4Xhy3152LUl3f2AacMrK9sY5KkERh19O8CVidZleRo4CJg24jnIEndGunlnao6mOSDwK3AMuCaqto1yjn8kpbU5aYx87k4zOfiMJ+Lw5b0czHSP+RKksbLT+RKUkeMviR1xOhLUkeW3H3645bktcx8SnhFG9oHbKuq3eOblcat/e9iBXBnVf1oYHxtVd0yvpmNXpIzgaqqu9rXqKwFvlNVN495amOX5Lqqunjc8zgS/5A7IMlHgPcy8/UQe9vwSmZuLd1SVVeMa25LTZIPVNXnxj2PUUjyIWAjsBt4A3BpVd3Ytt1dVaePc36jlGQTM9+dtRzYDpwF3AH8AXBrVV0+xumNVJLn3m4e4K3A7QBV9a6RT2oIRn9Akv8GTquqnz1n/GhgV1WtHs/Mlp4k36uqV457HqOQ5FvAm6rqR0kmga3A56vqU0nuqao3jnWCI9SeizcALwYeBVZW1ZNJXsrM/wv6nbFOcISS3A3cB3wWKGai/wVm3iRSVV8d3+zm5uWdZ/sF8Arg4eeMn9y2dSXJN+faBJw0yrmM2YsOXdKpqoeS/B6wNcmrmHkuenKwqn4OPJXkf6rqSYCq+nGS3v4dWQNcCvw18BdVdW+SHy/V2B9i9J/tw8BtSR4A9rSxVwKvBj4451G/vk4CzgMOPGc8wH+Ofjpj81iSN1TVvQDtHf87gWuA3x7v1Ebu6STHVNVTwBmHBpMcS2dvjKrqF8CVSf6t/XyMF0BTl/wER6mqbknym8x8BfTgH3Lvau9uevNl4GWHYjcoyVdGP52xuRg4ODhQVQeBi5P883imNDZvqaqfwjPRO+QoYP14pjReVbUXeHeS84Enxz2f+XhNX5I64n36ktQRoy9JHTH60oAkP5pn+2SSby/wMf+l/VfjpLEz+pLUEaMvzSLJy5LcluTuJN9Ksm5g8/Ik1yfZnWRrkmPaMWck+WqSnUluTXLymKYvzcnoS7P7CfCH7SsW3gr8XZJDH8R6DfCZqvotZm7R+5MkRwH/CFxYVWcwcw9/N19JoBcO79OXZhfgb5K8hZkPHa3g8KeQ91TVf7TlfwU+BNwCvA7Y3l4blgGPjHTG0hCMvjS79wETwBlV9bMkDwEvadue++GWQ9+7squq3jS6KUoL5+UdaXbHAo+34L8VeNXAtlcmORT3PwK+BtwPTBwaT3JUktNGOmNpCEZfmt31wJr2rZIXA98Z2HY/sDHJbuB44Kqqehq4EPhkkv8C7gV+d8Rzlubl1zBIUkd8py9JHTH6ktQRoy9JHTH6ktQRoy9JHTH6ktQRoy9JHTH6ktSR/wdWGdxwjADUkwAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Data is imbalance, let's calculate the class_weights"
      ],
      "metadata": {
        "id": "PVZIc6SNaQbh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.utils.class_weight import compute_class_weight\n",
        "class_weights = compute_class_weight(class_weight = \"balanced\", classes= np.unique(df.label), y= df.label)"
      ],
      "metadata": {
        "id": "9_SFTTxHbhQT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wv7xPZlNCvr1"
      },
      "source": [
        "Split off 80-10-10% for Training, validation and Testing.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "loJK0iLaCXDZ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "42e8f398-f98c-451d-8104-f15bb80c66f6"
      },
      "source": [
        "# Use 80% for training and 10% for validation and 10% for testing.\n",
        "train_df, val_df = train_test_split(df, random_state=2018, test_size=0.2)\n",
        "validation_df, test_df = train_test_split(val_df, random_state=2018, test_size=0.5)\n",
        "\n",
        "print(\"train_df size : \", len(train_df))\n",
        "print(\"validation_df size : \", len(validation_df))\n",
        "print(\"test_df size : \", len(test_df))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "train_df size :  16392\n",
            "validation_df size :  2049\n",
            "test_df size :  2050\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kFiArS_C4DlQ"
      },
      "source": [
        "# Model 1 Bert(bert base uncased)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aH5bIzq_DXct"
      },
      "source": [
        "\n",
        "reprocess_input_data: bool\n",
        "If True, the input data will be reprocessed even if a cached file of the input data exists in the cache_dir.\n",
        "\n",
        "supported models https://simpletransformers.ai/docs/classification-specifics/#supported-model-types"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ITLUMV1CBmQS",
        "outputId": "42e30020-e105-425a-f9f3-e0d7bb5f81de",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 284,
          "referenced_widgets": [
            "08eefeed053245a7bbfbd1ee57e9bf27",
            "85d25eee8ca74078aeab160967a1c8c3",
            "f12f99a6c33d40a2bdf7ecedb8831fc8",
            "63ce303ccc2d4b4681f4f9b8c43cc256",
            "8ec8e07e55c8401e9579647d3ce8d6e2",
            "ab89b96360b54999a5082d4a15ae737a",
            "e9afb324a41541cb97e5f6e747121751",
            "fc2f533472834b438a27944bd58dc0f1",
            "ff52a564b25049ac89946d6178acb7c4",
            "c88dd478b44849d2a77eaecf25ffe273",
            "64f0b280c4d145619b65ef90ad4a28b1",
            "f4b83c8e0bda4e4c8bc1e4c91e12c010",
            "8c4f33c49b8548899489a56f932656ad",
            "62866911859b460cb73eef076a1795a1",
            "6ebfb4bdbe5741b380e3965e5b62f2ed",
            "a89cd965899548198eac44fadeca99db",
            "89b6258549ba436d9c1c54d39562e63d",
            "d1453b9561a74200a129337a924b042d",
            "8fe5fb6556ee46b097993729af60f0b3",
            "11280d8b63614beaaee5b52351119c8e",
            "da72549a6514441298a95c122369a0f1",
            "ffff8ae6cd9041efae76cfb56f9bdd47",
            "4787bba030a548cbb860144df3477e1b",
            "5a41551d951340efb3436e9ebee43abc",
            "8656e2a28ba34e9d92fffdd59a37ea3a",
            "d292aeea3b7346a59e6635672c8deb73",
            "c3a6ff395822469eb31ddbb93df059ae",
            "2a598fd6e611474b863c9d5eee3e6947",
            "a83b52bb7d754128878c252da57328f8",
            "68c05b64fc284175b0706153d4ca6568",
            "cf9a78a4faf4480298c32f7283d6b386",
            "88a25dfd161f46f7b2f1f97546d983dc",
            "42d76ae68c4f4e609e096310aaae046c",
            "884a9a305acf4251b2b5a7fe51756c7d",
            "401230600af5487ba7ff271b1aa843f8",
            "3c8675b179614c4c87bba9aadb7953f1",
            "cc2118a382724b80ad5237e26c9211b3",
            "21292a67c69e4ae3b5cc0727f63bc70d",
            "66c625f4df2544ba98892d2db66e8130",
            "96e4347784ef4634876a1c0761b81cc6",
            "20eb815386bd426a896a42c90624e532",
            "fb774abfa21b473f8170c1d52557efe7",
            "9968135458f445278cf317c5e554c799",
            "c5b3a5d5c5e648d79d5612cdeed376e9",
            "75f0d705f2c440b6b786409a9a0e9a7e",
            "d079b7b5dbcf4eb0b095a65e9e6926b7",
            "a5f2c2a7ed0b40cca27a971f664688a9",
            "9b2fc4329bca4d48858ddff5b1800abe",
            "0ac4bef5461e4dd99e2dbdf4fcd7ac38",
            "cb5d385e9a8e4cd69cc1a2a98264d3cf",
            "42e064d5abaf4018a1092ada0e4c99df",
            "1ff1cabad1574dd7bd0409e99b185db6",
            "79ba570d80174baabc9f9f17dfa6b770",
            "1c53d0bbd07940f2a29140d3226559e7",
            "832961fa9ca24f4e959f082147a91bc7"
          ]
        }
      },
      "source": [
        "from simpletransformers.classification import ClassificationModel\n",
        "\n",
        "args = {\n",
        "    'reprocess_input_data': True, \n",
        "    'overwrite_output_dir': True\n",
        "}\n",
        "\n",
        "model = ClassificationModel('bert', 'bert-base-uncased', num_labels=5, args=args)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/570 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "08eefeed053245a7bbfbd1ee57e9bf27"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/420M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f4b83c8e0bda4e4c8bc1e4c91e12c010"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight']\n",
            "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "4787bba030a548cbb860144df3477e1b"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/226k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "884a9a305acf4251b2b5a7fe51756c7d"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/455k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "75f0d705f2c440b6b786409a9a0e9a7e"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wwHNCEdHEaSz"
      },
      "source": [
        "## Inspect Training Arguments"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HeBRWPzmqUze"
      },
      "source": [
        "Retrieves all of the arguments and their values, and prints them in a table.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ADMTSlPv4tsh",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "outputId": "e8840756-6506-4cf3-bf67-aa42c879971a"
      },
      "source": [
        "arg_values = []\n",
        "pd.set_option('display.max_rows', 80)\n",
        "\n",
        "# For all of the arguments...\n",
        "for arg in dir(model.args):\n",
        "    \n",
        "    # Skip over the special attributes and any functions.\n",
        "    if (not arg[0:2] == '__') and (not callable(getattr(model.args, arg))):\n",
        "    \n",
        "        # Store the argument and its value as a tuple.\n",
        "        arg_values.append((arg, str(getattr(model.args, arg))))\n",
        "\n",
        "# Store as a dataframe just to get the pretty printout.\n",
        "df_args = pd.DataFrame(arg_values)        \n",
        "\n",
        "df_args"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                           0               1\n",
              "0            adafactor_beta1            None\n",
              "1   adafactor_clip_threshold             1.0\n",
              "2       adafactor_decay_rate            -0.8\n",
              "3              adafactor_eps  (1e-30, 0.001)\n",
              "4    adafactor_relative_step            True\n",
              "..                       ...             ...\n",
              "90              wandb_kwargs              {}\n",
              "91             wandb_project            None\n",
              "92              warmup_ratio            0.06\n",
              "93              warmup_steps               0\n",
              "94              weight_decay             0.0\n",
              "\n",
              "[95 rows x 2 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-daba1c6b-2436-413a-b436-65f868230fe0\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>adafactor_beta1</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>adafactor_clip_threshold</td>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>adafactor_decay_rate</td>\n",
              "      <td>-0.8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>adafactor_eps</td>\n",
              "      <td>(1e-30, 0.001)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>adafactor_relative_step</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>90</th>\n",
              "      <td>wandb_kwargs</td>\n",
              "      <td>{}</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>91</th>\n",
              "      <td>wandb_project</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>92</th>\n",
              "      <td>warmup_ratio</td>\n",
              "      <td>0.06</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>93</th>\n",
              "      <td>warmup_steps</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>94</th>\n",
              "      <td>weight_decay</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>95 rows × 2 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-daba1c6b-2436-413a-b436-65f868230fe0')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-daba1c6b-2436-413a-b436-65f868230fe0 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-daba1c6b-2436-413a-b436-65f868230fe0');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fZKbpevQ7a-h"
      },
      "source": [
        "### Most Important Arguments\n",
        "\n",
        "Let's check the default parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lL-QZ_2w7fzL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "91b9e3f2-3d23-4e69-89bf-9674d4d79192"
      },
      "source": [
        "print(\"train epochs : \", model.args.num_train_epochs)\n",
        "print(\"max_seq_length : \", model.args.max_seq_length)\n",
        "print(\"train_batch_size : \", model.args.train_batch_size)\n",
        "print(\"learning_rate : \", model.args.learning_rate)\n",
        "print(\"model_name : \", model.args.model_name)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "train epochs :  1\n",
            "max_seq_length :  128\n",
            "train_batch_size :  8\n",
            "learning_rate :  4e-05\n",
            "model_name :  bert-base-uncased\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6QGPheuGXqIV"
      },
      "source": [
        "### Choose Hyperparameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4mHLBtrE-XTa"
      },
      "source": [
        "# These are the values we used in our Notebook:\n",
        "model.args.num_train_epochs = 4\n",
        "model.args.max_seq_length = 128\n",
        "model.args.train_batch_size = 32\n",
        "model.args.learning_rate = 2e-5"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wOz-aOhucHlf"
      },
      "source": [
        "### Configure Validation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dMSDFNtEcLIt"
      },
      "source": [
        "Periodically evaluate on our 10% validation set during training to monitor over-fitting.\n",
        "\n",
        "Calculate number of steps so we can specify how often to evaluate."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "W3h4WNIPcJe7"
      },
      "source": [
        "# Run evaluation periodically during training to monitor progress.\n",
        "model.args.evaluate_during_training = True\n",
        "\n",
        "# \"Print results from evaluation during training.\"\n",
        "model.args.evaluate_during_training_verbose = True\n",
        "\n",
        "# \"Perform evaluation at every specified steps.Checkpoint model and the evaluation results will be saved.\"\n",
        "model.args.evaluate_during_training_steps = 120\n",
        "\n",
        "# We only need to tokenize our validation set once, then we can read it from the cache.\n",
        "model.args.use_cached_eval_features = True"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AHM3fSwmXs39"
      },
      "source": [
        "### Configure Early Stopping"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OT1V5LmrXzwu"
      },
      "source": [
        "There's a nice intro to early stopping in their docs here:\n",
        "    \n",
        "* https://simpletransformers.ai/docs/usage/#using-early-stopping\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6m3mcaBnXjI_"
      },
      "source": [
        "# Turn on early stopping.\n",
        "model.args.use_early_stopping = True\n",
        "model.args.early_stopping_delta = 0.01\n",
        "model.args.early_stopping_metric = \"eval_loss\"\n",
        "model.args.early_stopping_metric_minimize = True\n",
        "# \"Terminate training after this many evaluations without an improvement in the\n",
        "#  evaluation metric greater then early_stopping_delta.\"\n",
        "model.args.early_stopping_patience = 2\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VHbXqbckfqEo"
      },
      "source": [
        "##Train"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XDTTZ6n7DECX",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 561,
          "referenced_widgets": [
            "ac9c48a50f0c478ab77d1f051d17bf33",
            "5f3c3d6b4dd540798289df58f654e1bd",
            "6178a8fff21a4040aa20ae6de1f462f8",
            "d34bda0c2d184ea1981be3fd0d4235d1",
            "567437c1d5514804b2c5e1e2c7aed1c5",
            "f302a54a62514c568824a1fa21159c60",
            "957d50af5b534bf6b7d7d56602bac662",
            "c8090b17bfa7482487ce8f056ab003cc",
            "9c58f67539414922935678617c24f230",
            "5e8a73b55d1c4890a1c8a58783b70a86",
            "26c5454bb548443f8b87f2125a89a6b5",
            "23b4f5f688cb4e15865a45c2e6619d57",
            "ef60cd9a29704439a98dbd094e080e35",
            "4ffadef1893c4ba4aec19af576f353a4",
            "0da4dafd7af84cf3a7b2f76908b3a976",
            "e02ac5faf1ba42978fcdb9c2660ecd6b",
            "6d09cdd72b224adcb505c375fd5d6b41",
            "f8834dd5d8c74330a367feb617940376",
            "33946fe5df734781904eb0148ce7cfc1",
            "0cec7d3006a94154b9e2e31244541deb",
            "8ff5a0e47f994881abf5f448646eb8c3",
            "83ef19d494364dcc89d6b9df3ac40d5e",
            "f61f4281ae904a6b8a220cadd2e7b577",
            "3b81c7a47c1a44e88dadfbe4173417c3",
            "11c026cd9eb1482ebd9702440c63fd4c",
            "30bb244d105f426385b7fa0173c717ba",
            "a61701e5fdde457c8745e28b1a6f3d3c",
            "8d4e10e1bd814f20ab35b3e3c844cff5",
            "e7a978b01a6f4c03a6e993deb3a02fe1",
            "b5fadd2aa986484aa36bcd38e90e5871",
            "3cdf9426fe67465f8e14a690939af449",
            "e67ae591701e4d85af3e57eda805a12b",
            "acd11d4c26c141a1a66c3a649f7cfd7a",
            "f413f435f6414477aae3a2f7852b94c2",
            "c1f11a7c1a8e4b66a450fdfae858719a",
            "cbb11761089c4460824587febaf4ad62",
            "140c152bb94a481084d59f1d7b09540b",
            "7fd0db0b0b084d5c88fdf8e27406910e",
            "6a4e752a97b94819b48562cdcedb35b6",
            "36041937ce184c9387bd7f190a6300e9",
            "93244fb56ce44d57a57b3898209e2a27",
            "2f2310cf4d594410a394a7831e4cf1e1",
            "a9882715342245839d7ea720602f8392",
            "4e86fd84bff54079a51b113f58d5ab48",
            "6590bc8a487e4ce1b2651d6838291625",
            "4d342afb5c3b4bafa2785ad21ea4bb70",
            "293ff68d77fa4cbd9e751244378d08d6",
            "a9ca7e847a494ae58daba9a1d5c53a76",
            "793aee45d3514104803d71511696ca5f",
            "ccbe1c01890c4d7491bed6e8f194007a",
            "583dfa136bc24b7983c0a358273f76d4",
            "499bc628cade4a33b2991027b428254a",
            "cb335f95a49b4ee39ae11ef90d019db1",
            "a0b32dc6ec2444f4bf3d093206f497b2",
            "bf4b90dc7c3d40fcaf47534338a24db8"
          ]
        },
        "outputId": "4288ad3e-6c6d-401e-bdd6-5fd2b1621a84"
      },
      "source": [
        "print('Training on {:,} samples...'.format(len(train_df)))\n",
        "\n",
        "# Train the model, testing against the validation set periodically.\n",
        "out = model.train_model(train_df, eval_df=validation_df)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training on 16,392 samples...\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:602: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/16392 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ac9c48a50f0c478ab77d1f051d17bf33"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:310: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
            "  FutureWarning,\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Epoch:   0%|          | 0/4 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "23b4f5f688cb4e15865a45c2e6619d57"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Epoch 0 of 4:   0%|          | 0/513 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f61f4281ae904a6b8a220cadd2e7b577"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/2049 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f413f435f6414477aae3a2f7852b94c2"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n",
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n",
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n",
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Epoch 1 of 4:   0%|          | 0/513 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "6590bc8a487e4ce1b2651d6838291625"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n",
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n",
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cLbK5JgBFkua"
      },
      "source": [
        "## Evaluate on Test Set\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "shHc3hNfh1OG"
      },
      "source": [
        "model = ClassificationModel(\n",
        "    \"bert\", \"outputs/best_model\"\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "apzWNC1pAHO7"
      },
      "source": [
        "**On Validation Set**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u24h6JjSFsJ_",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 190,
          "referenced_widgets": [
            "f3dd77a95bcb455a83cb72caa9566d20",
            "e980d8cef0794edbbd2354540580e526",
            "4d38d8e276594b0eb44423591d6b9fd3",
            "3a795f1ae48948cc83d6afe10ebc2ac2",
            "6a50ca0993ae4c5db7c69bd7c50259e6",
            "aaa1838545ce406bbc2779e9a9eeb956",
            "1b1a602aeaff40f289ba686c717b97f0",
            "e37744f05d2848ccb2022505109759cf",
            "300f8fcf106947c3a3653c40203f06c3",
            "c5177ae576db40449afaf6c9f356ca09",
            "3152f808dfdc4a67a0ac3b821e5c82b5"
          ]
        },
        "outputId": "f28dfed1-8e2d-44f5-bb41-fd876694be81"
      },
      "source": [
        "from sklearn.metrics import f1_score, accuracy_score\n",
        "def f1_multiclass(labels, preds):\n",
        "    return f1_score(labels, preds, average='micro')\n",
        "\n",
        "def conf_matrix(labels, preds):\n",
        "    return confusion_matrix(labels, preds)\n",
        "\n",
        "def class_report(labels, preds):\n",
        "    return classification_report(labels, preds, output_dict=True)\n",
        "    \n",
        "result, val_outputs, wrong_predictions = model.eval_model(validation_df, f1=f1_multiclass, cm=conf_matrix, acc=accuracy_score, clf_report = class_report)\n",
        "print(result)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Evaluation:   0%|          | 0/257 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f3dd77a95bcb455a83cb72caa9566d20"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'mcc': 0.49195203202937837, 'f1': 0.6500732064421669, 'cm': array([[ 94,  30,   5,   1,   1],\n",
            "       [ 44,  64,  63,  16,   3],\n",
            "       [  6,  17, 114,  78,   6],\n",
            "       [  0,   2,  49, 344, 202],\n",
            "       [  0,   1,   8, 185, 716]]), 'acc': 0.6500732064421669, 'clf_report': {'0.0': {'precision': 0.6527777777777778, 'recall': 0.7175572519083969, 'f1-score': 0.6836363636363636, 'support': 131}, '1.0': {'precision': 0.5614035087719298, 'recall': 0.3368421052631579, 'f1-score': 0.4210526315789474, 'support': 190}, '2.0': {'precision': 0.4769874476987448, 'recall': 0.5158371040723982, 'f1-score': 0.49565217391304356, 'support': 221}, '3.0': {'precision': 0.5512820512820513, 'recall': 0.576214405360134, 'f1-score': 0.5634725634725636, 'support': 597}, '4.0': {'precision': 0.771551724137931, 'recall': 0.7868131868131868, 'f1-score': 0.7791077257889011, 'support': 910}, 'accuracy': 0.6500732064421669, 'macro avg': {'precision': 0.602800501933687, 'recall': 0.5866528106834548, 'f1-score': 0.5885842916779638, 'support': 2049}, 'weighted avg': {'precision': 0.6485223206822255, 'recall': 0.6500732064421669, 'f1-score': 0.6464014860576704, 'support': 2049}}, 'eval_loss': 0.7763789432985774}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GbE5RYSfAKM8"
      },
      "source": [
        "**On Test Set**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NsTOCd8fALl5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 190,
          "referenced_widgets": [
            "d62d0eae60c24dd1bdaabc848f64099a",
            "b0122c640eaf46eca82ec1b75d7a8bd4",
            "55e5e97490e94fdaa3a1a72a4afa791b",
            "fb90d4f34d0c4727aeb3079347b22a25",
            "742e344fe62c4d69ae76913c46a8410e",
            "6b68c609cc0c46209dec753376e492e2",
            "a575ffdc913e4fe2951ba60f5e58e05f",
            "a42f2a799d51473594de3f9593a7470c",
            "c5dd2eebaae54e528f3e1231b70d2e2f",
            "52f457766fb940f6b97529ccdb8c91ae",
            "d064c05f03f440b8914d2d86042e5170"
          ]
        },
        "outputId": "f65fb87e-66b8-46ed-e9e3-d96dfc1bf1ed"
      },
      "source": [
        "result, test_outputs, wrong_predictions = model.eval_model(test_df, f1=f1_multiclass, cm=conf_matrix, acc=accuracy_score, clf_report = class_report)\n",
        "print(result)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Evaluation:   0%|          | 0/257 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "d62d0eae60c24dd1bdaabc848f64099a"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'mcc': 0.49195203202937837, 'f1': 0.6500732064421669, 'cm': array([[ 94,  30,   5,   1,   1],\n",
            "       [ 44,  64,  63,  16,   3],\n",
            "       [  6,  17, 114,  78,   6],\n",
            "       [  0,   2,  49, 344, 202],\n",
            "       [  0,   1,   8, 185, 716]]), 'acc': 0.6500732064421669, 'clf_report': {'0.0': {'precision': 0.6527777777777778, 'recall': 0.7175572519083969, 'f1-score': 0.6836363636363636, 'support': 131}, '1.0': {'precision': 0.5614035087719298, 'recall': 0.3368421052631579, 'f1-score': 0.4210526315789474, 'support': 190}, '2.0': {'precision': 0.4769874476987448, 'recall': 0.5158371040723982, 'f1-score': 0.49565217391304356, 'support': 221}, '3.0': {'precision': 0.5512820512820513, 'recall': 0.576214405360134, 'f1-score': 0.5634725634725636, 'support': 597}, '4.0': {'precision': 0.771551724137931, 'recall': 0.7868131868131868, 'f1-score': 0.7791077257889011, 'support': 910}, 'accuracy': 0.6500732064421669, 'macro avg': {'precision': 0.602800501933687, 'recall': 0.5866528106834548, 'f1-score': 0.5885842916779638, 'support': 2049}, 'weighted avg': {'precision': 0.6485223206822255, 'recall': 0.6500732064421669, 'f1-score': 0.6464014860576704, 'support': 2049}}, 'eval_loss': 0.7763789432985774}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "result  {'mcc': 0.49468331260672477, 'eval_loss': 0.775075949583536}"
      ],
      "metadata": {
        "id": "nRyI0elovGAW"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KkM5QS7-BWrU"
      },
      "source": [
        "##  Over-Fitting "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6O_NbXFGMukX",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 332
        },
        "outputId": "1258e992-0e11-4d08-886f-56dea99aa425"
      },
      "source": [
        "import pandas as pd\n",
        "pd.set_option('precision', 2)\n",
        "\n",
        "# Create a DataFrame from our training statistics.\n",
        "df_stats = pd.read_csv(\"/content/outputs/training_progress_scores.csv\")\n",
        "\n",
        "# Use the 'epoch' as the row index.\n",
        "df_stats = df_stats.set_index('global_step')\n",
        "df_stats"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "             train_loss   mcc  eval_loss\n",
              "global_step                             \n",
              "120                1.24  0.25       1.17\n",
              "240                0.83  0.44       0.86\n",
              "360                1.06  0.43       0.85\n",
              "480                0.85  0.49       0.79\n",
              "513                0.84  0.49       0.78\n",
              "600                0.73  0.49       0.79\n",
              "720                0.72  0.49       0.79\n",
              "840                1.17  0.50       0.78"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-34878928-1944-4e2a-b800-6274a52cdf2f\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>train_loss</th>\n",
              "      <th>mcc</th>\n",
              "      <th>eval_loss</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>global_step</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>120</th>\n",
              "      <td>1.24</td>\n",
              "      <td>0.25</td>\n",
              "      <td>1.17</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>240</th>\n",
              "      <td>0.83</td>\n",
              "      <td>0.44</td>\n",
              "      <td>0.86</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>360</th>\n",
              "      <td>1.06</td>\n",
              "      <td>0.43</td>\n",
              "      <td>0.85</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>480</th>\n",
              "      <td>0.85</td>\n",
              "      <td>0.49</td>\n",
              "      <td>0.79</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>513</th>\n",
              "      <td>0.84</td>\n",
              "      <td>0.49</td>\n",
              "      <td>0.78</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>600</th>\n",
              "      <td>0.73</td>\n",
              "      <td>0.49</td>\n",
              "      <td>0.79</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>720</th>\n",
              "      <td>0.72</td>\n",
              "      <td>0.49</td>\n",
              "      <td>0.79</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>840</th>\n",
              "      <td>1.17</td>\n",
              "      <td>0.50</td>\n",
              "      <td>0.78</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-34878928-1944-4e2a-b800-6274a52cdf2f')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-34878928-1944-4e2a-b800-6274a52cdf2f button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-34878928-1944-4e2a-b800-6274a52cdf2f');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1-G03mmwH3aI"
      },
      "source": [
        "Validation Loss is a more precise measure than accuracy, because with accuracy we don't care about the exact output value, but just which side of a threshold it falls on. \n",
        "\n",
        "If we are predicting the correct answer, but with less confidence, then validation loss will catch this, while accuracy will not."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "68xreA9JAmG5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 427
        },
        "outputId": "1277a060-5493-45be-e4d6-30d232826cd2"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "# Increase the plot size and font size.\n",
        "sns.set(font_scale=1.5)\n",
        "plt.rcParams[\"figure.figsize\"] = (12,6)\n",
        "\n",
        "# Plot the learning curve.\n",
        "plt.plot(df_stats['train_loss'], 'b-o', label=\"Training\")\n",
        "plt.plot(df_stats['eval_loss'], 'g-o', label=\"Validation\")\n",
        "\n",
        "# Label the plot.\n",
        "plt.title(\"Training & Validation Loss\")\n",
        "plt.xlabel(\"Epoch\")\n",
        "plt.ylabel(\"Loss\")\n",
        "plt.legend()\n",
        "plt.xticks([1, 2, 3, 4])\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 864x432 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuUAAAGaCAYAAACopj13AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xV9f0/8Ncdyc3ekyRkQfZgkwABCRAgAUFEoSKogOLga2trW7XaVvto+9NqFQdagboQkDKVJSOMAElYkpDFCNkkNyF73pt77/n9QUkNCXADuTn3Jq/nX95zzufcVy7x5n0/9zMkgiAIICIiIiIi0UjFDkBERERENNCxKCciIiIiEhmLciIiIiIikbEoJyIiIiISGYtyIiIiIiKRsSgnIiIiIhIZi3Ii6rdKS0sRHByMjz766J7v8corryA4OLgXU/Vft3u9g4OD8corr+h1j48++gjBwcEoLS3t9Xzbtm1DcHAw0tPTe/3eRET3Sy52ACIaOHpS3B46dAje3t4GTGN6Wlpa8Nlnn2HPnj2orKyEk5MTRo4cieeffx6BgYF63ePFF1/Ejz/+iB07diA0NLTbawRBwJQpU9DQ0IDjx4/DwsKiN38Mg0pPT8epU6fwxBNPwM7OTuw4XZSWlmLKlClYtGgR/vjHP4odh4iMCItyIuoz77zzTqfHZ8+exXfffYcFCxZg5MiRnc45OTnd9/N5eXkhMzMTMpnsnu/xl7/8BW+++eZ9Z+kNr7/+Onbv3o1Zs2ZhzJgxqKqqQnJyMjIyMvQuyufPn48ff/wRW7duxeuvv97tNWlpaSgrK8OCBQt6pSDPzMyEVNo3X8yeOnUKH3/8MR566KEuRfmcOXOQlJQEMzOzPslCRNQTLMqJqM/MmTOn02OtVovvvvsOw4YN63LuVk1NTbCxsenR80kkEigUih7n/DljKeBaW1uxb98+TJgwAe+9917H8ZUrV0KtVut9nwkTJsDT0xM//PADfve738Hc3LzLNdu2bQNwo4DvDff7b9BbZDLZfX1AIyIyJI4pJyKjEx8fj8WLFyMnJwfLli3DyJEj8eCDDwK4UZy///77eOSRRzB27FhERERg2rRpePfdd9Ha2trpPt2Ncf75scOHD+Phhx9GZGQkJkyYgLfffhsajabTPbobU37zWGNjI/70pz8hNjYWkZGRWLhwITIyMrr8PLW1tXj11VcxduxYDB8+HEuWLEFOTg4WL16M+Ph4vV4TiUQCiUTS7YeE7grr25FKpXjooYdQV1eH5OTkLuebmpqwf/9+BAUFISoqqkev9+10N6Zcp9PhX//6F+Lj4xEZGYlZs2bh+++/77Z9fn4+/vznPyMpKQnDhw9HdHQ05s2bh//85z+drnvllVfw8ccfAwCmTJmC4ODgTv/+txtTXlNTgzfffBOTJk1CREQEJk2ahDfffBO1tbWdrrvZPjU1FevWrcPUqVMRERGB6dOnY/v27Xq9Fj2Rl5eHF154AWPHjkVkZCQSExOxZs0aaLXaTteVl5fj1VdfxeTJkxEREYHY2FgsXLiwUyadTocvv/wSs2fPxvDhwzFixAhMnz4dr732Gtrb23s9OxH1HHvKicgoXbt2DU888QRmzJiBhIQEtLS0AACUSiW2bNmChIQEzJo1C3K5HKdOncLatWuRm5uLdevW6XX/o0ePYsOGDVi4cCEefvhhHDp0CP/+979hb2+PZ599Vq97LFu2DE5OTnjhhRdQV1eHL774As888wwOHTrU0auvVqvx1FNPITc3F/PmzUNkZCQuXryIp556Cvb29nq/HhYWFpg7dy62bt2KXbt2YdasWXq3vdW8efPw6aefYtu2bZgxY0anc7t370ZbWxsefvhhAL33et/q73//O77++muMHj0aTz75JKqrq/HWW2/Bx8eny7WnTp3CmTNn8MADD8Db27vjW4PXX38dNTU1WLFiBQBgwYIFaGpqwoEDB/Dqq6/C0dERwJ3nMjQ2NuIXv/gFioqK8PDDDyMsLAy5ubnYuHEj0tLS8J///KfLNzTvv/8+2trasGDBApibm2Pjxo145ZVXMHjw4C7DsO7VhQsXsHjxYsjlcixatAguLi44fPgw3n33XeTl5XV8W6LRaPDUU09BqVTiscceg5+fH5qamnDx4kWcOXMGDz30EADg008/xYcffojJkydj4cKFkMlkKC0tRXJyMtRqtdF8I0Q0oAlERCLZunWrEBQUJGzdurXT8cmTJwtBQUHC5s2bu7RRqVSCWq3ucvz9998XgoKChIyMjI5jJSUlQlBQkPDhhx92ORYdHS2UlJR0HNfpdEJSUpIwfvz4Tvf9/e9/LwQFBXV77E9/+lOn43v27BGCgoKEjRs3dhxbv369EBQUJKxevbrTtTePT548ucvP0p3Gxkbh6aefFiIiIoSwsDBh9+7derW7nSVLlgihoaGCUqnsdPzRRx8VwsPDherqakEQ7v/1FgRBCAoKEn7/+993PM7PzxeCg4OFJUuWCBqNpuN4VlaWEBwcLAQFBXX6t2lubu7y/FqtVnj88ceFESNGdMr34Ycfdml/083ft7S0tI5j//znP4WgoCBh/fr1na69+e/z/vvvd2k/Z84cQaVSdRyvqKgQwsPDhZdeeqnLc97q5mv05ptv3vG6BQsWCKGhoUJubm7HMZ1OJ7z44otCUFCQcPLkSUEQBCE3N1cICgoSPv/88zveb+7cucLMmTPvmo+IxMPhK0RklBwcHDBv3rwux83NzTt69TQaDerr61FTU4Nx48YBQLfDR7ozZcqUTqu7SCQSjB07FlVVVWhubtbrHk8++WSnxzExMQCAoqKijmOHDx+GTCbDkiVLOl37yCOPwNbWVq/n0el0+OUvf4m8vDzs3bsXEydOxMsvv4wffvih03VvvPEGwsPD9RpjPn/+fGi1WuzYsaPjWH5+Ps6fP4/4+PiOiba99Xr/3KFDhyAIAp566qlOY7zDw8Mxfvz4LtdbWVl1/LdKpUJtbS3q6uowfvx4NDU14erVqz3OcNOBAwfg5OSEBQsWdDq+YMECODk54eDBg13aPPbYY52GDLm7u8Pf3x+FhYX3nOPnqqur8dNPPyE+Ph4hISEdxyUSCZ577rmO3AA6fofS09NRXV1923va2NhAqVTizJkzvZKRiHofh68QkVHy8fG57aS8b7/9Fps2bcKVK1eg0+k6nauvr9f7/rdycHAAANTV1cHa2rrH97g5XKKurq7jWGlpKdzc3Lrcz9zcHN7e3mhoaLjr8xw6dAjHjx/HP/7xD3h7e2PVqlVYuXIlfve730Gj0XQMUbh48SIiIyP1GmOekJAAOzs7bNu2Dc888wwAYOvWrQDQMXTlpt54vX+upKQEABAQENDlXGBgII4fP97pWHNzMz7++GPs3bsX5eXlXdro8xreTmlpKSIiIiCXd/5zKJfL4efnh5ycnC5tbve7U1ZWds85bs0EAEOGDOlyLiAgAFKptOM19PLywrPPPovPP/8cEyZMQGhoKGJiYjBjxgxERUV1tPv1r3+NF154AYsWLYKbmxvGjBmDBx54ANOnT+/RnAQiMhwW5URklCwtLbs9/sUXX+D//b//hwkTJmDJkiVwc3ODmZkZlEolXnnlFQiCoNf977QKx/3eQ9/2+ro5MXH06NEAbhT0H3/8MZ577jm8+uqr0Gg0CAkJQUZGBv7617/qdU+FQoFZs2Zhw4YNOHfuHKKjo/H999/Dw8MDcXFxHdf11ut9P37zm9/gyJEjePTRRzF69Gg4ODhAJpPh6NGj+PLLL7t8UDC0vlreUV8vvfQS5s+fjyNHjuDMmTPYsmUL1q1bh+XLl+O3v/0tAGD48OE4cOAAjh8/jvT0dKSnp2PXrl349NNPsWHDho4PpEQkHhblRGRSdu7cCS8vL6xZs6ZTcXTs2DERU92el5cXUlNT0dzc3Km3vL29HaWlpXptcHPz5ywrK4OnpyeAG4X56tWr8eyzz+KNN96Al5cXgoKCMHfuXL2zzZ8/Hxs2bMC2bdtQX1+PqqoqPPvss51eV0O83jd7mq9evYrBgwd3Opefn9/pcUNDA44cOYI5c+bgrbfe6nTu5MmTXe4tkUh6nKWgoAAajaZTb7lGo0FhYWG3veKGdnNY1ZUrV7qcu3r1KnQ6XZdcPj4+WLx4MRYvXgyVSoVly5Zh7dq1WLp0KZydnQEA1tbWmD59OqZPnw7gxjcgb731FrZs2YLly5cb+Kciorsxro/7RER3IZVKIZFIOvXQajQarFmzRsRUtxcfHw+tVouvv/660/HNmzejsbFRr3tMmjQJwI1VP34+XlyhUOCf//wn7OzsUFpaiunTp3cZhnEn4eHhCA0NxZ49e/Dtt99CIpF0WZvcEK93fHw8JBIJvvjii07L+2VnZ3cptG9+ELi1R76ysrLLkojA/8af6zusZurUqaipqelyr82bN6OmpgZTp07V6z69ydnZGcOHD8fhw4dx6dKljuOCIODzzz8HAEybNg3AjdVjbl3SUKFQdAwNuvk61NTUdHme8PDwTtcQkbjYU05EJmXGjBl477338PTTT2PatGloamrCrl27elSM9qVHHnkEmzZtwgcffIDi4uKOJRH37dsHX1/fLuuid2f8+PGYP38+tmzZgqSkJMyZMwceHh4oKSnBzp07AdwosD755BMEBgZi5syZeuebP38+/vKXvyAlJQVjxozp0gNriNc7MDAQixYtwvr16/HEE08gISEB1dXV+PbbbxESEtJpHLeNjQ3Gjx+P77//HhYWFoiMjERZWRm+++47eHt7dxq/DwDR0dEAgHfffRezZ8+GQqHA0KFDERQU1G2W5cuXY9++fXjrrbeQk5OD0NBQ5ObmYsuWLfD39zdYD3JWVhZWr17d5bhcLsczzzyDP/zhD1i8eDEWLVqExx57DK6urjh8+DCOHz+OWbNmITY2FsCNoU1vvPEGEhIS4O/vD2tra2RlZWHLli2Ijo7uKM4TExMxbNgwREVFwc3NDVVVVdi8eTPMzMyQlJRkkJ+RiHrGOP+KERHdxrJlyyAIArZs2YK//vWvcHV1xcyZM/Hwww8jMTFR7HhdmJub46uvvsI777yDQ4cOYe/evYiKisKXX36JP/zhD2hra9PrPn/9618xZswYbNq0CevWrUN7ezu8vLwwY8YMLF26FObm5liwYAF++9vfwtbWFhMmTNDrvrNnz8Y777wDlUrVZYInYLjX+w9/+ANcXFywefNmvPPOO/Dz88Mf//hHFBUVdZlc+Y9//APvvfcekpOTsX37dvj5+eGll16CXC7Hq6++2unakSNH4uWXX8amTZvwxhtvQKPRYOXKlbctym1tbbFx40Z8+OGHSE5OxrZt2+Ds7IyFCxfi//7v/3q8i6y+MjIyul25xtzcHM888wwiIyOxadMmfPjhh9i4cSNaWlrg4+ODl19+GUuXLu24Pjg4GNOmTcOpU6fwww8/QKfTwdPTEytWrOh03dKlS3H06FF88803aGxshLOzM6Kjo7FixYpOK7wQkXgkQl/M0iEiok60Wi1iYmIQFRV1zxvwEBFR/8Ex5UREBtZdb/imTZvQ0NDQ7brcREQ08HD4ChGRgb3++utQq9UYPnw4zM3N8dNPP2HXrl3w9fXFo48+KnY8IiIyAhy+QkRkYDt27MC3336LwsJCtLS0wNnZGZMmTcIvf/lLuLi4iB2PiIiMAItyIiIiIiKRcUw5EREREZHIWJQTEREREYmMEz1vUV3dJHYEIiIiIuqHpFIJHB2tuz3HovwWOh2H2BMRERFR3+LwFSIiIiIikbEoJyIiIiISGYtyIiIiIiKRsSgnIiIiIhIZi3IiIiIiIpGxKCciIiIiEhmLciIiIiIikbEoJyIiIiISGYtyIiIiIiKRcUdP6hOp2RXYdjQf1Q0qONspMG9SIGLDPcSORURERGQUWJSTwaVmV+CrvXlQa3QAgOoGFb7amwcALMyJiIiIwOEr1Ae2Hc3vKMhvUmt02HY0X6RERERERMaFRTkZXHWDqkfHiYiIiAYaFuVkcM52ih4dJyIiIhpoWJSTwc2bFAhzeedfNXO5FPMmBYqUiIiIiMi4cKInGdzNyZw3V18BgFmxvpzkSURERPRfLMqpT8SGeyA23APNbe146aMTqGtWix2JiIiIyGhw+Ar1KWsLM4wKdkVathLqdq3YcYiIiIiMAoty6nNxUZ5oUWlw9lKV2FGIiIiIjAKLcupzwb6OcLG3QErGNbGjEBERERkFFuXU56QSCeKiByGvuA6VtS1ixyEiIiISHYtyEsWESE9IJEBKZrnYUYiIiIhEx6KcROFoq0BkgDNOXCiHVqcTOw4RERGRqFiUk2jiogahrkmNrKs1YkchIiIiEhWLchJN9BBn2FmZ4RgnfBIREdEAx6KcRCOXSTEuwhOZ+dWo52ZCRERENICxKCdRxUV7QqsTcDKLEz6JiIho4JKL+eSVlZX4+uuvkZGRgaysLLS0tODrr7/G2LFj79hOp9Nh+/btOHDgAHJzc1FfXw9vb2/MmjULS5cuhbm5eR/9BHS/PJ2tMcTbHscyyjFjzGBIJBKxIxERERH1OVF7ygsKCrBmzRoolUoEBwfr3a61tRWvvfYaamtrsXDhQrz22muIjIzEqlWr8MwzzxgwMRlCXJQnlDUtuFxaL3YUIiIiIlGI2lMeHh6OtLQ0ODo64uDBg3jhhRf0amdmZoaNGzdixIgRHcceffRReHl54aOPPkJ6evpde9vJeIwOccPGg5eRknkNQT4OYschIiIi6nOi9pTb2NjA0dGxx+3Mzc07FeQ3TZs2DQCQn59/39mo71iYyzEm1B2n8yrRqtKIHYeIiIioz/WriZ7Xr18HgHsq9ElccdGeULfrkJ6rFDsKERERUZ/rV0X52rVrYWtriwkTJogdhXoowNMOXq7WSMngKixEREQ08Ig6prw3ffbZZzh58iTeeust2Nra3vN9XF3vvS3dn5nj/LF2ZxaaNQL8PO3EjkNERETUZ/pFUb5nzx588MEHWLBgARYsWHBf96qqauylVNRTkb4OkEkl2HnkMh6bGiR2HCIiIqJeJZVK4Oxs0/25Ps7S606cOIHf/e53mDx5Mv70pz+JHYfug62VOUYEuSI1qwLtGp3YcYiIiIj6jEkX5RkZGVi5ciUiIyPx/vvvQyaTiR2J7lNctCea2zT46XKV2FGIiIiI+oxJFOXFxcUoLi7udCw/Px/PPPMMvLy88Nlnn8HCwkKkdNSbwvyc4GynQErGNbGjEBEREfUZ0ceUr169GsD/1hbfuXMnzp49Czs7Ozz++OMAgCeffBIAkJycDABoamrCsmXL0NDQgGXLluHIkSOd7hkcHIyQkJC++QGoV0klEkyIGoTvjxfgel0rXBwsxY5EREREZHCiF+WrVq3q9Hjr1q0AAC8vr46i/FZ1dXUoL7+xdN57773X5fzKlStZlJuwCZGe+P54AY5fKMfcuACx4xAREREZnEQQBEHsEMaEq68Yh/e+O4/y6ma88+w4SKUSseMQERER3bd+vfoK9U8TowehpkGFnMIasaMQERERGRyLcjJKw4a4wMbSDMc44ZOIiIgGABblZJTM5FLEhnvgp8vX0dCiFjsOERERkUGxKCejFRftCa1OQFpWhdhRiIiIiAyKRTkZLW9XGwQMssOxzHJwPjIRERH1ZyzKyajFRXni2vVmXL3WIHYUIiIiIoNhUU5GbUyoOxRmMqRkcsInERER9V+ibx5EA8OpinP4Pn8falV1cFQ44MHAGRjjMeKu7SwVcowOcUN6biUWThkKC3P+yhIREdH9S82uwLaj+ahuUMHZToF5kwIRG+4hWh72lJPBnao4hw15W1GrqgMA1KrqsCFvK05VnNOrfVy0J1RqLU7nVhoyJhEREQ0QqdkV+GpvHqobVACA6gYVvtqbh9Rs8RaXYFFOBvd9/j6069o7HWvXteP7/H16tR/iZQ9PZyukZJYbIh4RERENMNuO5kOt0XU6ptbosO1ovkiJWJRTH7jZQ67v8VtJJBLERQ3ClbJ6XLve3JvRiIiIaAC62UOu7/G+wKKcDM5R4dCj490ZF+EBmVTCCZ9ERER035ztFD063hdYlJPBPRg4A2ZSs07H5BIZHgycofc97KzNMWyIC05mVUCj1d29AREREdFtzJsUCKlU0umYuVyKeZMCRUrEopz6wBiPEXgs5OGOnnEppLAzt8Vo9+E9uk9ctCcaW9px/vJ1Q8QkIiKiAWJsqDsUcinM5DdKYWc7BZ6YGSLq6itcX476xBiPER1LIKZeO431ef9B5vUcRLuG632PCH9nONoqkJJZjlEhboaKSkRERP1cblEtWtVaPD83wmhqCvaUU58b4zECbpYu2F2wHzpB/6EoUqkE4yM9kVVQjZqGNgMmJCIiov4sNbsClgoZooc4ix2lA4ty6nMyqQwz/aeirKkc56uyetR2QpQnBAE4foHLIxIREVHPqdq1OHupCiOD3WAml4kdpwOLchLFKPdh8LByw+6CAz3qLXdzsESoryOOZ5ZDJwgGTEhERET9UcaV61CptaKOH+8Oi3IShVQiRaL/NFQ0K3FOmdGjtnHRnrhe34a8oloDpSMiIqL+KjWrAo62CgQP1n9p5r7AopxEM9wtEoOsPbC78AC0Oq3e7UYGucLaQo5jGVyznIiIiPTX2KJGVkENxoa5QyqR3L1BH2JRTqKRSqRICkhAZct1nFGe17udmVyGmDAPnLt0HU2t7QZMSERERP3J6bxKaHUCYsLcxY7SBYtyElW0Szh8bAZhT0HPesvjoj2h0eqQll1hwHRERETUn6RlK+Hlag0fNxuxo3TBopxEJZFIkBSQgOttNUivOKt3u8HutvD1sMWxjHIInPBJREREd1FZ14orZfWICXOHxMiGrgAsyskIRDiHwtfOB3sLD0Gj0+jdbmKUJ0qrmlBY0WjAdERERNQfpP/32/WxRjh0BWBRTkZAIpFgln8CatpqkVp+Wu92Y8PcYS6XIiWTa5YTERHR7QmCgNRsJYJ8HOBibyl2nG6xKCejEOoUhAB7X+wrTEa7Vr/Jm1YWZhgZ7Ib0nAqo2vUfj05EREQDS5GyERU1LYgJN85ecoBFORmJG73l01GnqseJa6f0bjcx2hOtKi3O5FUaMB0RERGZsrRsJeQyCUaHuIkd5bZYlJPRCHIMxFCHAPxYlAy1Vq1fGx8HuDlacggLERERdUunE5Ceo0RkgDOsLczEjnNbLMrJaEgkEiT5J6BB3YiUsjS928RFeeJSSR0qaloMnJCIiIhMTW5RLeqb1YgN9xA7yh2xKCejMtQxACGOQ7G/6DDaNCq92oyP9IRUIkFKJnf4JCIios5SsytgqZAjeoiz2FHuiEU5GZ2kgAQ0tTfjWNlJva53sFEgKtAZJy9UQKPVGTgdERERmQpVuxZnL1VhVLArzOQysePcEYtyMjoB9r4Icw7GwaKjaNW06dUmLtoT9c1qXMivNnA6IiIiMhXnL1+HSq1FjJEPXQFYlJORmuWfgGZNC46UnNDr+qhAZ9hbm3PCJxEREXVIy66Ao60CwYMdxI5yVyzKySj52vkg0iUMh0qOoaW99a7Xy6RSjI/0RGZ+NWob9RuLTkRERP1XY4saWQU1GBvmDqlEInacu2JRTkYryT8BrZpWJJek6HV9XJQndIKAk1nsLSciIhroTudVQqsTEBNmvBsG/RyLcjJaPraDMMw1EodLUtDU3nzX692drBDk44CUzHIIgtAHCYmIiMhYpWUr4eVqDR83G7Gj6IVFORm1JP9pUGnVOFR8TK/rJ0Z7orK2FZdK6gycjIiIiIxVZV0rrpTVIybMHRITGLoCsCgnIzfIxgMj3KJwpPQEGtVNd71+ZLAbLBUyHMvgmuVEREQDVXp2BQAgJsz4V125iUU5Gb1E/2lo17bjQPGRu16rMJNhbJgHzlysQktbu+HDERERkVERBAGp2UoE+TjA2d5C7Dh6Y1FORs/D2g2jPYbjWGkq6lWNd71+YrQn2jU6pOco+yAdERERGZMiZSMqaloQG24aEzxvYlFOJmGm31RoBS32FyXf9Vpfd1v4uNngWAZXYSEiIhpoUrOUkMskGBXiJnaUHmFRTibBzcoFYz1G4vi1dNS23XkSp0QiQVyUJ4qUjShW3r1nnYiIiPoHnU7AqVwlIgOcYW1hJnacHmFRTiZjpt8U6AQdfiw6fNdrY8I9IJdJkcLeciIiogEjt6gW9c1qxIabzgTPm1iUk8lwtnTCOM/ROHntFKpba+94rY2lGUYGuyI1uwLqdm0fJSQiIiIxpWZXwFIhR/QQZ7Gj9JioRXllZSXeffddLF68GMOHD0dwcDDS09P1anv8+HG89tprmD17NkJDQxEfH2/gtGQMZvhNgQTAvsJDd702LsoTLSoNzl2qMnwwIiIiEpWqXYuzl6owKtgVZnKZ2HF6TNSivKCgAGvWrIFSqURwcHCP2u7atQu7du2CtbU13N1Na3Yt3TtHCweM9xqLtIozqGqpvuO1Ib6OcLG3QEomh7AQERH1d+cvX4dKrUWMCQ5dAUQuysPDw5GWlob9+/dj+fLlPWr70ksv4ezZs9i0aRPCwsIMlJCMUYLvZMgkUuwtPHjH66T/nfCZW1SLytqWPkpHREREYkjLroCjrQLBgx3EjnJPRC3KbWxs4OjoeE9t3d3dYWZmWrNqqXc4KOwR5xWLUxXnoGy589CU8ZGekEiA4xfYW05ERNRfNbaokVVQg7Fh7pBKJGLHuSec6EkmKcF3MsykcuwpOHDH65zsLBAZ4IwTFyqg1en6KB0RERH1pdN5ldDqBJNcdeUmFuVkkmzNbTDJezzOKjNQ3nznnTvjojxR26hC1tWaPkpHREREfSk1uwJertbwcbMRO8o9k4sdwNi4utqKHYH0tMAuCSnXUnHw2mH8etzTt71uiqM11h+4hFMXqzA11r8PExIREZGhVVQ3I7+sAUsSQ026jmNRfouqKu4AaUoe8J6AfYWHcL7gErxsPG97XWyYBw6cKcGVwmrYW5v3YUIiIiIypD0nCgAAkb6ORl/HSaUSODt335vP4Stk0qb4xMFSboHdV/ff8bq4aE9odQJOZnHCJxERUaT2va4AACAASURBVH8hCAJSs5UI8nGAs72F2HHuC4tyMmlWZlaI94lDxvVsFDeU3vY6T2drDPGyR0pGOQRB6MOEREREZChFykZU1LQgNtz096wxiaK8uLgYxcXFYscgIzXZZwKs5JbYXXD33vKKmhZcKavvo2RERERkSKlZSshlEowKcRM7yn0TfUz56tWrAQD5+fkAgJ07d+Ls2bOws7PD448/DgB48sknAQDJyckd7fLy8joeFxYWorGxseNeo0ePxujRo/vqRyCRWcotMXXwJHx/dR8K6ovhbz+42+tGh7hhw8HLOJZxDUO9TXNjASIiIrpBq9PhVK4SUYEusLYw/b1rRC/KV61a1enx1q1bAQBeXl4dRXl3cnJyurS9+XjlypUsygeYSd7jkVySgt0F+7FyWPe7w1qYyzE21A1pOUo8NjUIlgrRf/2JiIjoHuUW1aK+WY2YMNMfugIYQVF+8eLFu17z8x7ym+bNm4d58+YZIhKZIAu5AtN8H8D2K7txpa4AQxy6X/owLnoQjmWU41SuEpOGefVxSiIiIuotadlKWCrkiB7iLHaUXmESY8qJ9DHRKxa25jZ3XIklwNMOXi7WOJbBVViIiIhMlapdi7OXqjAq2BVmcpnYcXoFi3LqN8xl5pjuG49Ldfm4VHul22skEgniojxRUN6A0qqmPk5IREREveH85etQqbWICfcQO0qvYVFO/cqEQWNhb26HXVf333bpw9gID8ikEqSwt5yIiMgkpWVXwNFWgeDB/WfhBhbl1K+Yycwwwy8e+fWFyKu93O01tlbmGB7kipNZ5WjX6Po4IREREd2PxhY1sgpqMDbMHVKJROw4vYZFOfU7sYPGwFHhcMfe8olRnmhu0+Cny1V9nI6IiIjux+m8Smh1AmL70dAVgEU59UNmUjlm+k1BYUMxsqvzur0mzM8JznYKpGRyCAsREZEpSc2ugJerNXzcbMSO0qtYlFO/FOM5Cs4WTthd0H1vuVQqwfhIT+QU1OB6fasICYmIiKinKutakV/W0O96yQEW5dRPyaQyzPSfiuLGMmRez+n2mglRngCA4+wtJyIiMglp2RUAgLGh/WPDoJ9jUU791hj34XCzdMHugv3QCV0ndLrYWyLM3wknLpRDp+t+7DkREREZB0EQkJatRLCPA5ztLcSO0+tYlFO/dbO3vKypHOersrq9Ji7KE9UNKuQU1vRxOiIiIuqJImUjKmpaEBPe/3rJARbl1M+Nch8GDys37C440G1v+fChrrCxNMMxDmEhIiIyaqlZSshlEowKcRM7ikGwKKd+TSqRItF/GiqalTinzOhy3kwuRWy4B366VIXGFrUICYmIiOhutDodTuUqERXoAmsLM7HjGASLcur3hrtFYpC1B3YXHoBWp+1yPi7aE1qdgNSsChHSERER0d3kFtWivlmNmLD+OXQFYFFOA4BUIkVSQAIqW67jjPJ8l/Perjbw97RDSmb5bTcbIiIiIvGkZSthqZAjeoiz2FEMRi52AKK+EO0SDh+bQdhTcACj3IdBJpV1Oj8x2hNf7buIq+UNCBxkL1JKMlWp2RXYdjQf1Q0qONspMG9SYL9cQ5eISAyqdi3OXqrCmBA3mMlld29gothTTgOCRCJBUkACrrfVIL3ibJfzY0LdYW4mRUrGNRHSkSlLza7AV3vzUN2gAgBUN6jw1d48pGZzOBQRUW84f/k6VGptv+/sYFFOA0aEcyh87Xywt/AQNDpNp3OWCjlGh7ghPbcSbWrNbe5A1NW2o/lQazqv7KPW6LDtaL5IiYiI+pfU7Ao42ioQNNhB7CgGxaKcBgyJRIJZ/gmoaatFavnpLucnRg+CSq3F6bxKEdKRqbrZQ67vcSIi0l9jixrZBTUYG+YOqUQidhyDYlFOA0qoUxAC7H2xrzAZ7dr2TueGeNnDw8kKKRlcs5z0Z2fV/dJcznaKPk5CRNT/nM6rhFYn9PuhKwCLchpgbvSWT0edqh4nrp3qci4u2hNXyupx7XqzSAnJlLSqNNB1s2KPTCrBvEmBIiQiIupfUrMr4OVqDR83G7GjGByLchpwghwDMdQhAD8WJUOt7bxh0LgIT8ikEhznDp+kh42HLqO5TYMHx/t19IwrzKTQ6gRYKbi4FRHR/aisa0V+WcOA6CUHWJTTACSRSJDkn4AGdSNSytI6nbO3Nkf0EBecyCqHRqu7zR2IgJ8uVeF4ZjkSY3wxNy4A/3h+PP79SjxWvRiHwW42WLsrB1V1rWLHJCIyWWn/XcVqbGj/3TDo51iU04A01DEAIY5Dsb/oMNo0nSfkxUV5orGlHRlXrouUjoxdQ7MaX+7Lw2B3G8yZ4N/pnLmZDM/Pi4QgAKu3Z6Fd03UXWSIiujNBEJCWrUSwjwOc7S3EjtMnWJTTgJUUkICm9mYcKzvZ6XhEgBMcbRVI4RAW6oYgCPhybx5aVVo8PTscclnXt1E3B0ssnxWGImUjvj1wSYSURESmrbCiERU1LYgJHxi95ACLchrAAux9EeYcjINFR9Gqaes4LpNKMT7SAxeuVqOmoe0Od6CBKCWzHOevXMf8SQHwcrG+7XXDhrogKdYXxzLKuSkVEVEPpWUrIZdJMCrETewofYZFOQ1os/wT0KxpwZGS452OT4gaBEEATlxgbzn9T2VdKzYeuoyQwQ6YOtrnrtc/FBeAUF9HrD9wCUUVjX2QkIjI9Gl1OqTnKhEV6AJri+6Xne2PWJTTgOZr54Mol3AcKjmGlvb/Tcpzc7BEqK8jUjLLu13yjgYenU7Aul05kEqAZUlhem1iIZVKsGJOOGwszbB6xwU0t7XftQ0R0UCXW1SLhmY1YsIGztAVgEU5EZL8p6FV04bkkmOdjsdFeeJ6fRvyimpFSkbGZN+pYlwurceiaUE9mnRkZ2WO5+dGoKZBhbU/5PBDHhHRXaRlK2GpkCN6iLPYUfoUi3Ia8LxtB2G4ayQOlxxHU/v/Ng0aEeQKK4WcEz4JxcpGbD92FaOCXe9pvdxAL3ssnDIUGfnV2JNaZICERET9g6pdi7OXqjAq2BVmcpnYcfoUi3IiAIn+06DSqnGo+H+95eZmMsSGe+DsxSo0tXLYwUDVrtFi7a4c2FiaYcmMEEj0GLbSnfgRXhgb5o7tKVeRXVjTyymJiPqH85evQ6XWDpgNg36ORTkRgEE2HhjpHo0jpSfQqG7qOB4X7QmNVtexgQENPNuPFaC0qhlPJYbCxvLeJxxJJBI8OSMEg5yt8a+d2VzZh4ioG6nZFXC0VSBosIPYUfoci3Ki/0r0m4p2bTsOFB3pODbY3Ra+7rZIySyHwLHAA87F4lr8eKoYDwz3QlTg/Y9tVJjL8PxDEWjX6vDpjizuGktE9DMNLWpkF9QgJsxdr8n0/Q2LcqL/crd2wxiPEThWdhL1qoaO4xOjPVFS2YQiJZe0G0haVRqs3ZULV0dLLJg8pNfu6+lsjWWJoci/1oDvDl3ptfsSEZm607mV0OoExAzAoSsAi3KiTmb4TYFW0GF/0eGOY2PD3GEml+JYBid8DiQbDl5CTWMbnp4VBoV57042GhXihoTRPjh0rpRDo4iI/istpwLertbwcbMRO4ooWJQT/YyblQtiPEbi+LV01LbVAQCsLMwwKtgV6TkVULVrRU5IfeHcpSqcuFCBpFg/BHrZG+Q55j8QiCBve3y5Lw9lVU13b0BE1I9V1rYgv6xhwPaSAyzKibqY4TcFgiDgx5/1lk+MHoRWlRZnL1aKmIz6Qn2zGl/uzYOvuy0eHO9nsOeRy6R4dm4ELMzl+Hh7FlpVGoM9FxGRsUvLUQIAxoYOrA2Dfo5FOdEtnC2dEDtoNE5eO4Xq1hsbBwX5OMDN0ZJDWPo5QRDw5Z5ctKm1WD47DHKZYd8iHWwUeG5OOKpqW/HvPbmcTExEA5IgCEjLViLYx6FHm7P1NyzKiboxwzceEgD7Cg8BuLGcXVyUJy6V1EFZ0yJuODKYlMxyZORX45EHAuHlYt0nzxk82BHzHwjE2YtV2H+6pE+ek4jImBRWNKKipgUx4QO3lxxgUU7ULUcLB4z3ikFaxRlUtVQDAMZFeEIqkXCHz36qsrYFGw9eRqivI6aM8u7T554+xgcjg1zxn8P5uFRS16fPTUQktrRsJeQyCUaFuIkdRVQsyoluY7rvZMgkUuwtPAgAcLRVICrQGSculEOr4/rS/YlOJ2Dt7lxIpRIsSwrt8/VxJRIJliaFwtXBAp/uyEJdk6pPn5+ISCxanQ7puUpEBbrA2uLeN2jrD1iUE92GvcIOcV6xOFVxDsqWKgBAXJQn6pvVyMyvFjkd9aa96UW4UlqPxxOC4GQnznhGS4UcL8yLRKtag8+4sRARDRC5RbVoaFYjdoAPXQFYlBPdUYLvZJhJ5dhTcAAAEBnoDHtrc6Rwwme/UaxsxI6UAowKcUNMmLh/FLxdbfDEjBBcKq3HtqNXRc1CRNQXUrOUsFTIe2XXZFPHopzoDmzNbTDJezzOKjNQ3qyEXCbFuEgPZOZXc4hBP9Cu0WLNDzmwsTLDkunBkBjBts6x4R6YPMIL+04VcwlOIurXVO1anLtchdEhrjCT9+4mbaZI1KK8srIS7777LhYvXozhw4cjODgY6enperfPz8/HsmXLMHz4cIwZMwa///3vUVNTY8DENBBNHTwJ5jIz7P5vb3lc1CDoBAEnLrC33NRtO3YVZdeb8dTMUNhYGs9YxoXxQxEwyA7rdueigqv9EFE/df7ydajUWsSEDdwNg35O1KK8oKAAa9asgVKpRHBwcI/aVlRUYNGiRSgpKcFLL72EpUuX4vDhw1i2bBna29sNlJgGIhtza0z2icNPlZkoayqHh5MVgnwckJJZznWlTVheUS32nyrB5OFeRve1qZlciufnRkAuk+KT7RegUnMnWSLqf1KzK+Boq0DQYAexoxiFXinKNRoNfvzxR2zevBlVVVV6twsPD0daWhr279+P5cuX9+g5P/vsM6hUKnzzzTdYsmQJnn32WXzwwQfIycnBzp07e/ojEN3RFJ84WMotsPvqfgA3JnxW1rZy+ToT1dKmwbrdOXBztMSjk4eIHadbTnYWWPFgOK5VNeOrfXn8AEhE/UpDixrZBTWICXPv8xWvjFWPi/J33nkHDz/8cMdjQRDw1FNP4Ve/+hX++Mc/Yvbs2SguLtbrXjY2NnB0dOxpBADA/v37ER8fD3f3/03MGjduHPz8/LB37957uifR7ViZWSHeJw4Z17NR3FCKUSFusFTIuMOnidp48BJqGlVYPisMCnPjHccY7u+EuXH+SMtR4vBPZWLHISLqNadzK6HVCYgJ59CVm3pclKekpGDUqFEdj5OTk3H69GksW7YM7733HgDg888/772E3VAqlaiurkZERESXc1FRUcjNzTXo89PANNlnAqzklthdsB8KMxnGhnng7MVKtLRxuJQpOXuxEieyKjAr1g+BXvZix7mrpHF+iAp0xsaDl5F/rV7sOEREvSItpwLertbwcbMRO4rR6HFRXlFRAV9f347Hhw8fhre3N15++WUkJSVh4cKFSE1N7dWQt6qsvLEigaura5dzrq6uqK6uhlbLMZjUuyzllpg6eBKyqvNQUF+MuChPqDU6pOcoxY5GeqpvUuGrfRfh62GL2eP9xI6jF6lEgqdnh8HRVoHV27PQ0KIWOxIR0X2prG1BflkDe8lvIe9pg/b2dsjl/2uWnp6OcePGdTz28fHp0bjye6FS3ViKztzcvMs5hUIBAGhra4O1tXWP7+3qant/4ahfm+8wHYfLjuNA2SG8NvH/4Odph9QcJR6dHip2NLoLQRCwemc21O1a/H7JaHi6m87/664A/rB0LH73UQq+3HsRf34mFjIpx2ASkWk6dP4aACBxQiBcHS1FTmM8elyUe3h44KeffsKjjz6Ky5cvo6SkBC+++GLH+erqalhZWfVqyFvdLLzV6q49RjcLdguLe9uVr6qq8d6D0YAw1WcStl/ZjbQrFzAu3B0bDl7G2axrGGxCRd5AdPR8Gc7kKvGLqUNhITW9/9ftFTIsmhaEL/fmYe32TMybGCB2JCKiHhMEAQdPFSPYxwHQaEzuvfh+SaUSODt3P2Snx8NXkpKSsGPHDqxYsQIrVqyAjY0NJk2a1HE+NzcXgwcPvve0enBzcwOAbnvkq6qq4OzsDJnMeCdvkWmb6BULW3Mb7L66HzHhHpDLpNzh08hV1rZg06ErCPV1xJSR3mLHuWcTowchLsoTu04WIuPKdbHjEBH1WGFFI5Q1LYiN4NCVW/W4KF+xYgUeeughnD9/HhKJBG+//Tbs7OwAAI2NjUhOTkZsbGyvB/05d3d3ODk5ISsrq8u5zMxMhIZyKAEZjrnMHNN943GpLh/X2oowIsgFaTkVaNdwHoMx0ukErN2VC6lUgmVJoSa/9NaiaUEY7G6DNT/koLKuVew4REQ9kpathFwmwajgrvMCB7oeF+Xm5ub429/+hvT0dBw6dAhTpkzpOGdtbY3jx49j5cqVvRqyuLi4yzKLCQkJSE5OhlL5v0l2qampKCwsxIwZM3r1+YluNWHQWNib22HX1f2Ii/JEc5sGZy8Zdi4F3Zu96UW4UlaPxQlBcLK7t2FtxsTcTIbnH4oEAKzefgHqdn4YJCLToNXpkJ6rRFSgC6wsjGcXZWPR4zHld6LRaGBr27NxtatXrwYA5OfnAwB27tyJs2fPws7ODo8//jgA4MknnwRwY/nFm5599lns27cPS5YsweOPP46WlhasW7cOISEhmDNnTi/8NES3ZyYzwwy/eHx3aQckftfhYm+BlIxybhVsZIoqGrEjpQCjQ9wwNsz97g1MhJuDJZbPDsOHWzLx7YFLeCqR3w4SkfHLLapFQ7MaseH95/24N/W4p/zo0aP46KOPOh379ttvMWLECAwbNgy/+c1verTN/apVq7Bq1Srs2rULALB161asWrUK//73v+/YztPTE+vXr4e3tzfee+89rF27FpMmTcIXX3zR7aosRL0tdtAYOCocsLvgAMZHeiC3qJbDCYxIu0aLNbtyYGNlhsXTgyEx8WErtxo2xAWzxvkiJbMcxzKuiR2HiOiuUrOUsFTIERXoLHYUo9TjnvJ169bB2fl/L2Z+fj7+9re/wcfHB97e3tizZw8iIyM7erfv5uLFi3e95uc95D83dOhQrFu3Tq/nIeptZlI5ZvpNwYaLWxE3pAkSCXA8s5yrYhiJrUev4tr1Zvz60WjYWPbPr0nnTgjA1WsNWL//EnzdbeHrwRWAiMg4qdRanLtchbGhbjCTczGO7vS4p/zq1auddtLcs2cPFAoFtmzZgrVr1yIxMRE7duzo1ZBExirGcxScLZxwVHkY4f5OOHGhHDqdIHasAS+3qBb7T5dg8ggvRAT03x4ZqVSCZx4Mh62VGT7ZfgFNrdxdloiM009XqqBSaznM8w56XJTX19fD0dGx4/HJkycRExMDG5sbay6OGTMGpaWlvZeQyIjJpDLM9J+K4sYy+A5tQW2jClkF1WLHGtBa2jRYtzsH7o6WePSBIWLHMTg7K3M8PzcCtY0qrN2VA53AD4VEZHzSspVwtFUgaLCD2FGMVo+LckdHR1y7dmP8YlNTEy5cuIBRo0Z1nNdoNNzingaUMe7D4Wbpgjx1Gmys5FyzXGQbDl5CXaMay2eHQWE+ML4iDfSyx8IpQ5GZX43dqUVixyEi6qShRY2sqzWICXM3+WVpDanHY8qHDRuGTZs2YciQITh27Bi0Wi0mTpzYcb6oqKhjcx+igeBmb/lXOZsQGtGK82e0qG9Ww96aE4772pm8SpzMqsCD4/0QOMhe7Dh9Kn6EF/Kv1WPHsasI8LRDuL+T2JGIiAAAp3MroRMExIZz6Mqd9Lin/MUXX4ROp8OvfvUrbNu2DXPnzsWQITe+IhYEAQcPHsSIESN6PSiRMRvlPgweVm6oUmRCq9MhNatC7EgDTl2TCl//eBF+HraYNc5P7Dh9TiKR4InpIRjkYo1/fZ+N6vo2sSMREQEA0nIq4O1qDW+37reXpxt6XJQPGTIEe/bswerVq/HNN9/g73//e8e5hoYGPPHEE3jiiSd6NSSRsZNKpEj0n4brqioMGlKPlMxrEDi2t88IgoAv9+ZB1a7F07PDIJf1+K2tX1CYy/D8QxHQaHVYvSML7Rqd2JGIaICrrG1BflkDYthLflf39JfLwcEB8fHxGD16dKfj9vb2eOKJJxASEtIr4YhMyXC3SAyy9oDG5SLKq5twpaxe7EgDxtHz15CZX41HHgiEp7O12HFE5elsjaWJoSgob8B3yZfFjkNEA1xazo2d12P60QZuhnLPO3oWFxfj0KFDKCkpAQD4+PhgypQpGDx4cK+FIzIlUokUSQEJWHPhayjclUjJKMdQb84yNzRlbQs2JV9GmJ8j4kd6ix3HKIwKccOMMYOx71QxAr3sOY6TiEQhCAJSs5UI9nGAk52F2HGM3j0V5R988AHWrFnTZZWVf/zjH1ixYgV++ctf9ko4IlMT7RIOH5tBqBx8FafOe+IXU4fCUnHPn33pLrQ6HdbuyoFcKsXSxFDO6v+Zhx8IwNXyBny1Nw8+rjYcy0lEfa6wohHKmhbMHMsOW330ePjKli1b8NlnnyEqKgqffPIJ9u/fj/379+OTTz7BsGHD8Nlnn2Hbtm2GyEpk9CQSCZICEqCSNELrUIJTuUqxI/Vre9KKkV/WgMenB7EX5hYyqRTPzgmHpUKOT7ZfQKtKI3YkIhpgUrMrIJdJMCrYVewoJqHHRfmGDRsQHR2Nb775pmO4yuDBgzFlyhR8/fXXiIqKwvr16w2RlcgkRDiHwtfWBxbeBTiWyY20DKWoohHfHy/AmFA37hB3Gw42Cjw3NwJVdW349+5cTj4moj6j1elwKrcS0YEusLIwEzuOSehxUZ6fn4/ExETI5V2/kpfL5UhMTER+fn6vhCMyRRKJBLMCEqAza0GxJhelVU1iR+p31O1arNmVA1srMzyeECx2HKMW5OOARyYH4uylKvx4qkTsOEQ0QOQW1aKhWY2YcE7w1FePi3IzMzO0tLTc9nxzczPMzPiJiAa2UKcg+NoOhtmgfBzNYCHU27YevYpr15uxNCkUNpZ8v7mbhNE+GBnsii1H8nGxuFbsOEQ0AKRmKWGlkCMq0FnsKCajx0V5ZGQkvvvuO1y/fr3LuerqamzevBnR0dG9Eo7IVEkkEswJnAGJuQqpFae4XnQvyi2swYEzJYgf4YUIf77Z60MikWBpYihcHS3x6c5s1DWpxI5ERP2YSq3FuctVGBXiCjO5TOw4JqPHRfnzzz+PqqoqJCYm4u2338bWrVuxdetWvP3220hMTMT169fx3HPPGSIrkUkJdhqCQRaDoXO5gjOXysWO0y+0tLVj3Z5cuDtZ4ZHJQ8SOY1IsFXK88FAE2tQafLojCxotPygSkWH8dKUKKrWW8316SPbnP//5zz1p4OXlhdDQUKSmpiIlJQXJyclITk7G+fPn4ezsjLfffhvjxo0zUFzDa2lRix2B+hEPaxekV6WjtEKN+OBIseOYvC/35iG/rAEvPRoNF3tLseOYHDtrc7jYW+DAmVKoNVp+00BEBrHlSD7UGh1+MXUoJFyqthOJRAIrK/Nuz93TAsrx8fF44IEHkJWVhdLSG6tL+Pj4IDw8HJs3b0ZiYiL27Nlz74mJ+okgpwA4SbxRrchGWU0dvJy4mdC9OpNXidRsJR4c7wd/Tzux45ismHAP5Jc14MdTJQgcZI9RIW5iRyKifqShRY2sqzWYPsaHe0f0UI+Hr3Q0lEoRFRWFxMREJCYmIjIyElKpFLW1tSgoKOjNjEQmbV7QDEjM1PjPhUNiRzFZdU0qfLUvD/6etpg1zk/sOCZvwZQhCBhkh3V7clFe3Sx2HCLqR07nVkInCNxJ+B7cc1FORPoZ7hUES9UgXFafQ7O6Vew4JkcQBHyxJw/tGh2WzwqDXMa3rfsll0nx/NwImMmk+GR7FtrU3FiIiHpHWnYFvF2tuYvwPeBfN6I+ED9oMiBrx3+yDogdxeQcOX8NF65W45HJQ+DpbC12nH7Dyc4CK+aEo7y6GV/tu8iNhYjovlXWtiD/WgN7ye8Ri3KiPjAtPBKod8fZ2nS0tLO3XF/KmhZ8l3wZ4f5OmDzCS+w4/U64nxMeigtAeo4SyefKxI5DRCYuLUcJCYCxYdww6F6wKCfqA2ZyKYbZjoNO0o69Vw+LHcckaHU6rNmVAzOZFEsTQzlhyEASY30RHeiMTYcu40pZvdhxiMhECYKA1Gwlggc7wMnOQuw4Jkmv1Ve++OILvW947ty5ew5D1J/NjIrE2WMncQwnMD1gEmzMOBTjTvakFuHqtQaseDAcjrYKseP0W1KJBMtnh+HNL07j0x1Z+NNTo2F3m+W6iIhup7CiEcqaFswcO1jsKCZLr6L87bff7tFNuSYlUVfebjZwUw1DtfAjDhYdxdwhiWJHMlqFFQ34/kQhxoa582vQPmBtYYYXHorE39afxb92ZuM3C4ZBKuX7OBHpLzW7AnKZBKOCXcWOYrL0Ksq//vprQ+cgGhDiw0Ow8XImjkhPYMrgibA15+z0W6nbtVjzQw7srM3xeEKQ2HEGDF8PWzyeEIQv9uRhe8pVPDwpUOxIRGQitDodTuVWIjrQBVYWZmLHMVl6FeVjxowxdA6iAWFsqDs2HR+KdqcKHCg6gnlDZ4kdyehsOZqP8uoW/GbBMFjzzb1PxUUNQn5ZPXanFiFwkD2GDXUROxIRmYDcwlo0NKsRE85vNu8HJ3oS9SFLhRyj/QMg1A7C0bKTqFc1iB3JqOQU1uDgmVJMGemNcH8nseMMSIumBcHX3RZrduWgso4rBRHR3aVmK2GlkCMq0FnsKCaNRTlRH4uLGgRVSQC0Oi32F3Ellpta2tqxbncuPJysMP8BDp0Qi5lchucfioAEwOptF6Bu14odiYiMmEqtxblLVRgV4gozuUzsCbVXpAAAIABJREFUOCaNRTlRHxvqbQ93a1dYNvvh+LV01LbViR3JKKw/cAn1TWo8PTsMCjO+sYvJ1cEST88OQ3FlE9bvv8SNhYjotn66UgVVu5YbBvUCFuVEfUwikSAu2hM1l32gE3T4kb3lOJWrRFq2Eg+O94O/p53YcQhA9BAXzB7nh+MXypGSWS52HCIyUmnZSjjZKTDUx0HsKCaPRTmRCMZFeEKqsYa7EIyT106hurVW7EiiqW1U4ZsfL8Lf0w5J43zFjkM/M2eCP8L9HLF+/yUUVnD+AxF11tCiRtbVGowNdecGb72ARTmRCOytzREV6Iyqize2jt9XeEjkROIQBAFf7MlFu0aHp2eHQSblW5IxkUoleObBcNhZm2H19iw0tbaLHYmIjMjp3EroBIFDV3oJ/wISiWRi9CA01ssRbBWNtIozqGqpFjtSnzvyUxmyCmrwaPwQeDhZiR2HumFrZY7n5kagtlGFtbtyoOP4ciL6r7TsCni7WsPbjXtu9AYW5UQiiQhwgoONOVpL/CCTSLH3/7N332FV3vf/x59nH/aSDSKigqCiuKPGmDhwZmnMaGabNmY0o2lG29/3m7ZZTdJommGbpPnW7MQVZ9TELDVaVBAFURTBBeeAMg7z7N8fKIrgQIGb8X5clxeH+5z7Pm+5GK/zOe/P51PwrdIltStTaQ1ffHeQAbGBTBgSqXQ54gLiIvy4fWJfduedZPXPBUqXI4ToAIrLasgrtMgoeSuSUC6EQjRqNWMGhpOTV8OI4BGkmdIx15QoXVa7cLpcvLdqLzqtmnun9UclvYgd3jVDIhmdFMqKTflk5Xe/d3WEEI1tyzajAkYmyoZBrUVCuRAKGjcoHLcb9OX90Km1rM3/RumS2sWarYfJL7Jw55R4AnwMSpcjLoFKpeKuKQlEBHvx7sq9nKyoU7okIYRC3G43W/eaie/pT6CvUelyugwJ5UIoKCTAk4Se/qTtLufqqDHsNGdSVG1Wuqw2lV9kYeXmAkYlhjKiv4ywdCYGvYaHbhyI0+Xina/2YHe4lC5JCKGAAlMl5tIaRknrSquSUC6EwsYlR3Cioo4YVTJ6jY41XXi03GZ38v7qvfh567ljcj+lyxGXISzQk/umJZJfVMnnGw8oXY4QQgFbs01oNSqGxQcrXUqXIqFcCIUN7ReMp0HLjuxyJkSPI6N4N8eruuZmLUt+yKPoZA33Te+Pl1GndDniMg2NDyZ1ZE++zzjOz1ld83tVCNE8p8tFWk4xyXE98JTf461KQrkQCtPrNIxKCmXH/hJGhYzGQ2tkzaENSpfV6rLzS/l25zEmDo0iqVeg0uWIK3Tz+N70i/bnw3X7OVZcpXQ5Qoh2klNQhqXaxqgkaT9sbRLKhegAxg2KwOF0sXu/hWujx5F5IpsjlmNKl9VqquvsfLA2h/AgT2ZfE6d0OaIVaNRq5l2fhIdBy1vL91BT51C6JCFEO9iabcbToGVQXJDSpXQ5EsqF6ABiwnyICfXhp8xCrokai6fWgzX5XWe0/JMNuViqbfxqRiJ6nUbpckQr8fM2MO+GAZwor+Pfa/bilo2FhOjSrDYn6bklDEsIRqeV3+WtTUK5EB3EuORwjhZXUXzSzsSe48k6uY/8iiNKl3XF0nLMbNtrZuaYXsSG+ypdjmhl/aL9uWVCHBkHTrAurfN/vwohzi/jYAlWu1M2DGojEsqF6CBGJYai06rZlFnE+KgxeOu8Ov1oeVmllY/W76d3hC/TR8coXY5oI5OGRzMsPpglP+Sx73CZ0uUIIdrItmwzgb4G+kb7K11Kl6RoKLfZbLz66quMHTuWQYMGccstt7B169ZLOverr75i5syZDBw4kLFjx/L8889TXV3dxhUL0XY8jTqGxQezba8ZlVvLpJhryCnN5WB5vtKlXRa3280Ha3OwO138akYiGrWMAXRVKpWKe6f1JzTAk3+uzKas0qp0SUKIVmapsZF1qJSRiaGoZRfmNqHoX8lnnnmGRYsWMWvWLP74xz+iVqu5//77ycjIuOB5ixYt4umnnyY4OJhnnnmGm266iSVLlvDggw9KT6Po1MYNiqDW6mDn/mKujhyNj967067E8l36cbLzS5k7oQ9hgZ5KlyPamIdBy0M3DqDO5mDhiiwcTtlYSIiuZHtOMS63m9GJ0rrSVhQL5bt372bNmjU8+eSTPPXUU8ydO5dFixYRHh7Oa6+9dt7zbDYbb775JqNGjeLf//43d9xxB0888QTz589n27ZtbNy4sR3/F0K0rvie/oT4e7Apswi9Rs+UmGvJLc8jt+yg0qW1SNHJahZ/f5ABvQO5Zkik0uWIdhIZ7M09UxM4eKyCJT/kKV2OEKIVbcs2ERXsTVSIt9KldFmKhfJ169ah0+mYM2dOwzGDwcDs2bPZuXMnxcXFzZ534MABKisrmTZtGqqz3j6ZMGECnp6erF27ts1rF6KtqFQqxiWHs/9oOebSGsZGjMRP78vqQxs6zbtADqeL91fvRadVc+/U/o1+TkXXNyoxjOuGRrFh+1G272v+97gQonMpLqshr9DCaFmbvE0pFspzcnKIjY3Fy8ur0fFBgwbhdrvJyclp9jybzQbUB/hzGY1GsrOzW79YIdrRVQPCUalg0+4idBodqb2uJa+igH1lnWNL8zVbD5NfVMldqQkE+DT9ORVd39xr+xAX4csHa3MoOilzfYTo7LZlm1EBIxMllLclxUJ5SUkJISEhTY4HBwcDnHekPCYmBpVKRXp6eqPjhw4dorS09LznCdFZBPgYGNQ7iC1ZRThdLkZHjCDA4N8pRsvziyys2lLAqKRQhic0/fkW3YNWo2beDQPQa9W8tWwPdTbZWEiIzsrtdrN1r5n4nv4E+hqVLqdL0yr1xHV1deh0uibHT4+AW63Nz94PDAxk6tSpLF26lN69e3PddddhNpv561//ik6nO+95lyo42OeKzheiNcy4Oo4X/i+NIydqGZEUxpyB03l3xycccxwhJWKA0uU1q87m4IN/pxHoa+DR24bi7dH051t0H8HBPjx953D+592f+fy7PJ78xVBpZRKiE8o9Uoa5tIY51/WTjNTGFAvlRqMRu93e5PjpUN1ce8ppf/nLX6irq+Oll17ipZdeAmDWrFn07NnzkpdUPJ+SksorOl+I1hDTwxNfLz2rfsojNsSLAd4DCDIG8umur4jS9uyQ4eaTDbkcL6niyVsHU1tVR21VndIlCYVFBBi58ereLP3xEFE9PJk4LFrpkoQQLfT1lkNoNSoSIn0kI7UCtVpFUFDzk2UVC+XBwcHNtpqUlJQANNvacpqPjw8LFy6ksLCQ48ePExERQWRkJLfeeisxMbJBiej8tBo1YwaEsT7tKOVVVvy9DUyNncjHOV+y+0Q2ycEda7Q8O7+UjenHmDgsisRegUqXIzqQqaNiyDtu4YvvDtIr3Jc+kX5KlySEuEROl4u0vWaS43rgaZR3P9uaYj3lCQkJ5OfnN9nwJzMzs+H+i4mIiGD48OFERkZisVjIyspi9OjRbVKvEO1tXHIELrebn7NMAIwIHUKIRw/W5H+Dy91x1oCuqrXz7zV7CQ/yZPb4OKXLER2MWqXilzP6E+hrYOFXWViqbUqXJIS4RDkFZVhq7IxKkrXJ24NioTw1NRW73c7ixYsbjtlsNpYtW0ZKSgqhofUzfAsLC8nLu/h6t3//+99Rq9XMnTu3zWoWoj2FBXrSL8qPTZmFuN1uNGoNU2MncryqiF0lWUqX1+DjDfuprLHz65lJ6HUapcsRHZCXUcdDNw6kqtbOP1dk4XR1nBeVQojz25ptxtOgZVBckNKldAuKta8kJyeTmprKa6+9RklJCT179mT58uUUFhY29IkDPP3006SlpbF///6GYwsXLiQvL4/k5GQ0Gg0bN25k8+bN/OUvfyE6WnoWRdcxLjmCf6/JIfdoOfE9AxgWOpj1Bd+xJv8bBgcPQK1Sduv6/+41k5ZTzI3jYokJkwlA4vx6hvpw5+R4Plibw1eb8rlZ3lURokOz2pyk55YwMjEUnVbZvzXdhaJf5VdeeYU777yTFStW8Pzzz+NwOHj33XcZOnToBc+Lj4+noKCA119/nVdffRWr1cp7770no+SiyxkWH4KHQcOm3UUAqFVqpsVOwlRtZqc5U9HayiqtfLR+P3ERvkwbLXM5xMWNHRTO1ckRrNl6mIwDJUqXI4S4gIyDJVjtTtkwqB2p3B194eN2JjOLRUfz4bp9/Jxl4vWHx+Jp1OJyu3gpbQEOt4M/jfgdGnX7t4y43G7mf7GLA8cr+PN9IwgN8Gz3GkTnZHc4efGjdIrLa/mfe4bJ944QHdSCxZkcK6nilXlXoe6AK351VhdafUXejxCigxuXHIHN4eK/OWagfrR8eu/JFNecYLs5Q5Gavk8/TnZBGXOv7SuhSrSITqvhwRsHoFbB28uysNqdSpckhDiHpcZG1qFSRiaGSiBvRxLKhejgeoX5EBXszabMwoZjyT2SiPaO4Ov8b3G62jfUFJ2sZvH3BxnYO4hrBke063OLriHY34P7ZyZxvKSKj9fv7/A71QrR3WzPKcbldjM6UVZdaU8SyoXo4FQqFeOSwykwVXLEXNlwbHrvyZyoK2WbaUe71eJwunhv1V70Og33TkvokJsYic5hUFwQM8f0YkuWiZ/OesEphFDetmwTUcHeRIU032Yh2oaEciE6gdFJYWg1qoYJnwADgvoT4xvN1/kbcbgc7VLH6p8LKDBVcteUePy9z7/rrhCXYtaYWJJiA/nkm1zyiyxKlyOEAMxlNeQVWmSCpwIklAvRCXh76EjpF8y2bBN2R327ikqlYkbsZMqs5fxcuL3NazhUaGH1z4cZnRTGsITz77grxKVSq1X8emYifl563lmeRVWtXemShOj2/pttRgWMTJRQ3t4klAvRSYxLjqC6zsHO3DNLyfUP7Edvv16sP/wddmfbBRqr3cl7q/fi76Pnjkn92ux5RPfj46nnwRsHUlFt5d1V2bikv1wIxbjdbrbuNRPf059AX6PS5XQ7EsqF6CT6xwTQw8/IpswzLSynR8vLrRVsLvxvmz334u8PYi6t4ZfT+uNpVGzPMdFFxYb7ctvEfmQdKmX1lgKlyxGi2yowVWIurWFUkkzwVIKEciE6CbVKxdhB4eQcLqOkvLbheHxgH/r692bD4e+xOW2t/rxZh07yXfpxJg+Ppn+vwFa/vhAA1wyOYHRSGCs257Pn0EmlyxGiW9qabUKrUTEsPljpUrolCeVCdCJjB4ajgkYTPgFm9J6CxVbJT8e3turzVdXa+ffaHCJ6eHHz+N6tem0hzqZSqbgrNZ7IYC/eXZnNiYrai58khGg1TpeLtL1mkuN64GnUKV1OtyShXIhOJNDXSFLvQLbsKcLlOtN728c/loSAvnxz+AfqHNZWe76PN+ynqsbO/TMS0Wnbf+dQ0b0YdBoeunEgLrebd5ZnYXe4lC5JiG4jp6AMS41dWlcUJKFciE7m6kERlFVaycovbXR8Ru/JVNmr+enYz63yPNv2mkjLKeb6sbHEhPm0yjWFuJjQQE9+OT2RAlMln208oHQ5QnQbW7NNeBq0DIoLUrqUbktCuRCdzOC+PfDx1DXa4RMg1i+GpKAEvj3yI7WOuit6jlJLHR+vzyUu0pepo3pe0bWEaKmUfsFMHdmTHzKOs2VP0cVPEEJcEavNSXruCYYlhKDTSjRUinzlhehktBo1o5PC2HXwBJbqxhM7Z8ROptpRww9HN1/29V1uNx+szcHpcvOrGYlo1PJrQrS/m8b3Jj7anw/X72/YyVYI0TYyDpZgtTtlwyCFyV9bITqhcckROF1ufs4yNTre0zeKQT2S2Hj0J2rslzdR7rudx9hbUMbca/sQGuDZGuUK0WIatZoHrk/C06jlneVZ1NTJxkJCtJVt2WYCfQ30jfZXupRuTUK5EJ1QZA8v4iJ92bS7EPc5m61Mj51EraOO//fzSzz03VP8acuLpJnSL+m6hSeqWfxDHoPighg/OKItShfikvl5G3jwhgGctNTx7zU5Tb7XhRBXzlJjI+tQKSMTQ1GrVEqX061JKBeikxo3KIKikzXkHbc0Ol5YbUKFijpnfV95mbWcT/ctvWgwdzhdvLd6LwadhnunJqCSX86iA+gb5c+cCX3IOHCCr/97ROlyhOhytucU43K7GS2rrihOtuYTopMa0T+EzzYe4KfdhfSJ8ms4vjJvHW4ajyjaXXY+3beE7JP78NR64KH1wENrrL+tq7+dtqeUI2Ul3Dt5IF6esvyh6DgmDYvi4PEKlv6YR2y4L/1jApQuSYguY1u2iahgb6KCvZUupduTUC5EJ2XUaxmREEJaTjG3XdcXD0P9j3OZtbzZx9tdDg5bjlLrqKPGUYvL3XQNaGMyfGbexGdm0Kt19eFd54Gn1tg4yGs98DwV5j20HqeONb6tUUuwF61DpVJx79QEjpdU8a8VWfzvvSMI8DEoXZYQnZ65rIa8QgtzrolTuhSBhHIhOrVxyRFs2l3E9n3FXJ1c3wMeYPBvNpgHGPx5bvTTALjdbmwuO7WOWipqqnlrZTpOrNx0bQxOlY1aRy01jlpq7bXUOOqoddRSaavEXFNCraOWWkdds6H+bHqN/pywfjrYn7p9TqhvGMHXGfHQSKgXjXkYtDx440CeX7SDv32SjsPlotRiJcjXwE3j4+StdyEuw3+zzaiAkYmy6kpHIKFciE4sLsKXiB5ebMosbAjls+JS+XTfUuyuM6tV6NQ6ZsWlNnyuUqkwaPQYNHpW/WjiZJEnv79tDP2jL60twO12Y3XaGgJ6jaO28W173anPz4R6i60SU3Vxw2PObbE5l0GjbzIKXz9C33TUvtFtCfVdVmQPL8YMDOO79OMNx05arCz6eh+ABHMhWsDtdrM120R8T38CfY1KlyOQUC5Ep6ZSqRg3KJwvvjvI8ZIqIoO9GRGWAtT3lpdZywkw+DMrLrXh+Nn2HDrJ9+nHmTw8ukV9uiqVCqPWgFFr4HK6e+tDvfWsQF8f3GvsZ90++7ijjgqbBVO1+ZJDvVFjaAjsjcN8M6P2p8L82SFfrbr0efBppvRL+nqLK5d58ESTYzaHiyU/5EkoF6IFCkyVmMtqmToqRulSxCkSyoXo5EYPCGPJD3ls2l3Erdf1BWBEWMpFQ2FVrZ0P1uYQ2cOLm8f3bo9SG9SHeiNGrZEAWr4u7rmhvj7Mn2fU3lFHrb2W8roKCh3mhvtaHurP7Z2v/3i8qohNx7fhcDuAM6vdABLM28BJi7XZ42WVVv72STpJsYEkxQYSE+Yjy7sJcQFbs01oNSqGxQcrXYo4RUK5EJ2cr6eeIX178HOWiZvHx13SFslut5sP1++nqsbOY7OT0Wk7V6vHlYZ6l9vVuP3mnFDfEOztZ0bty+rKKXQUUeOoo+4iod7usrMyb52E8jYQ5GtoNpgb9RpqbQ6W/XSIZT8dwttDR2KvAJJiAxkQGyQTQ4U4i9PlIm2vmeQ+PfA06pQuR5wioVyILmBccgQ79pew6+AJhieEXPTx2/aa2bGvmJvH9yYmzKcdKuxY1Cr1qRHvy+ujrA/1VmrsdfzP1peafUyZtZyv8zcyPGwIPTwCr6RccZabxsex6Ot92BxnJhrrtWrunBLP6KQwLNU2sgtKyc6v/5eWUwzU96OfHkXvF+2PQde5XogK0ZpyCsqw1NgZlSgtXx2JhHIhuoCkXoEE+hrYlFl40VBeaqnj4w259In0Y+pI6SW8HPWhvr7//Hyr3WhVWlbnr2d1/nri/HoxPCyFoSGD8NR5KlBx13G6b3zZj3mcbGb1FV8vPaOTwhidFIbb7eZYSfWpgH6S79KPs2H7UbQaNf2i/RpG0aOCvWSzLNGtbM024WnQMiguSOlSxFlUbtm3uJGSkkqlSxDisny16RCrthTwyryrCPJrfgTY5Xbz9893cajQwp/vG05IgATEK5VmSm92tZvbE26mj38s200ZpJnSMdUUo1VpSOrRnxFhKSQFJaBTy7hIe7LanRw4Wk7WqVH04yeqAfDz0pPYK5ABvQNJ7BWIn5eerdmm8wZ/ITozq83JY29uZmRiKPdMTVC6nG5HrVYRFNT8Rk3yF0GILmLswHBWbSlg854irh8b2+xjNu44Rs7hMu5OjZdA3kouttrNlF7XMjlmAkerjrPdlMF2cwaZJVl4aj1ICRnE8LAU4vx6yUhtOzDoNAzoHcSA3vWjg2WVVrLyT5KdX8qeQyfZmm0CINDHQHm1DZerfsxKll0UXUnGgRKsdiejk2Rt8o5GRsrPISPlojN77fMMzKW1/G3e6CYrTxw/Uc1f/rOdxJgAfjt7kIRAhThdTvaXHSTNlE5mSRY2l50gYwDDw1IYETqEUK+LzwkQrc/ldnPYVEl2fikrt+TjcDb90xjka+DVB8coUJ0QrWfB4kyOlVTxyryrZIUiBchIuRDdxNXJEfxzRTY5BWUkxZ6ZXOhwunh/1V4MOg33TE2QQK4gjVpDYlA8iUHx1DmsZJZksd2cwfqC71hXsJEYn2hGhKUwNDQZH33zv7hF61OrVMSG+xIb7suynw41+5jzLccoRGdhqbaRdaiUKSOjJZB3QBLKhehChvQNxsuo5afMwkahfOWWAg6bK3noxgH4ecvScB2FUWtgZPhQRoYPpcJqYYd5F2mmdBYfWMHSg6voH9iPEWEpDOqRiF6jV7rcbuN8yy7Ksoqis9u+rxiX2y1tWB2UhHIhuhCdVs3opDB+2HWcqlo73h468o5XsGZrAWMGhDE0XlojOio/gy/X9bya63peTWGVie3mDLabMvi/7E8xagwMDh7I8LAh9AuIa9Fuo6Llmlt2sZ6bimobfl7yAkl0TtuyTUQFexMVLO/CdUSa55577jmli+hIampsSpcgxBXx9zHwXfpx/L0NRAV78/cvd2HUaXjk5uRL2lhIKM9H701CYF+uiR5Dv4A4XG4Xu0qy+Lkoja1F27HYKvHV++Cr735rzLeH6BBvgvyMHDZZqLU6CfI1MGFIJPuPlrNzXzEp/YLxMMiYluhczGU1LP4hjykjo+kb1fJN10TrUKlUeHo2/8JeJnqeQyZ6iq7gqYVbKKu04Ty1esSM0THcND5O4arElbA57ew5sZft5nSyT+7H5XYR6R3OiLAUhoUOxt/gp3SJXd6BY+UsWJyJp0HHk7cNJlRWMBKdyMrN+azYnM+rD15FoO/lbZwmrtyFJnpKKD+HhHLR2W3NNvHBmpyGQA71Ox7ePTVB+gi7iEpbFenFu0kzpVNgOYIKFfEBfRgeNoTBwQMwXuZOpeLiDpsq+fsXu9CoVTx562AipQ1AdAJut5s/vLuNAB8DT92eonQ53ZqE8haQUC46u9+/s6XZSWqynFvXVFxTQpopg+2mdE7UlaJT60gOTmJEWAoJAX3RqGU7+dZ2/EQ1r32egdPp5om5yfQK81W6JCEuKL/Iwl8X7eCeqQlcnRyhdDndmoTyFpBQLjq7+17+7rz3ffDMte1YiWhPbrebfMth0kwZpJszqXbU4KPzZljoYIaHDaGnT5QshdmKistqePWzXdRY7Tw6O5l+0dKjKzquT7/J5Yddx1nwyFg8jTqly+nWLhTKZaLnOWSip+jsNu8upNbqbHI8yNfA5OE9FahItAeVSkWA0Z8BPfozIXosPX2iqHXUscOcwabCbews3k2to5YAgz+eOg+ly+30vDx0DIsPZmfuCb7POEZsuC8hAfJ1FR2P0+Xi/9bmkBgbyJiB4UqX0+1daKKnhPJzSCgXnZ2Pp56sQyeb9JTfNrEf0SHS/9odqFVqwrxCGBqazPioMQR7BnGi9iTbinbw/bHN7C89iMvtoodHIDqNjJpdLg+DluEJIew+eJKNO48SFexNeJCX0mUJ0Uh2fik/7irkpqvjiOgh359Kk9VXWkDaV0RXsDXbxLIf8zhpsRLka+Cm8XEyyVNwsraM7eYM0kzpmGuK0ao0DOjRnxFhKSQFJaBVyzJ/l6Oq1s78LzM5bKrkVzP7MypRftZEx/HeqmwyD55k/iNjZVncDkB6yltAQrkQoqtzu90crTxOmjmdHaZdVNqr8NJ6MiR0ECPDUoj1jZH+8xaqtTp4c+lu9h8p567UeMYPjlS6JCGw2pw89uZmRiaGcs/UBKXLEUgobxEJ5UKI7sTpcrKv7CBppp1klmRjd9npYQxkeNgQhoelEOoZrHSJnYbN7uTt5VnsOXSSW6/tw+QRModDKGtbtol3V+3l6duHEN8zQOlyBBLKW0RCuRCiu6pz1JFZkk2aKZ39ZQdx4ybGN5oRYSkMDUnGRy9zEi7G4XTxr5XZ7Nxfwg3jYpl5VS9510EoZsHiTI6VVPHKvKtQy/dhhyChvAUklAshBJRbK9hh3kWaKZ3jVUWoVWoSA+MZETaEgT2S0MsE0fNyulz8Z+0+tmSZSB3ZkznXxEkwF+3OUm3jibe2MGVkNHOu6aN0OeKUC4VyRWf12Gw23njjDVasWIHFYiEhIYHHH3+c0aNHX/Tcn3/+mYULF5Kbm4vL5aJ3797cfffdTJs2rR0qF0KIrs3f4MfEnuOZ2HM8x6uK2G7KYLs5g6zsHIwaA4NDBjIyLIU+/r1Rq2Ty2Nk0ajX3Tu+PXq9h3X+PUGdz8ovJ/WSkUrSr7fuKcbndMsm/E1F0pPyJJ55gw4YN3HXXXcTExLB8+XKysrL46KOPGDJkyHnP+/7775k3bx5Dhgxh+vTpAKxZs4b09HSef/555syZc9k1nW+k3G63UVlZjsNhw+Vquga0EAAajRZvb388PGTZKdH1uNwuDpQdIs2czq7iPdQ5rfgb/BgeOoQRYSlEeMsf/7O53W6W/JjH19uOMDopjPumJ6BRywsY0T6e/3AHNruLv/xyhNKliLN0yPaV3bt3M2fOHJ599lnuueceAKxWKzNmzCAkJIRPPvnkvOf+6le/Yv/+/WzcuBG9vn6tR5vNxnVYBnLsAAAgAElEQVTXXUdMTAwff/zxZdfVXCivra2msrIMb28/DAYP1GqNvBUpmnC73djtNsrLS/DxCZBgLro0m9PGnhN7STOls7c0F5fbRZR3RP0E0dAh+Blk63mo/72weuthlv90iKH9gvn1rCRZlk60OXNZDc/+axtzJsQxdWSM0uWIs3TI9pV169ah0+kajWobDAZmz57N/PnzKS4uJiQkpNlzq6qq8PPzawjkAHq9Hj8/PwwGQ6vXWlVVgb9/D/R6Y6tfW3QdKpUKvd6Av38wFRUnJJSLLk2v0TM0dDBDQwdTaatipzmTNHM6yw+u4auDa4kP6MOIsBSSgwdg1Lb+7+XOQqVSMfOqXhh1Gj7beIA3l+3moRsHYtBplC5NdGH/zTajAkb2D1W6FNECioXynJwcYmNj8fJqHFwGDRqE2+0mJyfnvKF8xIgR/Otf/2LBggXcdNNNACxbtoyCggKeffbZVq/V6bSj03XfPyqiZXQ6PU6nQ+kyhGg3PnpvrokewzXRYzBXF5/aoCiDD3O+QL9/GcnBAxgelkJCQB806u4ZRicNj8ag17Do633M/zKTR2cPwsMgmzWJ1ud2u9mabSK+pz+BvjKY2Jko9huhpKSE0NCmr+CCg+vXxC0uLj7vuQ888ABHjhzhn//8JwsXLgTA09OTd955hzFjxrRJvdKuIi6VfK+I7izUK4QZvacwPXYyhyoOk2baSXrxbrabM/DRezMsdDAjQlOI9onsdj8rVydHYNBpeH/1Xl77PIPHbxmMt4esYiNaV4GpEnNZLVNHSdtKZ6NYKK+rq0Ona/rL6HT7idVqPe+5er2eXr16kZqayqRJk3A6nXz55Zc89thj/Oc//2HQoEGXXVdwsE+TY8XFarTSAyhaQK1WN/u9JER3EhIykFF9B2J32skoyuanw/9l0/FtfH90M5G+YYyLGcG4mBEEewUpXWq7mTHeh+AgL17+cAevf5nJX34zmgAfGc0UrWf5lgJ0WjVTxvSWF32djGKh3Gg0Yrfbmxw/HcYv1Bv+17/+lT179rBkyRLUp2ayT506lRkzZvDiiy/y+eefX3ZdzU30dLlcOByuy75md/bww78G4K233m3Xc5XmcrlkzXshzhJriCO2XxxzYmtIL95NmimDz/es5PM9K+nr35vhYUMYEjwIT52H0qW2ud6h3jw6ZxBvLt3N79/4iSdvHUKQnwRzAWmmdFbmraPMWk6AwZ9ZcamMCEu55POdLhc/7jzKoLggaqvqqK2qa8NqxeXokBM9g4ODm21RKSkpAThvP7nNZmPJkiX85je/aQjkADqdjnHjxvHZZ5/hcDjQaqVX70LGjh12SY9bvHgl4eERbVyNEKK78NR5MjZyFGMjR3GitpQd5gzSTOl8um8pX+auYGBQf4aHpZAUFI9W3XV/jyf1CuR3cwezYHEmL3+ykydvG0JogKfSZQkFnf45sLvqByzLrOV8um8pwCUH870FZVhq7IxKlOVJOyPFfuMlJCTw0UcfUV1d3WiyZ2ZmZsP9zSkvL8fhcOB0Nl0r3OFw4HA4kE1KL+7//b+/NPr8yy8/w2wu4pFHnmh03N8/4IqeZ/78txU5VwjR8fXwCCS113VMibmWI5XHSDOls8O8i4ySPXjpPBkakszwsBRifXt2yf7zvlH+PHVbCn//Yhcvf5zO724dTFRw8yNoovNwuV04XA5sLjt2p72ZjzbsLseZ26fu++bwDw2B/DS7y86S3JX46L3x1nk1/NOdZ0fdbdkmPA1aBsV1n5awrkSxdcozMzO55ZZbGq1TbrPZmDFjBkFBQXz22WcAFBYWUltbS1xcHABOp5NRo0YRHBzMihUrGvrSq6urmTZtGr6+vqxateqy62qu5cBkOkxYWNeeMPHss7/jwIFcliy58Neurq4Oo1HeZr2Y7vA9I0RbcLqc5JTmst2cQWZJNnaXnR4eQYwIHcLwsCGEeAZf8Vv8Hc3xE9W89nkGDoeL3906mF5hssZ7a3O5Xdicduwu+5mPLht2p6MhGDe+r2mgPn1f/eNOBeuzbtd/tGN3tf3qWwaN/lRA98ZL74mPzhuj2oPvd5TQO7gHk1P61N+vrw/xHlqj7LzbQXTIzYMAHn30UTZu3Mjdd99Nz549G3b0XLRoEUOHDgXgzjvvJC0tjf379zect3DhQhYsWEBSUhKzZs3C5XKxZMkS8vLymD9/PtOmTbvsmiSUnwnlDz/8a6qqqnjqqT/w5pvz2b9/H3fccRe//OVv2LTpB1auXE5u7n4slgqCg0OYNm0md955LxqNptE14ExfeHr6Dn772wd44YVXyM8/xFdfLcViqWDgwGR+//s/EBUV3SrnAixd+iWff/4JJ0+eIC4ujocffpz33lvY6JptpTt8zwjR1moddWSWZJFmSie3LA83bnoYAymzVuB0n3m3VKfWcXvCzZ06mBeX1fDqZ7uosdp5dHYy/aL9lS4JuPIe5wtxupyNwu7Zofb07QuF5LNHnRvOPStk204FaLvTjsN9eTtxq1Vq9Go9Oo321EcderUWnVqPXqNDr9ah0+jQqXXoT39s7phGf+o+7Znbp+47ffu5ra9QZi1vUoOf3pf7BtxBlb2aKlsVVfYaquxVVNnqP1bbq6m0VWOxVuGk+RcEapUaL60nXnovfHReeJ0V2Bv+nfP5+UbjxZXpkD3lAK+88goLFixgxYoVVFRUEB8fz7vvvtsQyM9n3rx5REVF8eGHH/L2229js9mIj4/nrbfeYtKkSe1U/ZXZmm1i2Y95nLRYCfI1cNP4OEYndbwesPLyMp566nEmT04lNXU6oaH1Na5duxoPD0/mzr0DT08Pdu7cwfvv/5Pq6moeeujRi1530aJ/o1ZruP32u6istPDZZx/x5z//iffeW9Qq5y5fvoT5819h8OAU5s69jaKiIp599kl8fHwIDm5+voIQomPx0BoZFT6MUeHDKLdWsN2UwapD63C6G0+8t7vsrMxb16lDeUiAJ8/+IoVXP9/F61/s4pGbB5EUG6hoTf8t2sln+5c16nH+OGcxh8oLiPaJbBSMbc0F4vME6tPHXO7LW0BBq9Kg0+jrw7FGXx+CTwVbL50nARq/RmG3UQDW6NCrzzn3rHDd6KNa167r6s+KS23UUw71Lzhv6DONPv6xFz1/weJMjp4o59l7BlJjr6kP8af/2RrfNlWbqSqvptpeg5vmx2abG4330p36qPfEW+cto/GtTNFQbjAYePrpp3n66afP+5iPPvqo2eMzZ85k5syZbVVam9qabWLR1/uwnVrR5aTFyqKv9wF0uGB+4kQJzzzz/5gx4/pGx5977nkMhjNtLDfcMJtXX32R5csXc//98xrtttoch8PBBx8sapiQ6+vrxxtvvMahQwfp3bvPFZ1rt9t5//2FJCUNZMGCdxoe16dPX1544TkJ5UJ0Qv4GPybFXMNXeWubvb/MWs77WR/TP7Av/QP7EWi8svkwSgj0NfLMHSn8/fNdvLEkk3nXD2BIv+BWfQ63202dsw6LtRKLrQqLrRKLrZLKRrfr7yu3VjQ53+l2sqlwW5PjZ48QnxtsffXGJvfVP77pCPT5RpHPjDZru2zwO/2i8nLembBU28g6VErqyJ708AgEj0t7Qedyu6ix154T4JuOxlfZqiiqMlNtr8bmarpyHpwZjT8d0mU0vuW67tT2drBlTxGbdxe1+Ly8wgoczsavTG0OF/+3NoefdhW2+HpjB4UzZmB4i8+7FEajkdTU6U2Onx3Ia2qqsdnsJCcPYcWKZRw+XEDfvv0ueN3p02c1WiEnOXkwAIWFxy8ayi927r59e6moqODBB29s9LhJk1L5xz9ev+C1hRAdW4DBv9m3+PVqPfkVh8ko3g1AqGcwCYH96B/Yl77+cRi1nWNXZj8vPU/dPoT5X2by9vIsfjWjP6MuYbDG6rRhsVZSaa88FbjPH7odzfQ8q1VqfHTe+Oq98TH4EOEVzjbTjvM+3/NX/aFh1Fmr1nTZoNzeRoSlXNY7Ptv3FeNyuxmV1HRTxgtRq9T1QVnvdfEHn2Jz2pqOvjczGl9Ubaa6g4/Gd7T5KRLKFXBuIL/YcSUFB4c0u7zkoUN5vPfeQtLTt1NdXd3ovurqqote93QbzGk+PvUTmyorL76298XONZnqXyid22Ou1WoJD2+bFy9CiPZxvrf4b0u4ieGhQzDXFLO3NJec0lx+Lkzjx2Nb0Kg09PaLISGwH4mB/YjyiejQIdLbQ8eTtw7mH0szeG/9Tgprw+gTY2wUrC3njGxbnbYm11GhwlvnhY/eG1+9DyH+PRpun/3PR18fhM79muwvO9jsC6AAgz8Bxo7R8y7qbc02ER3i3S6r9+g1egI1+kt+N6otR+O9dV6N+uR9Tn1+KaPxrbEEZWuTUH4Fxgy8vBHq37+zhZOWpjuWBvkaePqOjtUTefaI+GmVlZU88siv8fT05pe/fIDIyCj0ej25uftYuPBNXK6L9wmqz9Ondynzjq/kXCFE53axt/jDvEIJ8wrl2uhx2F0ODpUXkFOay77SXFYdWseqQ+vw0nmSEFDf5pIQ2LddA6bT5aTSfipQW89tG2kcumvDajGGwUYLbNxz5hpeWs+GcN3LN7rhtk+jsF0/wnglPdHnewE0Ky71Sr4EopWZy2o4VGhhzoQ4pUtpVluOxhdewWj81qLtzS5BqeT8FAnlCrhpfFyjnnIAvVbNTeM75g/UuTIydlJRUcELL7zK4MFnvnGLilreetMWwsLqXygdO3aU5OQhDccdDgdFRUXExV24PUYI0bFd6lv8OrWW+MA+xAf2AaZRaatiX+mBhpC+s7h+X4wwr9CGXvQ+/r0xaPQtelvb5XZRba85E6itzfVp19+uslc3ew2jxljfOqL3IcI7nIRTQdtL68WWjDIOFNQxeXAfbrwqAX079eFeSY+zaD/bss2ogJH9W9a60pEpORrf3LtD7UVCuQJOT+bsDKuvNOf0Tqpnj0zb7XaWL1+sVEmNJCQk4ufnx8qVy5kyZVpD+80336yjstKicHVCCKX46L0ZHla/3rnb7aao2sze0v3sKz3A5uPb+P7oZrQqDT08giipPdGwysvpVUf2ntiHv9G/SeCuslc3u5JI/STH+pHrYM8e9Pbv1TCKfaZ1pP5zveb8k+OvinDxn7X7WL/FhMruwZwJce22mdLl9jiL9uF2u9mWbSK+pz+Bvt13D5HLGY3/05YXKGtmMnOAQbnWLAnlChmdFNZpQvi5Bg4chI+PLy+88ByzZ89FpVKxfv1aOkr3iE6n4777fs38+a/y2GMPMmHCdRQVFfH116uIjIzqkjsDCiFaRqVSEeEdRoR3GBN7jsfmtJNXkU9OaS4/HN3cZNlFp9vJ9uJdaFWahjaRAKMfPX2i8DX4nNOrXX/boDG0yu8bjVrNvdP7o9drWJd2hDq7k19M7odafpd1e/lFlZjLapk6SvbFaKlZcVM7XHuWhHLRYn5+/rzyynzeemsB7723EB8fXyZPnsqwYSN44omHlS4PgJtvnovb7ebzzz/h7bffIC6uLy+//DoLFryGXt85VmEQQrQfvUZH/8B+9A/sx8YjP533cQuueVGRF/ZqlYpfTOqHUa/h621HsNqc3Dc9AY26405YFW1vW7YJrUbNsPjWXTqzO+iI7VmK7ujZEXXXHT27A5fLxYwZkxg/fgJPP/2nNn0u+Z4RovP605YXz7vqyPNj/qBARWe43W5Wbz3M8p8OMbRfML+elYROK8G8O3K6XPzurS30jfbnoRsHKl2OuEQX2tFTfpJFl2S1Nl3dZt26NVgsFQwZcuEdY4UQ3dusuFR06saTKZV+W/s0lUrFzKt6cdt1fdmZW8KbS3djtV/eFvKic9tbUIalxt5pW2FFU9K+Irqk3bt3sXDhm1xzzbX4+vqRm7uPNWtW0rt3HBMmTFS6PCFEB9YR39Y+16Th0Rj0GhZ9vY/5X2by6OxBeBjkT3p3si3bhKdBy8DeQUqXIlqJ/ASLLikiIpIePYJZsuQLLJYKfH39SE2dzgMPPIxOJ9v6CiEurDOsOnJ1cgQGnYb3V+/ltc8zePyWwXh7yO+37sBqc5Kee4JRSaHSvtSFSCgXXVJkZBSvvDJf6TKEEKJNjUwMxaDT8M5XWfzt03SenDsYP2+ZzN7VZRwowWp3Miqx66xNLqSnXAghhOjUBvftwWNzBlFSXsvLn6RzsqJO6ZJEG9uabSbQ10DfaOXW1BatT0K5EEII0ckl9grkyblDsNTYefmTnZjLapQuSbQRS7WN7PxSRiWGyVr1XYyEciGEEKIL6BPlx1O3DcFqd/Hyx+kcK6lSuiTRBtJyzLjcbkYlSetKVyOhXAghhOgiYsJ8ePqOFFDB3z5JJ7/IonRJopVt22smOsSbqODm17oWnZeEciGEEKILiezhxbN3pGDUa3n1swxyjzbdCEl0TuayGg4VWmSUvIuSUC6EEEJ0MSEBnjz7ixT8vQ28/sUusvNLlS5JtIJt2WZUwMj+Esq7IgnlQgghRBcU6GvkmTtSCAnw5I0lmaTnlihdkrgCbrebbdkm4nv6E+hrVLoc0QYklAshhBBdlK+XnqduH0LPUB/eWZ7FtmyT0iWJy5RfVIm5rJbRSWFKlyLaiIRy0SrWrl3F2LHDKCoqbDg2e/ZMXnjhucs690qlp+9g7NhhpKfvaLVrCiFEZ+TtoeN3cwfTL9qP91bt5Yddx5UuSVyGbdkmtBo1Q+NDlC5FtBEJ5d3UU089zsSJY6mtrT3vY5544mGmTBmP1Wptx8pa5ttv1/Pll58qXYYQQnRoHgYtj81JZkDvID5ct5/1aUeULkm0gNPlIi3HTHKfIDyNshl7VyWhvJuaNGkKdXV1bN78Y7P3l5WVsnPndq6+egIGw+Vt2fzpp0t5+uk/XUmZF7Vx4wa+/PKzJscHD05h48YtDB6c0qbPL4QQnYVep+GRmwcyND6YL747yMrN+bjdbqXLEpdgb0EZlhq7tK50cRLKu6lx467Bw8OTb79d3+z93333LU6nk8mTUy/7OfR6PVqtMq/o1Wo1BoMBtVq+xYUQ4jStRs0D1ycxZkAYX23OZ/H3eRLMO4Ft2SY8DVoG9g5SuhTRhuQ9kG7KaDQybtx4vv/+WywWC76+vo3u//bb9QQFBREdHcNrr73Mzp1pmM1mjEYjKSnDeOihRwkPj7jgc8yePZMhQ4byxz8+13Ds0KE8Fix4laysPfj5+XH99TfRo0dwk3M3bfqBlSuXk5u7H4ulguDgEKZNm8mdd96LRqMB4OGHf82uXekAjB07DICwsHCWLFlFevoOfvvbB/jHP/5JSsqwhutu3LiBjz/+D4cPF+Dp6cWYMeOYN++3+Pv7Nzzm4Yd/TVVVFf/zP3/h9ddfIScnGx8fX+bMuZU77ri7ZV9oIYToYDRqNfdO749er2Fd2hHq7E5+MbmfbNneQVltTtJzTzAqKRSdVgaaujIJ5QpJM6WzMm8dZdZyAgz+zIpLZURY+7ZaTJqUyoYNX/PDDxuZNevGhuMmUxFZWbuZPftWcnKyycrazcSJUwgODqGoqJCvvlrKI4/8ho8/XozReOnLMp08eYLf/vYBXC4Xv/jF3RiNHqxcubzZ9pi1a1fj4eHJ3Ll34Onpwc6dO3j//X9SXV3NQw89CsDdd99HbW0tZnMRjzzyBAAeHp7nff61a1fx4ot/JilpIPPm/ZbiYjNLl35BTk427733YaM6LJYKfve73zJhwnVcd91kvv/+WxYufJPevfswevSYS/4/CyFER6RWqfjFpH4Y9Rq+3nYEq83BfdP7o5F3FzucjAMlWO1ORiXK2uRdnYRyBaSZ0vl031LsLjsAZdZyPt23FKBdg/nw4SPx9w/g22/XNwrl3367HrfbzaRJU4iL68OECRMbnTdmzNU88MC9/PDDRlJTp1/y833yySIqKsp5//2PiI9PAGDq1BncdtuNTR773HPPYzCcCfw33DCbV199keXLF3P//fPQ6/UMHz6KZcsWU1FRzpQp0y743A6Hg4UL36RPn368+ea/0Ov1AMTHJ/Dcc39k1arlzJ59a8Pji4vN/O//Ps+kSfXtOzNmXM/s2TNYs2aFhHIhRJegUqmYPT4Oo17L8p8OYbW7+M2sJBmN7WC2ZpsJ8jXQN9r/4g8WnZqE8ivw36KdbC3a3uLz8iuO4HA7Gh2zu+x8krOEnwvTWny90eHDGRk+tMXnabVarr12Il99tZQTJ07Qo0cPAL79dgNRUdEkJg5o9HiHw0F1dRVRUdF4e/uQm7uvRaF869YtDByY3BDIAQICApg0aSrLly9u9NizA3lNTTU2m53k5CGsWLGMw4cL6Nu3X4v+r/v27aWsrLQh0J927bWTePvtN/j55y2NQrm3tzcTJ05p+Fyn09G/fxKFhbKUmBCi61CpVMy8qhdGnYbPNh7gzaW7eeimgRh0GqVLE4Cl2kZ2fimpI3tKe1E3IKFcAecG8osdb0uTJqWybNlivvtuA7fccjsFBfkcPJjLvffeD4DVWsdHH/2HtWtXUVJS3GhCUFVVVYuey2w2MXBgcpPjPXvGNDl26FAe7723kPT07VRXVze6r7q6Zc8L9S05zT2XWq0mKioas7mo0fGQkFBU5/wC9PHxJS/vYIufWwghOrpJw6Mx6DUs+nof87/YxaNzkvEwSERQWlqOGZfbzegkaV3pDuQn7gqMDB96WSPUf9ryImXW8ibHAwz+PJbyQGuUdskGDkwmPDySb75Zxy233M4336wDaGjbmD//VdauXcWcObcxYMBAvL29ARXPPfeHNpuxX1lZySOP/BpPT29++csHiIyMQq/Xk5u7j4UL38TlcrXJ855NrW5+lEhWKRBCdFVXJ0dg0Gl4f/VeXvs8g8dvGYy3h07psrq1bXvNRId4ExnsrXQpoh1IKFfArLjURj3lADq1jllxl7/84JWYOHEyH330fxw7dpSNGzcQH9+/YUT5dN/4I4883vB4q9Xa4lFygNDQMI4dO9rk+JEjhxt9npGxk4qKCl544dVG64w3v+Pnpb2dFxYW3vBcZ1/T7XZz7NhRYmPjLuk6QgjRlY1MDMWg0/DOV1n87dN0npw7GD/vy9urQlwZc2kNhwotzJkgf5+6C5nNoYARYSncnnAzAYb6SRsBBn9uT7i53VdfOW3y5KkAvPXWfI4dO9pobfLmRoyXLv0Cp9PZ4ucZPXoMe/Zksn//voZjZWVlfPPN140ed3pt8bNHpe12e5O+cwAPD49LeoGQkJBIQEAgX321BLv9zIuh77/fSElJMVddJZM3hRACYHDfHjw2ZxAl5bX87wdpPPHWFu57+Tt+/84WtmablC6v29i214wKGNlfWle6CxkpV8iIsBTFQvi5YmN706dPPzZv/gm1Ws11152Z4HjVVWNZv34tXl7e9OoVS3b2HnbsSMPPz6/Fz3P77Xezfv1annjiIWbPvhWDwcjKlcsJDQ2nqupAw+MGDhyEj48vL7zwHLNnz0WlUrF+/Vqa6xyJj09gw4avefPN10lISMTDw5OxY69u8jitVsu8eY/w4ot/5pFHfsPEiZMpLjazZMkX9O4dx8yZTVeAEUKI7iqxVyCpI3qycktBw7GTFiuLvq4fVJGdJduW2+1mW7aJ+J7+BPpe+tLDonOTUC4AmDw5lYMHcxkyZGjDKiwAjz76JGq1mm+++Rqr1cbAgcksWPA2TzzxSIufo0ePHvzjH/9i/vxX+Oij/zTaPOjll//a8Dg/P39eeWU+b721gPfeW4iPjy+TJ09l2LARPPHEw42uef31N5Obu4+1a1fzxRefEhYW3mwoB5g2bSZ6vZ5PPlnE22+/gZeXF5MmpfLAA480u1a6EEJ0Z1v2FDU5ZnO4+GBNDj/tKsSg16DXqtHrNPX/Tt026NTotZpz7j917NRtw1nn6LTqJhPru7v8okrMZbVMG9V0IQTRdancMnOtkZKSyibHTKbDhIXJD4a4dPI9I4To7O57+bvz3tcv2h+b3YnN4cJmd2K1O7HZ629fTqg4E9rPhHzD2YH/1P1nB3rDOYG/4QXBee5Xqzt+8N+abWLZj3mctFgBuDs1nvGDIxWuSrQmtVpFUFDzE3dlpFwIIYQQTQT5GhrC4bnHn7mj+fZLt9uNw+nCeiqgNxfarY4zt5u//9R9dic1VgflVdYz9zucWG0uXJcxnqjVqC8Y2huN4OvUGLTnf0Fw5v5T19BqMOjVV7Qj6tZsE4u+3ofNcWaFsc++PYBep5F2oW5CQrkQQgghmrhpfFyTkKjXqrlp/PlXA1GpVOi0GnRaDbThcooOp6shpNeH+jNB/kyoPxXizwr8Nrvr1IuCxvdX1tjPeZHgwuFs+fK7GrXqPKG9caCvf1Fw1m2dhq82HWr0tYb6dqFlP+ZJKO8mJJQLIYQQoonTQfB0O0WQr4Gbxsd1iICo1ajRatR4tmGMcbncp0L7mcB/dmiv/7zp/WeP6tvspz93Ul5la/wi4tT9F9PcuxWia5JQLoQQQohmjU4K6xAhXAlqtQqjXotR33bP4Xa7G4L+/36wnfKq5tuFRPcg65QLIYQQQihApVJh0Gnw8dQzZ0Icem3jWHaxdiHRtchIuRBCCCGEwjpyu5BoHxLKL5Hb7ZZ1VMUlkVVGhRBCXI7u3C4kpH3lkmg0Oux2mWghLo3dbkOjkde7QgghhLh0Esovgbe3H+XlJ6iursTpdMhIqGiW2+3GZrNSXl6Ct7e/0uUIIYQQohNRdDjPZrPxxhtvsGLFCiwWCwkJCTz++OOMHj36gudde+21HD9+vNn7YmJi2LBhQ6vW6eHhhVaro6qqnOrqClwuZ6teX3QdGo0WH58APDy8lC5FCCGEEJ2IoqH8mWeeYcOGDdx1113ExMSwfPly7r//fj766COGDBly3vP+8Ic/UF1d3ehYYWEhCxYsYMyYMW1Sq06nJyAgpE2uLYQQQgghujeVW3n3wyIAAAlTSURBVKFejN27dzNnzhyeffZZ7rnnHgCsViszZswgJCSETz75pEXXe+edd3jjjTf47LPPSElpfvvfS1FSUnnZ5wohhBBCCHE+arWKoCDv5u9r51oarFu3Dp1Ox5w5cxqOGQwGZs+ezc6dOykuLm7R9VavXk1UVNQVBXIhhBBCCCGUoFgoz8nJITY2Fi+vxr23gwYNwu12k5OTc8nX2rt3L3l5ecyYMaO1yxRCCCGEEKLNKRbKS0pKCAlp2qMdHBwM0KKR8lWrVgEwa9as1ilOCCGEEEKIdqTYRM+6ujp0Ol2T4waDAajvL78ULpeLNWvWkJiYSFzclW9FGxzsc8XXEEIIIYQQoiUUC+VGoxG73d7k+OkwfjqcX0xaWhpms7lhsuiVOnmyqlWuI4QQQgghxNnUahUBAc0vm6xYKA8ODm62RaWkpASg2daW5qxatQq1Ws306dNbpa7zzYgVQgghhBCirSjWU56QkEB+fn6T9cYzMzMb7r8Ym83Ghg0bGDFiBKGhoW1SpxBCCCGEEG1NsVCempqK3W5n8eLFDcdsNhvLli0jJSWlIWQXFhaSl5fX7DV+/PFHLBYLM2fObJeahRBCCCGEaAuKta8kJyeTmprKa6+9RklJCT179mT58uUUFhby0ksvNTzu6aefJi0tjf379ze5xqpVq9Dr9UyZMqU9SxdCCCGEEKJVKRbKAV555RUWLFjAihUrqKioID4+nnfffZehQ4de9Nyqqip++OEHrrnmGnx8ZMUUIYQQQgjReancbrdb6SKEEEIIIYTozhTrKRdCCCGEEELUk1AuhBBCCCGEwiSUCyGEEEIIoTAJ5UIIIYQQQihMQrkQQgghhBAKU3RJRCUtWrSIN998k8rKyoZjfn5+jBs3jtWrV5OQkMCKFSsUrFAIIYQQQnQX3Xak/PPPP6eyshK9Xk9AQAAAFouF1atXo9frFa5OCCGEEEJ0J902lP/pT39i06ZN7Nmzh+effx6AwYMHAyBLtwshhBBCiPbUbdtXxowZ0+RYZmYmOp0Op9OpQEVCCCGEEKK76rYj5adVVVU19JVHRkZit9vR6XQKVyWEEEIIIbqTbjtSftof/vAH1q9fD8CxY8cACA8PV7IkIYQQQgjRzXT7kfJ7770XT09PAFQqFUFBQQQGBipclRBCCCGE6E66fSj/9ttvG1ZbCQgIwN/fn/z8fIWrEkIIIYQQ3Um3DuXFxcUsWrSoYbWVX/3qV4wePZqysjJsNhvHjh2joqJC4SqFEEIIIURXp3J34/X/MjMzueWWWy74mPvvv58nn3yynSoSQgghhBDdUbcdKS8pKeGtt95CrVaTmpoKwLx58wgMDESr1RIZGcnbb7/NDTfcoHClQgghhBCiq+u2I+Xjx4/HZDLh7++Pj48PR48eRa/XY7PZ8PHxITIykhUrVihdphBCCCGE6Aa67ZKIHh4eAJSXl1NeXg6AzWYDaFi3XAghhBBCiPbQbUfKhRBCCCGE6Ci6bU+5EP+/vfsJiaKP4zj+2VXsEhH+6VKb9Ada1EU9RK6hiCZIGNtBWGzdosxDq4FKnaJDUHSoLq0eNE9d9JCCMIfIckFtoFNItC1RSboEFqYnpSj3OURD+0zPUxcb232/br/vfGfnN3v68NvfzAIAAGwWhHIAAADAYYRyAAAAwGGEcgAAAMBhhHIAAADAYYRyAAAAwGGEcgAAAMBhhHIAgGPC4bDq6+udngYAOC5r/9ETADLVkydPdPLkyf88npOTo3g8/gdnBAD4FUI5AGSo5uZm1dbW2upuNz+SAsBmQygHgAxVUlKiQCDg9DQAAL+B5RIAyFLJZFIHDhxQNBqVYRg6duyYfD6f6urqFI1G9eXLF9s5iURCnZ2dOnTokHw+n44ePao7d+7o69evtt4PHz7o6tWramhoUFlZmfx+v06fPq3Hjx/behcXF9Xb26uDBw+qvLxc7e3tmpub25D7BoDNiJVyAMhQa2tr+vjxo62el5enrVu3WuPJyUktLCwoFAqpsLBQk5OT6uvr07t373T9+nWr79mzZwqHw8rNzbV6Y7GYbt68qUQioVu3blm9yWRSra2tWlpaUiAQUFlZmdbW1jQ7OyvTNHX48GGrd3V1VW1tbSovL1dPT4+SyaTu3r2rSCQiwzCUk5OzQd8QAGwehHIAyFDRaFTRaNRWr6ur08DAgDVOJBK6d++eSktLJUltbW3q6urS2NiYgsGgKioqJEnXrl3T58+fNTIyIq/Xa/V2d3fLMAy1tLTI7/dLkq5cuaL3799raGhINTU1addfX19PGy8vL6u9vV0dHR1WLT8/Xzdu3JBpmrbzASATEcoBIEMFg0E1NTXZ6vn5+Wnj6upqK5BLksvl0tmzZ/Xw4UNNTEyooqJCS0tLevr0qRobG61A/r333Llzun//viYmJuT3+7WysqLp6WnV1NT8NFD/+0FTt9tte1tMVVWVJOnt27eEcgBZgVAOABmquLhY1dXVv+zbt2+frbZ//35J0sLCgqRv21F+rP9o7969crvdVu/8/LxSqZRKSkp+a547duzQli1b0mrbt2+XJK2srPzWZwDA344HPQEAjvq/PeOpVOoPzgQAnEMoB4As9/r1a1vt1atXkiSPxyNJ2rVrV1r9R2/evNH6+rrVu3v3brlcLr148WKjpgwAGYdQDgBZzjRNPX/+3BqnUikNDQ1Jko4cOSJJKigoUGVlpWKxmF6+fJnWOzg4KElqbGyU9G3rSW1traampmSapu16rH4DgB17ygEgQ8XjcY2Pj//02PewLUler1enTp1SKBRSUVGRHj16JNM0FQgEVFlZafVdunRJ4XBYoVBIJ06cUFFRkWKxmGZmZtTc3Gy9eUWSLl++rHg8ro6ODh0/flylpaX69OmTZmdntXPnTl28eHHjbhwA/kKEcgDIUIZhyDCMnx578OCBtZe7vr5ee/bs0cDAgObm5lRQUKBIJKJIJJJ2js/n08jIiG7fvq3h4WGtrq7K4/HowoULOnPmTFqvx+PR6Oio+vv7NTU1pfHxcW3btk1er1fBYHBjbhgA/mKuFL8jAkBWSiaTamhoUFdXl86fP+/0dAAgq7GnHAAAAHAYoRwAAABwGKEcAAAAcBh7ygEAAACHsVIOAAAAOIxQDgAAADiMUA4AAAA4jFAOAAAAOIxQDgAAADiMUA4AAAA47B8YnnLCnaj7gQAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# BERT(Handle Class inbalancy)"
      ],
      "metadata": {
        "id": "OD4ChDhHKQoh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class_weights = dict(zip(np.unique(df.label), class_weights))\n",
        "class_weights"
      ],
      "metadata": {
        "id": "Fv4EsRiFfHrf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "69e47f56-c2a0-407d-8c49-0c8c9488393e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{0: 2.8840253342716395,\n",
              " 1: 2.2856664807585054,\n",
              " 2: 1.8764652014652015,\n",
              " 3: 0.678622288458354,\n",
              " 4: 0.45263971725204327}"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from simpletransformers.classification import ClassificationModel\n",
        "\n",
        "args = {\n",
        "    'reprocess_input_data': True, \n",
        "    'overwrite_output_dir': True\n",
        "}\n",
        "\n",
        "model = ClassificationModel('bert', 'bert-base-uncased', num_labels=5, args=args, weight=[2.88, 2.28, 1.87, 0.68, 0.45])"
      ],
      "metadata": {
        "id": "Hp149qWZKV0N",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "109351a5-0fef-4fe0-c08f-fdb94d9a8dcd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight']\n",
            "- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Inspect Training Arguments"
      ],
      "metadata": {
        "id": "vcAguIh0WxLZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"train epochs : \", model.args.num_train_epochs)\n",
        "print(\"max_seq_length : \", model.args.max_seq_length)\n",
        "print(\"train_batch_size : \", model.args.train_batch_size)\n",
        "print(\"learning_rate : \", model.args.learning_rate)\n",
        "print(\"model_name : \", model.args.model_name)"
      ],
      "metadata": {
        "id": "L-ONRr39WvlZ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d2bdc5c6-ab15-4ffd-bdfb-cdf679b78612"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "train epochs :  1\n",
            "max_seq_length :  128\n",
            "train_batch_size :  8\n",
            "learning_rate :  4e-05\n",
            "model_name :  bert-base-uncased\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print('Training on {:,} samples...'.format(len(train_df)))\n",
        "\n",
        "# Train the model, testing against the validation set periodically.\n",
        "out = model.train_model(train_df, eval_df=validation_df)"
      ],
      "metadata": {
        "id": "Ds7nFa6gMCGU",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 220,
          "referenced_widgets": [
            "79ecda6afca14d618e9a8acdf3b9c80a",
            "9efe5be6ab08466e98dba5a6b45cb578",
            "2b81d875112e4ba0adb9f2c056c1a16b",
            "603d2a7871ca4cc0800d6eea8e61ef67",
            "00cda36dcc8940d584174f4a33eba3d9",
            "ab144ea22798488380e136a46fbe41bc",
            "05bb21dbeac34cd8bc83edf309d50607",
            "deb470b150554df7ba59c59578a94868",
            "ed1acd951f7444d596f83be2eab59012",
            "984c1f1e7bf74a089b28796764c92a62",
            "e706b51965574707b4d3ae89df409263",
            "aec1c216bbcc4a2ea06f504cee89fd55",
            "f0c0d2eec6db425ba4abedd14595b8d0",
            "befb6384ed7743058ba4d60ccfce52a5",
            "e9f53bb7775e44188d9351fd8c72a0e5",
            "c47b34f61dfa4a4cabccdcd9fa92bb41",
            "330e8f4644c848499e199f1c3973bb5e",
            "da10279328c54a9a903f68b92bc92ab1",
            "001852548c0240da9b52b83d66269698",
            "41de63a8e78246dd97c4576fec923b00",
            "a4652c622d3b4582b78c82d36a9fd209",
            "fdf933635ec3457287f80871d4d264a2",
            "f1710fec06fb4c08b22c56aa4b6047c2",
            "8a68f39446c84ad491b1d7e4a0ec060d",
            "3ee45aa344ad401ba151f302c34a5607",
            "1057636a268a40d29c03ccb2b0093ad6",
            "a6a20d8caa7342a096456871f792447a",
            "07f8474345724683af79ccbe45e901c1",
            "d7cd8fa1d9354f30bd760f0969eafc57",
            "eb20bea387154403a7119f1ad74f37cc",
            "226096de70d648d09300842178b57e63",
            "464b65be94bb4accaa5a16c0c83d9c8e",
            "96477534e9a44a5890e1537d5e7353f8"
          ]
        },
        "outputId": "c2f30701-5d57-44bc-e2df-a234355855a4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training on 16,392 samples...\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:602: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/16392 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "79ecda6afca14d618e9a8acdf3b9c80a"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:310: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
            "  FutureWarning,\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Epoch:   0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "aec1c216bbcc4a2ea06f504cee89fd55"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Epoch 0 of 1:   0%|          | 0/2049 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f1710fec06fb4c08b22c56aa4b6047c2"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I6ZGs3FpMRuk"
      },
      "source": [
        "## Model Evaluation\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On Validation"
      ],
      "metadata": {
        "id": "RlwmXYREmS5F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import f1_score, accuracy_score\n",
        "def f1_multiclass(labels, preds):\n",
        "    return f1_score(labels, preds, average='micro')\n",
        "\n",
        "def conf_matrix(labels, preds):\n",
        "    return confusion_matrix(labels, preds)\n",
        "    \n",
        "result, model_outputs, wrong_predictions = model.eval_model(validation_df, f1=f1_multiclass, cm=conf_matrix, acc=accuracy_score)\n",
        "print(result)"
      ],
      "metadata": {
        "id": "4-wgXwD2jNPD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222,
          "referenced_widgets": [
            "ecc9509491c648d9851f23771f508057",
            "eae6719dda6146c098490b89e3351282",
            "7d269da0b5a84ec285bd438528fe55a3",
            "347ca5c5d42d416fa37237e57af040ca",
            "acd3265dc8d54b28b68ef61c136055f5",
            "22d9e75a0c3f47f9800f256ee1d92388",
            "e0b12dffacee4700bcbb3431d66e2821",
            "72e3fbdb7aba47f7a875f418ef3d54d7",
            "35af3236ec5b47078b957198053c8e83",
            "5654ca2669bf4e7992822197934bf09a",
            "1e26806f164946b8a9adbe22d69505bf",
            "1cf1402e3a044d8e8fc245a5c9254dff",
            "5a4a5eb4f1724b7290578ce1499d119e",
            "3d2fa79a53aa41da9118a8c6bce83bf6",
            "b3f64198f3ff4720813d73eab143a39f",
            "e765cec9405843038df0fe0dfaee7577",
            "036e5986a79547be87a2c6034dc5bc76",
            "40ce8b4e50624df9853444a6d9d27f8d",
            "57cce49da2894e2f8306a866aca82bca",
            "93bf6611aa54475bb55cc25d1511bed3",
            "efd5ee9c08f6429483ee2bd05f8ee4de",
            "6f410d2afe6e4fc28f0f6eb46acb31fe"
          ]
        },
        "outputId": "eed9e945-21ab-4a9e-9094-3b0666b85102"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/2049 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ecc9509491c648d9851f23771f508057"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Evaluation:   0%|          | 0/257 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "1cf1402e3a044d8e8fc245a5c9254dff"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'mcc': 0.49005145681030304, 'f1': 0.6412884333821376, 'cm': array([[ 96,  32,   2,   1,   0],\n",
            "       [ 47,  89,  48,   3,   3],\n",
            "       [  4,  49, 106,  58,   4],\n",
            "       [  2,  14,  65, 357, 159],\n",
            "       [  1,   5,  11, 227, 666]]), 'acc': 0.6412884333821376, 'eval_loss': 0.8688666312611056}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(classification_report(validation_df.label,np.argmax(val_outputs,axis=1)))"
      ],
      "metadata": {
        "id": "MmQaruh9MRup",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "47c12ba2-480c-4b3a-b016-6c8f72da380a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.65      0.72      0.68       131\n",
            "           1       0.56      0.34      0.42       190\n",
            "           2       0.48      0.52      0.50       221\n",
            "           3       0.55      0.58      0.56       597\n",
            "           4       0.77      0.79      0.78       910\n",
            "\n",
            "    accuracy                           0.65      2049\n",
            "   macro avg       0.60      0.59      0.59      2049\n",
            "weighted avg       0.65      0.65      0.65      2049\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On Test"
      ],
      "metadata": {
        "id": "vY0pHkn6mhYm"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c00eYvx2MRur",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222,
          "referenced_widgets": [
            "ab13a8fe32634dadb643459344481e9d",
            "9226d414ba6c44aaa33f695f26096974",
            "ea8e16a51ce54b4abb890401ada6895d",
            "eabd8812b54b40ba8e91e2f428f130a5",
            "b58310f2028a4a0a9227a59cc2f2264d",
            "8f1a7ba49f374706973bbfabe568c27f",
            "e46ab0286fc041a89c7380adab64b08b",
            "bd293ccbe1334c5691a93627204a91dd",
            "3c04ada2700342128c824f4c3e058c6e",
            "8faacde480e945ca9c0565e51feafde0",
            "6c53eb2d682f4cfcb1e0c08d5ad55341",
            "f0fd145c131f4587b3e7a772786a65a7",
            "c276a5cd42a447c5bbd0c03ceb633845",
            "08a7150fcb0c4fe4b424aeaeb45f173f",
            "b567dbcaaeae4302b9e69c4808320396",
            "b02fe42e1fd043a597ec1b093ef32810",
            "9ae428a8161f44b5810fe4d82fc1fb4e",
            "bfe784f505d84dbb934d622796665784",
            "98d972ce3302413a9437bc742ebf1949",
            "8bbfc5ac4df84d119028779bba3da64b",
            "00b0029112b9483bb681043d96b21194",
            "cb7394e522ae470cb88510432997fe49"
          ]
        },
        "outputId": "69797ce3-74f6-47e2-d8fe-18286932bdfd"
      },
      "source": [
        "# Evaluate the model\n",
        "result, model_outputs, wrong_predictions = model.eval_model(test_df, f1=f1_multiclass, cm=conf_matrix, acc=accuracy_score, clf_report = class_report)\n",
        "print(result)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/2050 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ab13a8fe32634dadb643459344481e9d"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Evaluation:   0%|          | 0/257 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f0fd145c131f4587b3e7a772786a65a7"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'mcc': 0.525265369065681, 'f1': 0.6619512195121952, 'cm': array([[126,  41,   2,   1,   0],\n",
            "       [ 39,  87,  40,  14,   0],\n",
            "       [  8,  31, 125,  52,   1],\n",
            "       [  1,  12,  74, 345, 138],\n",
            "       [  1,   5,  14, 219, 674]]), 'acc': 0.6619512195121952, 'clf_report': {'0.0': {'precision': 0.72, 'recall': 0.7411764705882353, 'f1-score': 0.7304347826086955, 'support': 170}, '1.0': {'precision': 0.4943181818181818, 'recall': 0.48333333333333334, 'f1-score': 0.4887640449438202, 'support': 180}, '2.0': {'precision': 0.49019607843137253, 'recall': 0.576036866359447, 'f1-score': 0.5296610169491526, 'support': 217}, '3.0': {'precision': 0.5467511885895404, 'recall': 0.6052631578947368, 'f1-score': 0.5745212323064114, 'support': 570}, '4.0': {'precision': 0.8290282902829028, 'recall': 0.7382256297918949, 'f1-score': 0.7809965237543454, 'support': 913}, 'accuracy': 0.6619512195121952, 'macro avg': {'precision': 0.6160587478243995, 'recall': 0.6288070915935295, 'f1-score': 0.620875520112485, 'support': 2050}, 'weighted avg': {'precision': 0.6762443064737604, 'recall': 0.6619512195121952, 'f1-score': 0.667129175811563, 'support': 2050}}, 'eval_loss': 0.8421840965747833}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On raw text"
      ],
      "metadata": {
        "id": "AN3OInedo89r"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "predictions, raw_outputs = model.predict([df.text.iloc[5]])\n",
        "predictions, raw_outputs, np.argmax(raw_outputs)"
      ],
      "metadata": {
        "id": "E_ev_Il-pBDW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 133,
          "referenced_widgets": [
            "f7059e9e3961489b973a12b6fcdb557a",
            "32d002b05f54441687469439ffcc7abf",
            "ecdd1698d5724e7fac2a3ff6589acb93",
            "e55dcbbba27a41fda7a4c63b9415e62a",
            "9ba4ee7407c74966b54d3649d45c989d",
            "294fe24cf7c7429eab76aacd32acee70",
            "17060d877cc348d189957486b6eb7134",
            "891dadc44fcf44b09d160781e941d53c",
            "57c1e1a813e047d39a14adaa608595ca",
            "4399140c09274c74ac555e42eccb34b4",
            "2c84be5a9b50474fb64006073e80b6f8",
            "e2310c5b716a4af2b59f0743dba8a3e6",
            "c98e637b18364720ab99856b3a97064b",
            "c7326f62ef5a4bbba55c25a6ddd1d39f",
            "2d380e03028e4f54955d0feb0ab70b4a",
            "8661ce24a7e44048becfc61cf45d3b39",
            "e8c018457cbb465ab84aef744df63f78",
            "50fe9b5e1024417184dac5f449296475",
            "0ccadc27f79a4bcc900d36ed258d3e86",
            "9c7a40d6c5e145a699756b143f1612af",
            "c453142acae94737ac2b6cb8a00569c0",
            "fa0ebf3c4d7746ddbbb460f57f7513e8"
          ]
        },
        "outputId": "6e34e5f5-a816-47d9-818e-aa635b83b12b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f7059e9e3961489b973a12b6fcdb557a"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "e2310c5b716a4af2b59f0743dba8a3e6"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([4]),\n",
              " array([[-2.28515625, -1.69335938, -0.96923828,  1.92871094,  3.77929688]]),\n",
              " 4)"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# DistillBert(Handle Class inbalancy)"
      ],
      "metadata": {
        "id": "kINbb8yIwFff"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from simpletransformers.classification import ClassificationModel\n",
        "\n",
        "args = {\n",
        "    'reprocess_input_data': True, \n",
        "    'overwrite_output_dir': True\n",
        "}\n",
        "\n",
        "model = ClassificationModel('distilbert', 'distilbert-base-uncased', num_labels=5, args=args, weight=[2.88, 2.28, 1.87, 0.68, 0.45])"
      ],
      "metadata": {
        "id": "SAdfcXBTwFff",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 284,
          "referenced_widgets": [
            "3f3b1de2c532493ab1a16b0300655493",
            "f1b76d0997ba4e55987286c1779dbd47",
            "a923af6f5ce142c4acf51606662554bf",
            "1762e17b31af4ae6b120baef8153f29f",
            "cdbd6a3d674b4d5281054b3b3663cacb",
            "cf81916081d947f68e8e2d4d09828ee7",
            "6c592e16de6245ae9a230b96dab21ac3",
            "26726633143f4fb790ace3d63b60a044",
            "df4bda411497490a885f61a606eb0ae5",
            "4e57705b970e48c784b1039fd85ab67b",
            "994c34ac741540f982c0aa12b76a2954",
            "bf811802ead54bc69789f3a4c58c8173",
            "dc3d0fd2b51746fb9733dd407164adfb",
            "a931e55c089d4bada5b15c1d1342e85a",
            "511a0df508c24b948eeb2941c503b7c8",
            "85945dac46e64fb8ac27d4432a5d3a44",
            "ab6a91210fd74c6a86eefda8bd407eac",
            "be2b600090cf44d4a340a70def1a6509",
            "4067b8cc763e4ecb9f8f59abc07bb9c5",
            "f5cd5b3ace0b4e0e941e9760086721e6",
            "62a8e3d77a6845dea7d778d04f548441",
            "64ed0a500a4c4094bf7b00796abd9ba1",
            "076318912a5b4e1a85e438abc1248934",
            "7e2ce96eaa8f4f52a4d3838dac1ee875",
            "1193850e0caa4583b2ccbe04dd1510d9",
            "c09fd194d1b94687834fb9468949e188",
            "d3301483e7e84dd49b553a505019936d",
            "26a10fe551a044319e104d2afa28ff3f",
            "7cf9fcdabd9148deb18f6265e97b6064",
            "84ba86500d714541bdc08d6b60fa55bc",
            "b3cad876ee6f4ce1b117d42c55908d20",
            "1c33711f4ee84c08877512e8f1ac1cf0",
            "7deb27c207664490b47efd68dc8aa5c8",
            "f2d59c22e07f4f00a8a124f3486edd21",
            "1fa8e8be460e4872ab44c9815b2931a0",
            "52fb9fa3398142bd816906f4408ecc08",
            "90934f2735b543aba340306ffb418eec",
            "fdd67138198c436ab84c0e5b4757bd11",
            "b2f608688f77479c9663cc95ee493ffd",
            "18d1be4d972d44f2a88a34ce00d7acd5",
            "907cc617251244ecbe26aa2bf6606442",
            "2756edbf03804f428c15e0c8d6c674c9",
            "a886b581cd124bde86325b759f97de59",
            "d4b18b657e65495882ec82494c4e94c1",
            "6651ac898cbd42caa9a5b142ae4762ea",
            "773b99fbcc2b448582936ed16c6c5857",
            "b40aa19ae36a4fcf899b07fed0c7771c",
            "84086e75dd2c435fa8d88da8e907ee50",
            "ad7999f1fb974969bf897441007dd541",
            "0d51c6c448fc43b99118371bc0782a11",
            "dd028f23aebc448eba09be49acd4a9dd",
            "2ddc90f71be04e8ea58af037591de5e2",
            "4f7a355c619d4e148e9ccdb826d0e95b",
            "25d33e6d6d5d4c9cad0159dddcbd0216",
            "c526ee131489467d9626b81cf28982f7"
          ]
        },
        "outputId": "a8e88c24-a8bb-4741-ef58-7a23b020384d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/483 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "3f3b1de2c532493ab1a16b0300655493"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/256M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "bf811802ead54bc69789f3a4c58c8173"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_projector.weight', 'vocab_transform.weight', 'vocab_projector.bias', 'vocab_layer_norm.bias']\n",
            "- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias', 'pre_classifier.weight', 'pre_classifier.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "076318912a5b4e1a85e438abc1248934"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/226k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f2d59c22e07f4f00a8a124f3486edd21"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/455k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "6651ac898cbd42caa9a5b142ae4762ea"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Inspect Training Arguments"
      ],
      "metadata": {
        "id": "-THS6Ok_XCSH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"train epochs : \", model.args.num_train_epochs)\n",
        "print(\"max_seq_length : \", model.args.max_seq_length)\n",
        "print(\"train_batch_size : \", model.args.train_batch_size)\n",
        "print(\"learning_rate : \", model.args.learning_rate)\n",
        "print(\"model_name : \", model.args.model_name)"
      ],
      "metadata": {
        "id": "HBogMT4XXCSI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "53544a17-13de-46f3-e36b-e54a7f5ef9cf"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "train epochs :  1\n",
            "max_seq_length :  128\n",
            "train_batch_size :  8\n",
            "learning_rate :  4e-05\n",
            "model_name :  distilbert-base-uncased\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print('Training on {:,} samples...'.format(len(train_df)))\n",
        "\n",
        "# Train the model, testing against the validation set periodically.\n",
        "out = model.train_model(train_df, eval_df=validation_df)"
      ],
      "metadata": {
        "id": "1oZ9jlbVwFff",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 220,
          "referenced_widgets": [
            "17596e3278a14810a78c321637b9ea6f",
            "90a5af8e14ce41d5bfb82b7f84dcb4e4",
            "356f880ec9f24baaa3a624c505edb451",
            "e8ea381047db4d898375bb294a9b3b92",
            "4f905f63f4ed4ed8b2f42bdf30b8fa92",
            "41cd9d821bdb4fde89cc9dabdbdf2bbb",
            "98140fd3ddfa452291f1da4fdf38e718",
            "c3df2f9f391841859076d75b5cd59f3c",
            "6b702d6a5c31410d80a11c548cc310f8",
            "9722cd01d2a34af4b27eaecf412f1f0b",
            "2654282712b94270a0b8c31828d08a3b",
            "2e427064a37343cdb44504cb2f28bfb5",
            "b85f871e26a04eb9a5af29de7d8d0b21",
            "f895580f116e401c8999b2aa3ad74e5d",
            "27a07dab82254c6f9deea39e00950a10",
            "b876e3c81b124674b94cb255a98f5bdc",
            "31ec81dfdff74d2c838ab7d03e4019d1",
            "301e526e14e048e8baaa060ac10a79aa",
            "fe090e1fa0674b21a3dc06da5916f03b",
            "7deb69c3bb554f9bb860c524a516e114",
            "3abdaf03a46f423e826a70483d08c111",
            "5d765cda3e7c4f53b9d894208cdc35ca",
            "6d976e0739914d15a958b157f2865e17",
            "49cce310b01e4e8a8e7e31966669fa84",
            "18d8eb8d2ac541c3903f1ce15c7f4366",
            "d8178166354f45a68358986561c99d82",
            "111f421d87f646d78abdca360b25d953",
            "447302410f1d4c8e91b349c2c84d52ee",
            "797923358c24466688af9c064447f1f1",
            "9c991730d2594bdda25bb6d4025e4eea",
            "885aa3ae828743cd9626e259248d452e",
            "fffb479e16ca4d9681c289d985a8e83c",
            "daed4f6de80e4f3ea66901234bc00695"
          ]
        },
        "outputId": "e86e7dfa-4cda-43c8-b4c6-bbebbc06e439"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training on 16,392 samples...\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:602: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/16392 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "17596e3278a14810a78c321637b9ea6f"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:310: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
            "  FutureWarning,\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Epoch:   0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "2e427064a37343cdb44504cb2f28bfb5"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Epoch 0 of 1:   0%|          | 0/2049 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "6d976e0739914d15a958b157f2865e17"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kiSeT-4YwFfg"
      },
      "source": [
        "## Model Evaluation\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OfoqKmsWwFfg"
      },
      "source": [
        "# model = ClassificationModel(\n",
        "#     \"bert\", \"outputs/best_model\"\n",
        "# )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On Validation"
      ],
      "metadata": {
        "id": "XSybv8T4wFfg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import f1_score, accuracy_score\n",
        "def f1_multiclass(labels, preds):\n",
        "    return f1_score(labels, preds, average='micro')\n",
        "\n",
        "def conf_matrix(labels, preds):\n",
        "    return confusion_matrix(labels, preds)\n",
        "    \n",
        "result, val_outputs, wrong_predictions = model.eval_model(validation_df, f1=f1_multiclass, cm=conf_matrix, acc=accuracy_score)\n",
        "print(result)"
      ],
      "metadata": {
        "id": "6JNjQDVNwFfg",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222,
          "referenced_widgets": [
            "2793506281c84c8e96204ea7a0952b8c",
            "d77d28acc8fc467181c4337efcd97fae",
            "17c471365ec54881a594b44a4c0a53eb",
            "66e50de7a6534c8d8d38c4bad108ea33",
            "d5387ed7bcd14812bacc5a9b204f1020",
            "f595b6c4ea8d48738e0e25edfa518a5c",
            "a96123bc93214d518f8bc5357830eedc",
            "65da1c17363b44efbc397af540b4a326",
            "8815058fa16a43ef82157fd6218e315b",
            "4c93a44cbb764e538685509607ba1584",
            "d8f72c1b384648a8916091bc220882ff",
            "10c6ab4dd32345d4adc4c96e98839460",
            "e680bfef16544a369af19ba03a850349",
            "c146236d97d64529971e29becb67c8f8",
            "40414c37a84945d6881ab1f0a4a2b0bd",
            "2cef115856644e71b49182d662d33641",
            "a7522f96596246309914f34ec6386152",
            "875c4ee155c246fcbe5f6b53f22611fc",
            "2b3bddf56e9143ba859172a284b8eb7f",
            "f041e10718e14c58807b9ce888ff6560",
            "ecb3e83a5d9e432c971a6911516bd255",
            "cb6c665e42604caa9c63ae375e0068ec"
          ]
        },
        "outputId": "dcbd4222-725d-4b26-b796-8dda8a6a0ed8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/2049 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "2793506281c84c8e96204ea7a0952b8c"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Evaluation:   0%|          | 0/257 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "10c6ab4dd32345d4adc4c96e98839460"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'mcc': 0.48300126555636014, 'f1': 0.6364080039043436, 'cm': array([[ 97,  31,   3,   0,   0],\n",
            "       [ 40,  94,  49,   5,   2],\n",
            "       [  2,  48, 109,  55,   7],\n",
            "       [  0,  19,  87, 322, 169],\n",
            "       [  2,   6,  13, 207, 682]]), 'acc': 0.6364080039043436, 'eval_loss': 0.8958661991102687}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(classification_report(validation_df.label,np.argmax(val_outputs,axis=1)))"
      ],
      "metadata": {
        "id": "wVyAPKyfwFfg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a7dd59a8-55a3-49a4-a7b9-c84acbe0c4f4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.69      0.74      0.71       131\n",
            "           1       0.47      0.49      0.48       190\n",
            "           2       0.42      0.49      0.45       221\n",
            "           3       0.55      0.54      0.54       597\n",
            "           4       0.79      0.75      0.77       910\n",
            "\n",
            "    accuracy                           0.64      2049\n",
            "   macro avg       0.58      0.60      0.59      2049\n",
            "weighted avg       0.64      0.64      0.64      2049\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On Test"
      ],
      "metadata": {
        "id": "jGRK8fUfwFfg"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vBoO4Rw_wFfh",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222,
          "referenced_widgets": [
            "9c7acfc0f9a947ee96c6f9a30eb7a772",
            "600e9844a5374ba2920ebb6857923cca",
            "e5b61f9ee065468b99418de46a175ba1",
            "7f25a66e326842a79246f87808ec7510",
            "2c3d4350226e48b5852a5eb3ff7046db",
            "82072d8f2d6d405bb6e75b35517edfaa",
            "c8351c3c57164b0e8764a1b29077c7c3",
            "57fec60d479b4d4c86b3046615926c46",
            "b8a462b833df47bfa19d1b582b67e9f9",
            "028286f5321549a685c08e877827c593",
            "3d544e8c85f54629892ed9e4202e5194",
            "0a869b57cd914cf99a26f7eab6a1dc9d",
            "365fc25813cc4616b36c587a4834d1aa",
            "5987d3c1938342b4a413ef3adb3c3058",
            "5c1ae1ce7cc04b02bb5cafd33b26bce4",
            "bbf96b45d5fe4df38d7a707afce0b63d",
            "a8259d6346944e10bacd46928b0c1c62",
            "9924a98a87d94685b3f6a01f70c36ce6",
            "665ce9ea5e0d4693b50205c756e324e5",
            "986f2d7ca2034b56a3281832bf96a6e3",
            "a2343870057d4bb287fb54b335a896da",
            "d76afb695c204cca81fd7c0a5ad41de3"
          ]
        },
        "outputId": "c98b35f3-e7e8-4595-9758-e470aad67b0b"
      },
      "source": [
        "# Evaluate the model\n",
        "result, test_outputs, wrong_predictions = model.eval_model(test_df, f1=f1_multiclass, cm=conf_matrix, acc=accuracy_score)\n",
        "print(result)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/2050 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "9c7acfc0f9a947ee96c6f9a30eb7a772"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Evaluation:   0%|          | 0/257 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "0a869b57cd914cf99a26f7eab6a1dc9d"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'mcc': 0.4927383190101385, 'f1': 0.6390243902439025, 'cm': array([[118,  48,   3,   1,   0],\n",
            "       [ 35,  96,  35,  14,   0],\n",
            "       [  5,  43, 111,  56,   2],\n",
            "       [  1,  15,  86, 314, 154],\n",
            "       [  2,   5,  19, 216, 671]]), 'acc': 0.6390243902439025, 'eval_loss': 0.8843171392433374}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(classification_report(test_df.label,np.argmax(test_outputs,axis=1)))"
      ],
      "metadata": {
        "id": "v9eUgvXfwFfh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "89f68314-2e8b-4ed2-a582-362f77b1643c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.73      0.69      0.71       170\n",
            "           1       0.46      0.53      0.50       180\n",
            "           2       0.44      0.51      0.47       217\n",
            "           3       0.52      0.55      0.54       570\n",
            "           4       0.81      0.73      0.77       913\n",
            "\n",
            "    accuracy                           0.64      2050\n",
            "   macro avg       0.59      0.60      0.60      2050\n",
            "weighted avg       0.65      0.64      0.65      2050\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On raw text"
      ],
      "metadata": {
        "id": "BRMtnpJ5wFfh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "predictions, raw_outputs = model.predict([df.text.iloc[5]])\n",
        "predictions, raw_outputs, np.argmax(raw_outputs)"
      ],
      "metadata": {
        "id": "MTempoJIwFfh",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "referenced_widgets": [
            "fd234aacdfb64f6eb0d7f89cc77aa31a",
            "07e752756a3c4fb2838f3b7da39c47a3",
            "ca9cd803e9de404fbf9b49c40ba97aa4",
            "94e7a151a0f14bc0b2ac6ae37171d67b",
            "81dda269a41147a9bef487bb9214d20f",
            "bca434dc9e6c4e16b5fa0b3f2130e1f4",
            "58e88237aef24fd0841f5af5a6a06003",
            "1e42d08747004de0bc969317cfba533b",
            "cb518767eda24f9d9011fc61ce2b1b9f",
            "740ad644098947aa8836ff03c20fea48",
            "428f7412af154cb2ad656f8c7c8c9ffa",
            "25b366f79c72485f87e46fcdc332dd62",
            "7dcabe3f382a4fc5b0bae4c87994d1b5",
            "2f13c802c9ee42f7a15d79bee8a6d7e5",
            "fbb64ace75c548dc85b7aa077888c1dd",
            "6c093e48bf6f4639afc8dbf86b218383",
            "35c31cc18f8841a9aa5869f5a3f73f53",
            "ccf6b360d7394560a70b04fb8edf9bff",
            "eb99c2c3e49e44aa8abbb4c0c4c084c2",
            "c1052879de59462fba9b5059e971c3a0",
            "3d8f7948ffd947b593402880a1842db1",
            "6a9f7c734f344840a2979e59b6d5d23c"
          ]
        },
        "outputId": "07c98185-3b20-47b5-f703-c2949fac2eb3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "fd234aacdfb64f6eb0d7f89cc77aa31a"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "25b366f79c72485f87e46fcdc332dd62"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([4]),\n",
              " array([[-2.81054688, -1.96972656, -0.94921875,  1.93554688,  2.92382812]]),\n",
              " 4)"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Albert(Handle Class inbalancy)\n"
      ],
      "metadata": {
        "id": "t8fzWG5hw7rE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from simpletransformers.classification import ClassificationModel\n",
        "\n",
        "args = {\n",
        "    'reprocess_input_data': True, \n",
        "    'overwrite_output_dir': True\n",
        "}\n",
        "\n",
        "model = ClassificationModel('albert', 'albert-base-v2', num_labels=5, args=args, weight=[2.88, 2.28, 1.87, 0.68, 0.45])"
      ],
      "metadata": {
        "id": "Na9sLXr7w7rI",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 220,
          "referenced_widgets": [
            "a25fe64e127049e6ab36c74cf6915508",
            "127f6105d06e4d848e74258a77fea8b5",
            "8e739b1cdc504ce18666fd9dde1397a0",
            "18a364deab124c7fa6e510b03eebc514",
            "ec523360e74b4a23a6ee2fd172d4ef83",
            "55d7721a0d474fe09bba801498932660",
            "21ab919276634eea8313d163be3035a8",
            "fc4d7935ef834c6a8c051451f388fede",
            "e047b599e7b1448b9908e7ff25cc8ae9",
            "e2926eca697e43448facfcd6547be098",
            "1579e26efb9f4665b4d674e32f872389",
            "92413e03d5ce483fae97e335173a6fd5",
            "1e1dde4a9c3f4db4a6c4a5ec13f43193",
            "b6d5d70f3a8841c3ba34849307ed6b2d",
            "58f3725d08f3466797b2862077516f1a",
            "b616a6306c69403d85e9d2d70d2a4224",
            "396a41d5270a4b7c9a19672d525bf471",
            "8caad73b16d64bd1ba132b364d2a6888",
            "c09c410c33074ea7a0ea131722be72be",
            "ed8262b7dfb54949baa699a83c163c9f",
            "ed6ae93a3ac04062a4c56a925d87634d",
            "1afc0778f33a46b7ace3deae7f06f0a4",
            "53f789445cfa497cbbe646d2ef80c172",
            "6ec2f9665be042389010a78d6ab0a079",
            "5e78bc317c224d978d3c0dfd18b785ca",
            "6087005d4a29484a8c51f83a6eec3dbf",
            "ced3664396b242a49e187b871bf87203",
            "ebf689ea72134b758324a19d21c4eac6",
            "5ab1db6895904260b2fc01de85964711",
            "68e2a7f5690a4c98aae9154650bf0fea",
            "c0d58f15701741efbf53995176334cd7",
            "1817bea4f46e4eb085dbf03ee3024ca6",
            "ba54a08dff1648a9b99b67d64a361c5c"
          ]
        },
        "outputId": "76a4466e-c760-4bb8-8eee-3e753f2eacd5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/684 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "a25fe64e127049e6ab36c74cf6915508"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/45.2M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "92413e03d5ce483fae97e335173a6fd5"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at albert-base-v2 were not used when initializing AlbertForSequenceClassification: ['predictions.dense.bias', 'predictions.LayerNorm.bias', 'predictions.decoder.bias', 'predictions.bias', 'predictions.LayerNorm.weight', 'predictions.decoder.weight', 'predictions.dense.weight']\n",
            "- This IS expected if you are initializing AlbertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing AlbertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of AlbertForSequenceClassification were not initialized from the model checkpoint at albert-base-v2 and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/742k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "53f789445cfa497cbbe646d2ef80c172"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Inspect Training Arguments"
      ],
      "metadata": {
        "id": "c95d2LP5XGFm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"train epochs : \", model.args.num_train_epochs)\n",
        "print(\"max_seq_length : \", model.args.max_seq_length)\n",
        "print(\"train_batch_size : \", model.args.train_batch_size)\n",
        "print(\"learning_rate : \", model.args.learning_rate)\n",
        "print(\"model_name : \", model.args.model_name)"
      ],
      "metadata": {
        "id": "pfdR3ADmXGFn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9f037c28-fdb2-4f93-e83b-1ecea5055a8b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "train epochs :  1\n",
            "max_seq_length :  128\n",
            "train_batch_size :  8\n",
            "learning_rate :  4e-05\n",
            "model_name :  albert-base-v2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print('Training on {:,} samples...'.format(len(train_df)))\n",
        "\n",
        "# Train the model, testing against the validation set periodically.\n",
        "out = model.train_model(train_df, eval_df=validation_df)"
      ],
      "metadata": {
        "id": "w-5zJjkZw7rJ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 254,
          "referenced_widgets": [
            "71705b2120ff4a1eae00cefd4ae3f9b7",
            "5cdab61513cc4ce99494a2c6eff9dd0f",
            "080d094fa6cb4e68942edb29d428c57e",
            "393800ad1bf14178b712768aa360f662",
            "29457f3d055c4a27b54f8b996a52be44",
            "f79b9d3bd2df4958829478d37d418036",
            "d70b28ae3d3e42bba33f8a8c355cdce0",
            "3c84d7e733c1422c8137a01a0322f83a",
            "56aa065f099a410f9bc26533f21a4070",
            "c0d8c822a7d7435db5f75379290fed24",
            "0cf5aa58cdba4636a85077bae5230fbd",
            "5c5f1b3498924a87a27a10ba99a1e41f",
            "d10433d33f40472289a3906478c90a91",
            "166701294b4e469b8b4e81ce82e2a77f",
            "d920f34c6a5348bda5333f26b4f46a7a",
            "155cc92b5fe64a429caf0f9ee13a284e",
            "39ffb7df9d414b47a010101c544194c5",
            "15bb592979734b369d5acbfde69db667",
            "468181dc5b9941a5962c5f2f60be93c1",
            "885cec87e3e24efb8d0bc136fa3adb20",
            "a610fb06e8de4352b95341495185601b",
            "0c21ff0269564c848594fe3f11dbbe2d",
            "f27e419b51d84c209e49e104bc2f1f67",
            "bfe287858b074b7ab9e6f789748aa524",
            "96310accec9344b199bf68339f5cb05e",
            "03977cb2f15b40eab7f82b938235e520",
            "54ad2f68304a4cc289b7d0ea9f2aa160",
            "81a9a7ff237847fc944ddd55e90ab195",
            "ad110a349e1b4f038e63ac1bb2004043",
            "4ecc59bf86384799af0548734e457e2d",
            "431fa0fcd86c4c9ea347df30d141060d",
            "a37c5881e7ae4f1e8cad87755eebc93a",
            "5d7c0dcab7b54a9faaac2d8978453405"
          ]
        },
        "outputId": "55a6d4bf-efc7-4b3f-fcf3-603f1e9c9719"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training on 16,392 samples...\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:602: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/16392 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "71705b2120ff4a1eae00cefd4ae3f9b7"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:310: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
            "  FutureWarning,\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Epoch:   0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "5c5f1b3498924a87a27a10ba99a1e41f"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Epoch 0 of 1:   0%|          | 0/2049 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f27e419b51d84c209e49e104bc2f1f67"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:136: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\n",
            "  \"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\", UserWarning)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fYn3lsgYw7rL"
      },
      "source": [
        "## Model Evaluation\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On Validation"
      ],
      "metadata": {
        "id": "oxy_rKYUw7rN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import f1_score, accuracy_score\n",
        "def f1_multiclass(labels, preds):\n",
        "    return f1_score(labels, preds, average='micro')\n",
        "\n",
        "def conf_matrix(labels, preds):\n",
        "    return confusion_matrix(labels, preds)\n",
        "    \n",
        "result, val_outputs, wrong_predictions = model.eval_model(validation_df, f1=f1_multiclass, cm=conf_matrix, acc=accuracy_score)\n",
        "print(result)"
      ],
      "metadata": {
        "id": "ANRusBS5w7rN",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222,
          "referenced_widgets": [
            "83fe1a70ddd64445a68ab231d1ffbb4a",
            "2fee0ebb79cf4b17a18bd73e19a608e9",
            "e78c9cb6d2624f4e88c7c7f2dea16198",
            "ee9db8bf56764836a48553149b983837",
            "5e2adf8cc56040859b8d6744c31bed38",
            "21a7ff1a6f4a42798ad400b985fad74c",
            "c3524e9295a246a6acbbabe2a438dcc3",
            "f8f8d0b5df884301867fe73517153d21",
            "82770990d6274646a9608112b54eaf95",
            "4e7e7b84221a4479bd9738dcda96f73d",
            "9c39318d08dc4edd963864fa9ba25f8d",
            "0cb54431f5ed40b296f559cdb80d3849",
            "ffd6d5c5952a429d8d1754f4821fed69",
            "422e55cf1f0743448a24a3e9c1153e12",
            "729f17c59e5f4974a03e2f1960c0abba",
            "d84a10794fa24f6c98e3287bb60d4ec5",
            "c21a49ce1e774c2cb7360309513c0ea8",
            "b7dc15c393d64ffdbe2f5d3ed31d3f50",
            "8834f9126ff2440baa7a649337f047b6",
            "e195de65a7fe42c59b9dacce90025f27",
            "064d0bf64f904240b81f7130e0482a0e",
            "10defbacb1b44af0a6a85a6e44112990"
          ]
        },
        "outputId": "2fc38f5c-ca44-4c37-aed5-93ffcc91ab29"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/2049 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "83fe1a70ddd64445a68ab231d1ffbb4a"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Evaluation:   0%|          | 0/257 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "0cb54431f5ed40b296f559cdb80d3849"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'mcc': 0.38716628377556467, 'f1': 0.570522205954124, 'cm': array([[ 93,  32,   6,   0,   0],\n",
            "       [ 52,  73,  47,  16,   2],\n",
            "       [  7,  60,  72,  72,  10],\n",
            "       [  2,  14,  76, 294, 211],\n",
            "       [  0,   8,  31, 234, 637]]), 'acc': 0.570522205954124, 'eval_loss': 1.0570328591630616}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(classification_report(validation_df.label,np.argmax(val_outputs,axis=1)))"
      ],
      "metadata": {
        "id": "ri4xmoCvw7rP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f1641bf8-ecbf-4af2-8c06-153b92891138"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.60      0.71      0.65       131\n",
            "           1       0.39      0.38      0.39       190\n",
            "           2       0.31      0.33      0.32       221\n",
            "           3       0.48      0.49      0.48       597\n",
            "           4       0.74      0.70      0.72       910\n",
            "\n",
            "    accuracy                           0.57      2049\n",
            "   macro avg       0.50      0.52      0.51      2049\n",
            "weighted avg       0.58      0.57      0.57      2049\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On Test"
      ],
      "metadata": {
        "id": "Q4cPsPMKw7rQ"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "97Tk27Myw7rR",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222,
          "referenced_widgets": [
            "507834cbaac543a69e18f65d4e681b37",
            "57b051220d3c4efd9bd7902143e9b7a4",
            "c3fa008956004f8e98635891d58be600",
            "5f145f3d79804a83975a4c89cf3e98ad",
            "2126770b8b424259bdd2d19d4adbfa22",
            "b68c89978f6b40e38d1b40add55030b6",
            "48a34a3aa52842208f62f277ffd0f8d4",
            "20a2b181ac854f45b71ff4521d68d290",
            "0adc7ff2dd934a46a044a861d88d1f81",
            "e9d0489d43b34ea8bccab7402a37bbfd",
            "fcb363a30db04609806517d67bc9ffb7",
            "ca802caf2fdf418ab53ace32c657fbbe",
            "0953505309c9415193b26e585ee11ec6",
            "df54db6e2d854553b988506cf62860d8",
            "71c578c67ce14d9f80286cb71368beb1",
            "73490a34aa7442b0987a048d41779134",
            "364fae5c03bf4abeb065efa3026ba8dd",
            "cad24d9a750944768f70ea694774761b",
            "ab92a9d71ce64cd0b322925718020cc1",
            "c29cfdecb5b0491d90e60e7d577037cc",
            "eaace2c9d3da41c7aa178aed0b6e644c",
            "4b37f270ee2447b08953dfa964c6f013"
          ]
        },
        "outputId": "e0e530dc-0085-4950-c985-1f66b94ff791"
      },
      "source": [
        "# Evaluate the model\n",
        "result, test_outputs, wrong_predictions = model.eval_model(test_df, f1=f1_multiclass, cm=conf_matrix, acc=accuracy_score)\n",
        "print(result)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/2050 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "507834cbaac543a69e18f65d4e681b37"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Evaluation:   0%|          | 0/257 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ca802caf2fdf418ab53ace32c657fbbe"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'mcc': 0.40282137435463217, 'f1': 0.5770731707317073, 'cm': array([[113,  46,   8,   3,   0],\n",
            "       [ 42,  85,  27,  26,   0],\n",
            "       [ 11,  59,  81,  60,   6],\n",
            "       [  2,  18,  71, 273, 206],\n",
            "       [  2,   7,  31, 242, 631]]), 'acc': 0.5770731707317073, 'eval_loss': 1.0493019080579513}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(classification_report(test_df.label,np.argmax(test_outputs,axis=1)))"
      ],
      "metadata": {
        "id": "4W6lCJlSw7rS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cb9c3fe1-7ab0-44c8-acc6-427f01054dba"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.66      0.66      0.66       170\n",
            "           1       0.40      0.47      0.43       180\n",
            "           2       0.37      0.37      0.37       217\n",
            "           3       0.45      0.48      0.47       570\n",
            "           4       0.75      0.69      0.72       913\n",
            "\n",
            "    accuracy                           0.58      2050\n",
            "   macro avg       0.53      0.54      0.53      2050\n",
            "weighted avg       0.59      0.58      0.58      2050\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On raw text"
      ],
      "metadata": {
        "id": "HnM7tMYDw7rT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "predictions, raw_outputs = model.predict([df.text.iloc[5]])\n",
        "predictions, raw_outputs, np.argmax(raw_outputs)"
      ],
      "metadata": {
        "id": "4-WUs5KNw7rT",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "referenced_widgets": [
            "fcdfde45da5e42dc9a0ef36a599ec951",
            "2b0e797cee75413584e7518bdf670f04",
            "8a0e8929d489479f8a99a8bc3438086e",
            "ceb0fd62b75c41a6afc740a2bcd9fb78",
            "463d4c7874cf4f03835337fffccadd61",
            "624e4d6a79d34091bd643379ec0e3b8a",
            "fe56f652e20c43c8898337fa5018eac7",
            "1a25965bc3364d54866327583cd46fa8",
            "996fe0468bc842a286c3f9af4875529a",
            "5076f2fac1094d52bcbb7d013ac4aeee",
            "4844a3b7d23f48d0a7ced7d28ebf7e56",
            "bd63052f5cc441e8972520acd2d3e484",
            "d6e852dfe913484b8d2165e2f376f3ab",
            "e09d0a150a094c62a03fb4f1015bedc4",
            "2940775301d249c59697febcc15c7ab3",
            "77fa7124854e4a68bdbd04093bff5752",
            "d2148233e8d64447ab98bc288cdb8263",
            "80aed73466584b73b1669b4c347d6ada",
            "5de5b26cf94648c285af5880ec1aef92",
            "3e4e75b992814c689c0fd5fae9eb9823",
            "33e18989a6a349f39bdd0d3015bf3ca5",
            "85104e36ed114551a2642f8da49a47bf"
          ]
        },
        "outputId": "8b0f9558-facb-42ee-b528-f959c2958620"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "fcdfde45da5e42dc9a0ef36a599ec951"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "bd63052f5cc441e8972520acd2d3e484"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([4]),\n",
              " array([[-2.58984375, -1.28808594, -0.14978027,  1.83105469,  2.828125  ]]),\n",
              " 4)"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Roberta(Handle Class inbalancy)\n"
      ],
      "metadata": {
        "id": "7afhBl6Lr2-V"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from simpletransformers.classification import ClassificationModel\n",
        "\n",
        "args = {\n",
        "    'reprocess_input_data': True, \n",
        "    'overwrite_output_dir': True\n",
        "}\n",
        "\n",
        "model = ClassificationModel('roberta', 'roberta-base', num_labels=5, args=args, weight=[2.88, 2.28, 1.87, 0.68, 0.45])"
      ],
      "metadata": {
        "id": "UbLsfKHCr2-X",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 284,
          "referenced_widgets": [
            "7632baf373d64fa29b62bc457e057605",
            "9b0081b63e4a43a49b6078ae30f15be4",
            "7f1f0f3ea0f84b1dbfaa5e8511174178",
            "bf5ccf8ae0314d668cb64cc96315c8ba",
            "1aacac2a8b484390b8036df666845ec6",
            "d69615d9a8204c48a8922de6361281e3",
            "630e44aa4a1e4fa6a01bbccd327dbb50",
            "2e2d4e8c7cf947cb9ea6b5f42733f7c8",
            "b20b29a6b1b143928af8cd181e130d59",
            "693baa76444c4d52bd6e508640c35aea",
            "456c41c43e7b429fa626304e1b3de497",
            "cce5a65413dd4298a88b54274f93dddb",
            "d5c792e4fb9c45c284763c6a5a9a670e",
            "02f64d2e40454701a793373cd1f29ec8",
            "7c06000dd25f4c2cbf1fb168740fdebd",
            "d8f8dbd7f9de4374a2789b08c5ee9d30",
            "b83e9682b1d6458e959cfa20e68c6799",
            "c77baaee72e2482188618f3701fa90bc",
            "0b2919f5bc104df88beea78bd376c40d",
            "b0f00e343c7640beba8e21e2ddfbd119",
            "bcba8d15fde84793bfb44368abb9d3ea",
            "65a72e43fe8a4aa7946e5f6d8bf9bdc9",
            "d5118cbc6ff64b08992c8b8071d9a864",
            "de2391e0fd2b4095955e41012b8378fd",
            "ca54b4e80b7347e1a0450c85e8dd8a3c",
            "911f4a161cfc491d8fb886c25c614cf7",
            "8b39fe6e3be347538573f3cc67fe46e9",
            "7b3cd4f13c34479290ebeae3456b3465",
            "5f2a54307e3b41b58bd13cbe6472ccb6",
            "be3c52f3f63b45f5bfb50f0e1212035d",
            "cf0248d65a4c46bb9d1c038216ec5e3c",
            "c9bc0d42bb794c5282f832c4566b0f64",
            "c4e1e48d7fec492098522ffe0addac8d",
            "b0b5d7a62bb7456da5ea3ba2b85266d7",
            "ce33ebf8f83e4d1fb539952409b6da6a",
            "57378945c25946958679cee2869c736f",
            "38da651f7d83484a9d1bc14d15fb3fde",
            "7a5e8d70ad6740b4bf67f8c8d916ea8f",
            "b492c5b395474c3e899b777893e63257",
            "d9edf655473b4531af3d816857340189",
            "cd47a91689dd4e93bf6d5ad0a4471950",
            "0554c085383e4c48804fa40a79f5fead",
            "15584bd7678e46b6b42ea3de03ee3625",
            "d60fa415eb3a4566b7940a266728296a",
            "3bd0ed79b39b41d5a34c26cf3b96e1b1",
            "a5e12c3c8ff34a3093d288f878cb40f3",
            "a73a565fc9df4493bb88b94fd48f56d3",
            "22767a6480cf43bb837ca8e641875256",
            "1a0cb2b995414c9bab47d05bdadd139e",
            "7be9706b1b674d5bbef2efd6c0457eea",
            "46a2b6b538fa47c5ab7a9483df863292",
            "b704e0fb701546bf8d763c8fa5fe0e74",
            "8b427ac3fe154e09a99137b65dae550d",
            "59f9e127825940198f16c04e2c096615",
            "18e63ca4383b488bb65a35b31b19a77b"
          ]
        },
        "outputId": "aefe8d1b-0db3-41b8-bda3-b844e12cdaa5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/481 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7632baf373d64fa29b62bc457e057605"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/478M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "cce5a65413dd4298a88b54274f93dddb"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at roberta-base were not used when initializing RobertaForSequenceClassification: ['lm_head.decoder.weight', 'roberta.pooler.dense.bias', 'roberta.pooler.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.dense.weight', 'lm_head.bias']\n",
            "- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at roberta-base and are newly initialized: ['classifier.out_proj.weight', 'classifier.out_proj.bias', 'classifier.dense.weight', 'classifier.dense.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/878k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "d5118cbc6ff64b08992c8b8071d9a864"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/446k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b0b5d7a62bb7456da5ea3ba2b85266d7"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/1.29M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "3bd0ed79b39b41d5a34c26cf3b96e1b1"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Inspect Training Arguments"
      ],
      "metadata": {
        "id": "lhbg8DNEXKbl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"train epochs : \", model.args.num_train_epochs)\n",
        "print(\"max_seq_length : \", model.args.max_seq_length)\n",
        "print(\"train_batch_size : \", model.args.train_batch_size)\n",
        "print(\"learning_rate : \", model.args.learning_rate)\n",
        "print(\"model_name : \", model.args.model_name)"
      ],
      "metadata": {
        "id": "N549OlHrXKbm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bc54b92e-4971-4bbb-fcf7-5c87ec9cfe25"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "train epochs :  1\n",
            "max_seq_length :  128\n",
            "train_batch_size :  8\n",
            "learning_rate :  4e-05\n",
            "model_name :  roberta-base\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print('Training on {:,} samples...'.format(len(train_df)))\n",
        "\n",
        "# Train the model, testing against the validation set periodically.\n",
        "out = model.train_model(train_df, eval_df=validation_df)"
      ],
      "metadata": {
        "id": "TfVHJ960r2-Y",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 220,
          "referenced_widgets": [
            "b340ad9083af4d6393efa108f572f674",
            "af66666f614941a4a44d45ffa5957bfe",
            "bbf9ba86be6d4121bf3e3123741eb03f",
            "21d7c1748328432cac9f9354abf0886a",
            "2dda447e991646688a4afad77dd1e462",
            "9571f743a7834e2d91bef13ec5af0c36",
            "84c428d589dd4c07af39ee9b96be0433",
            "bca4f7b117384b21b68e463cb279762d",
            "c1523faaaff64d9086cc3860519d349e",
            "917de8c5fadb4ef18d220b5bf3e811ff",
            "b390a42880cd4e37aa072e6ea0d3d41d",
            "0dbd568d981d4e2cb65d979d403474d7",
            "33934d65b5db4cac824034386a6c947c",
            "9d76d071bd82436a82e879e3dc215cb3",
            "cb6ad6161e3a4bcd8ab0a814bf77c8e4",
            "70dbd08b390d4c60941c8ec3fd3a7198",
            "9b73f88ad77249cfaca2a67be5f593f7",
            "9b888c74817f42bc9d79fc3c5dc65570",
            "a48047cc546f4010a8fb69952bc8c2a7",
            "3ee90f4f3a8141daa427b7925668db3e",
            "4f237c35b70b4dcd81a4a5799aaeb1b3",
            "b3c55e5cca994b0d922d4dff20abe6aa",
            "4ee05fdc59b44e75b36a99f9b4a024d2",
            "41e979246f234596a65fbe6f1851893f",
            "33e1d3ca10d9437a8390e4d77b163300",
            "a7a60d56c27b4617b2e406dbab937301",
            "0c7deda2f0b44bfab987f5832bd2224a",
            "16dc56e4069949078a4cb26198eb7eec",
            "6a2a34579df64db29719b17529dde48e",
            "e752d9741fe844e697d9b47167cddee2",
            "a578b7d217864dc1b457904c85bc3fd1",
            "9adf92e20f0446be9c6003cd64a07989",
            "667913db436e4dedae75032cb51c0134"
          ]
        },
        "outputId": "5df5f3d6-a8fa-4333-df31-d87dca5bf81f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training on 16,392 samples...\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:602: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/16392 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b340ad9083af4d6393efa108f572f674"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:310: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
            "  FutureWarning,\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Epoch:   0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "0dbd568d981d4e2cb65d979d403474d7"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Epoch 0 of 1:   0%|          | 0/2049 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "4ee05fdc59b44e75b36a99f9b4a024d2"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sr-OL53Or2-Z"
      },
      "source": [
        "## Model Evaluation\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On Validation"
      ],
      "metadata": {
        "id": "Re320PdZr2-a"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import f1_score, accuracy_score\n",
        "def f1_multiclass(labels, preds):\n",
        "    return f1_score(labels, preds, average='micro')\n",
        "\n",
        "def conf_matrix(labels, preds):\n",
        "    return confusion_matrix(labels, preds)\n",
        "    \n",
        "result, val_outputs, wrong_predictions = model.eval_model(validation_df, f1=f1_multiclass, cm=conf_matrix, acc=accuracy_score)\n",
        "print(result)"
      ],
      "metadata": {
        "id": "rIQ1DR7Mr2-b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222,
          "referenced_widgets": [
            "01b70a5f61454965ad3fe24081bbf7ea",
            "66e8576c37274ded873245b8bc06389c",
            "71a1f587dac848b396f510a5444800e1",
            "be042735f294433185548ac21810c04e",
            "8a5d27f1d2fa42f38c16b298351b2887",
            "2a3866aee9a5444b929bf759420e1f24",
            "da68bdb02a8f4693af728b19bee74ba0",
            "a0fe0ced220846b58fb799952f298e3b",
            "864a836ab08941da9bd502610186436c",
            "06c9f93dd01d4b6e8acf901fd1c7e39c",
            "b9944e40195b4ac69edcc30ac322004d",
            "947853ca6dc54578bd83fcc048eea183",
            "f5dfd8c948294199b1fe1b8ff99983be",
            "5f84b97bc88d4e0293eb94ce76e3d143",
            "7d22d26920984284818d5ee186cb36ef",
            "2674b5c4dd8c4c77966d2c8f88b13ed2",
            "a2101a1027d948abad4d068f41557eb2",
            "0cb45f187eb1468f805b01c5f15c5615",
            "972d3b81838b496ba662a45db71dfe5c",
            "ca1d6aa50f6e49289de1caf7cd733c13",
            "1edac7273dfa42d9b50b7d9a79eeac75",
            "c534d1d2079541d2b19080739daff1c7"
          ]
        },
        "outputId": "4e94e0f9-99a5-41bb-d29a-0cb033c69d3a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/2049 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "01b70a5f61454965ad3fe24081bbf7ea"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Evaluation:   0%|          | 0/257 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "947853ca6dc54578bd83fcc048eea183"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'mcc': 0.2415652139376959, 'f1': 0.4924353343094192, 'cm': array([[ 90,  21,   2,   1,  17],\n",
            "       [ 68,  31,   7,   0,  84],\n",
            "       [ 25,  19,   6,   2, 169],\n",
            "       [  7,  22,   2,   1, 565],\n",
            "       [  7,  18,   4,   0, 881]]), 'acc': 0.4924353343094192, 'eval_loss': 1.3320476920688198}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(classification_report(validation_df.label,np.argmax(val_outputs,axis=1)))"
      ],
      "metadata": {
        "id": "cx9dvBl_r2-b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3cbe3681-dcfc-450b-a132-a4a703035441"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.46      0.69      0.55       131\n",
            "           1       0.28      0.16      0.21       190\n",
            "           2       0.29      0.03      0.05       221\n",
            "           3       0.25      0.00      0.00       597\n",
            "           4       0.51      0.97      0.67       910\n",
            "\n",
            "    accuracy                           0.49      2049\n",
            "   macro avg       0.36      0.37      0.30      2049\n",
            "weighted avg       0.39      0.49      0.36      2049\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On Test"
      ],
      "metadata": {
        "id": "bLOxFxOdr2-c"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2l8S5FM9r2-c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222,
          "referenced_widgets": [
            "c509e23bf4d6489cbdc953b08ee96b8b",
            "8727b664bcb54cef8dbadb7e3248e76b",
            "31122ea6b7a84e4d82e03dbba5faa3d3",
            "901e13b90b894b019f509915244d10c3",
            "7b9f4dd43d0349e8abbc7f414556743e",
            "9e5b2cf1b46d4bbbb75def36556748d4",
            "b6c6f8f5e983408ab1d68e504396c764",
            "40ffa9ca6dab437a980df2f3c91e19bb",
            "3ceac70fe97744a0b6db65ae2dca80cd",
            "1999f71558cf495faddc4170ca4f48ba",
            "e7914c3c564b4d17a41c85537ca78fd7",
            "3776a37c362647368a6def681496a5a8",
            "c3d558c696364463a04d9226929471d3",
            "263fc5429891463a8ce22deab48d00a4",
            "19ecb5e235274ebfb0525c9cebf19255",
            "40da560d2f73405bb82ac0c4ac1fa696",
            "61b6bd68f6a845f3b05826758c5fa12e",
            "3c44892a749042f58a5e4f5eebed1726",
            "937e1bf8be034afabb86b37024eb630a",
            "7eebc0132e534ee98be11dbf5cff4942",
            "3b7dc1ad9ca241e3871745d3558c28ff",
            "9a5d890b344749988b56f2a95b4ed2a2"
          ]
        },
        "outputId": "29929f5d-7259-477b-d4e2-67c213171600"
      },
      "source": [
        "# Evaluate the model\n",
        "result, test_outputs, wrong_predictions = model.eval_model(test_df, f1=f1_multiclass, cm=conf_matrix, acc=accuracy_score)\n",
        "print(result)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/2050 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "c509e23bf4d6489cbdc953b08ee96b8b"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Evaluation:   0%|          | 0/257 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "3776a37c362647368a6def681496a5a8"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'mcc': 0.26970460566367793, 'f1': 0.5068292682926829, 'cm': array([[119,  27,   4,   1,  19],\n",
            "       [ 69,  29,   7,   3,  72],\n",
            "       [ 25,  22,   7,   1, 162],\n",
            "       [ 11,  17,   3,   1, 538],\n",
            "       [ 16,  11,   3,   0, 883]]), 'acc': 0.5068292682926829, 'eval_loss': 1.3352248819421701}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(classification_report(test_df.label,np.argmax(test_outputs,axis=1)))"
      ],
      "metadata": {
        "id": "ferJ2-n0r2-d",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d89daf27-0eb3-40da-f588-b5a1b1c7f092"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.50      0.70      0.58       170\n",
            "           1       0.27      0.16      0.20       180\n",
            "           2       0.29      0.03      0.06       217\n",
            "           3       0.17      0.00      0.00       570\n",
            "           4       0.53      0.97      0.68       913\n",
            "\n",
            "    accuracy                           0.51      2050\n",
            "   macro avg       0.35      0.37      0.31      2050\n",
            "weighted avg       0.38      0.51      0.38      2050\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On raw text"
      ],
      "metadata": {
        "id": "es0jaUt_r2-d"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "predictions, raw_outputs = model.predict([df.text.iloc[5]])\n",
        "predictions, raw_outputs, np.argmax(raw_outputs)"
      ],
      "metadata": {
        "id": "llMQp986r2-e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "referenced_widgets": [
            "f4a98a86e4f642ff85f4e20035212d7c",
            "63c426c6eda047e9ac4f2a88eb2ce989",
            "4a2bf7a4ec094ac7a10827277d365b72",
            "d68e0b04658f46178580de8e1269036c",
            "4137db2de78b4f7c85e71bc2ef2fbef2",
            "ecd5086508cc4f2196db0da4125f0dc9",
            "5093ebcad40645a089dd55f1a4b35f0f",
            "cef683931aa34c10987dc599c8c6ddea",
            "5786a63a47834dfd996cd6a497699118",
            "3d4f95ddbc6f4153b81d07a7e19b8b33",
            "9121fba3af7a43a5b968979b39878996",
            "1572268f31ea47f4b64045ab15ed0228",
            "9813e8b4e2fc47edaa6dd7613d3a284b",
            "fbb3d6d3a62f41e8acee0867f3ebe05b",
            "90b800b49c444c8f8f040dbce6876521",
            "3e88e57772d7435aadebf47818d4d560",
            "9f4b15ac90f243e6b373e03be1053236",
            "3fac5e468e9247c0af455f1f7516b60f",
            "4dc5ea60a4084434aa5f31721623ce99",
            "43f94bbbc13e46179ef8fcd22c639b7f",
            "fab16186cf114b059629897a94d31851",
            "bedf3a5cf61444b38959136325ba8e70"
          ]
        },
        "outputId": "be370888-2663-42df-e58a-e2fe7d3f0238"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f4a98a86e4f642ff85f4e20035212d7c"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "1572268f31ea47f4b64045ab15ed0228"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([4]),\n",
              " array([[-2.640625  , -0.7265625 ,  0.58496094,  1.16796875,  1.35546875]]),\n",
              " 4)"
            ]
          },
          "metadata": {},
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "VX-Gm0_mwEga"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "AZvFD-uSwFNK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# XLNet(Handle Class inbalancy)\n"
      ],
      "metadata": {
        "id": "CmBcXmA_whrO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from simpletransformers.classification import ClassificationModel\n",
        "\n",
        "args = {\n",
        "    'reprocess_input_data': True, \n",
        "    'overwrite_output_dir': True\n",
        "}\n",
        "\n",
        "model = ClassificationModel('xlnet', 'xlnet-base-cased', num_labels=5, args=args, weight=[2.88, 2.28, 1.87, 0.68, 0.45])"
      ],
      "metadata": {
        "id": "xr1oNDaZwhrP",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 252,
          "referenced_widgets": [
            "68dbd73e43a449c6a4439931dc9cf74c",
            "f796c2bfd07c46648f76a8609a911575",
            "ec6ff381c66a47a6a87c8dfff7e19b4b",
            "967cd991ced8472eabb7bb8251fb794f",
            "7a56c7ab5da24de6a6d8c996dec6b070",
            "b2299936504244e59d534fa1a5dd501a",
            "f186ee2beb864618aee2735e1dbb6d3e",
            "455361a192864226894bcd3b317fb840",
            "f4b8079d2ebd4447a0977b844e6be627",
            "4b8b5a2e662543d9ae0101d837c987ef",
            "ab33d85e6e3947029283acd2234f6a57",
            "865364b6d529480db92cc134f97daaf4",
            "efeeb139ecf249ca85ce37edb8a19f85",
            "d5e6283eb9bb4a78824a4da4d86ab448",
            "b4e7b3f9293f42ab839dfa2d90c78d6e",
            "84cb0bb440d04534b4a8f1a3cc842a01",
            "72a6dd35b4314f7a9d00d4a6e40ee7f0",
            "16be57f5e8e2412a8abf5ee62a6c7752",
            "fdb9b919d77240e7aea910b53eeb694f",
            "28e65e11257441a79a2df0fdc5f050a3",
            "ef9eae0de8e448beae5f4dbaa0676ae8",
            "6d0f863385564979bdba542adc5c421d",
            "38e0d2b83bd349aeb5f3ccf42eacf9d4",
            "96ca997731164dc1a73f93ddd5f5e511",
            "498334e3df9b4b2986a69d80819dc622",
            "c41d0c6b8f42490e8c3e228891de9aa8",
            "4652c8fa59ce4a6cbecacbdaa17b94f3",
            "9fb66f81836b4b3287c9f95a3dc2bb24",
            "823e2e4fe043427da2936e8cbd6bdc7c",
            "b8a350f5c15b4d769cd0d9af45d45c97",
            "1729c9e6a9504279872385384a3f0f10",
            "0065c71d24f2462cb32ad6c3550a3f8b",
            "1e0b7b2d316f431e9f23121117c70913",
            "8859d26ce2a44669a3cc7465ee28fe73",
            "33423aacfd0849ccbb9767a88278a285",
            "3b3b13b3d276453187e8e2e994c1c040",
            "c1550444c8af42809680ddd8b2495a8a",
            "23b935d26bb845babee21fcc5ebcd954",
            "e13e3ee12b1041198a16cc5507d81c3f",
            "d66639aeb0ac4945b4d2dd5ad0bdd385",
            "a21bdd7276fd48d0b39a51a9532f16bc",
            "508b741b14814f2db23a10c4817bb4eb",
            "a0cc8c4565b24443be657cda71b56410",
            "721eab986bd349c2a08211cf4e96cd0f"
          ]
        },
        "outputId": "9d4d9464-a34f-4923-8ede-76661fb11b71"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/760 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "68dbd73e43a449c6a4439931dc9cf74c"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/445M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "865364b6d529480db92cc134f97daaf4"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at xlnet-base-cased were not used when initializing XLNetForSequenceClassification: ['lm_loss.weight', 'lm_loss.bias']\n",
            "- This IS expected if you are initializing XLNetForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing XLNetForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of XLNetForSequenceClassification were not initialized from the model checkpoint at xlnet-base-cased and are newly initialized: ['logits_proj.bias', 'logits_proj.weight', 'sequence_summary.summary.bias', 'sequence_summary.summary.weight']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/779k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "38e0d2b83bd349aeb5f3ccf42eacf9d4"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/1.32M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "8859d26ce2a44669a3cc7465ee28fe73"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Inspect Training Arguments"
      ],
      "metadata": {
        "id": "yU3E8E2uXTWL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"train epochs : \", model.args.num_train_epochs)\n",
        "print(\"max_seq_length : \", model.args.max_seq_length)\n",
        "print(\"train_batch_size : \", model.args.train_batch_size)\n",
        "print(\"learning_rate : \", model.args.learning_rate)\n",
        "print(\"model_name : \", model.args.model_name)"
      ],
      "metadata": {
        "id": "pyhLj7kbXTWN",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "275fc5f3-0bb9-419e-9715-c218b234a0ef"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "train epochs :  1\n",
            "max_seq_length :  128\n",
            "train_batch_size :  8\n",
            "learning_rate :  4e-05\n",
            "model_name :  xlnet-base-cased\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print('Training on {:,} samples...'.format(len(train_df)))\n",
        "\n",
        "# Train the model, testing against the validation set periodically.\n",
        "out = model.train_model(train_df, eval_df=validation_df)"
      ],
      "metadata": {
        "id": "98p7-KdkwhrQ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 254,
          "referenced_widgets": [
            "a42372c6535a43d6a4ee2a9fdd7057db",
            "1d3af713e7544f8e90925080383565f4",
            "f1107ee9d1164180910df2eebb44af20",
            "a1b4fae0bdf04c57970b47b5f8da469c",
            "cd312e9ceb984f20b3ad71f1ad4b4c39",
            "7340602ca7874892a1c9580ddd01da35",
            "2f952273041c46f49567c07eeda4f778",
            "602ce1b465314ea49dcd0325e583266c",
            "83b2bcbf9d6e4aceace99d31002486a9",
            "6a5faffc30734e74a488b8e4bc58ac6b",
            "e9e339d50ffb41b2829337a0c680825f",
            "bbc72b4bfc454bac87692fc1b4254a0a",
            "4bef954412ce4ded9c54957267752f54",
            "bea9054ab6ed469ab908eac3cf1795e7",
            "8019e605f6264790b793577b6bac6d9d",
            "a5d74a089fea4c2f99c23a5af700b9d0",
            "ce54cd2dcab34d09ba406b4ff677e703",
            "a4c193d27f32424fa81b7c2fc1218c07",
            "75402775f2e14eb8ac1c115fe45ae73f",
            "fcb6cd407d4f4e88af3311127574ad89",
            "19fcd90c5242438a84451d4e9bd7bc4e",
            "6739768699a746819d87e69fb6672071",
            "906ddf324cfb48eeb526e6f343b7f5c7",
            "09fca96cd543438bb1c3e285b691eecd",
            "3dbd638da392429dbafcebcc1ac69667",
            "ff4d30255eaf423388388ab195d722c2",
            "546c699c63ba4ff8a1175c55f2499195",
            "3afe717bfc4d4a53ad7fa55a296c1dce",
            "8ff6cff190ef47b1b53a90504b806743",
            "622ddfc0ca8247a0b86e8edce599b2c2",
            "41c2ad5fe14e4d1dbf1e8f2b129ce9ba",
            "45bde7f10f97449e998bb2fb68d5198c",
            "a81fe15ebc8440329e7f4d6fc76802d8"
          ]
        },
        "outputId": "1e5a4d80-7e9b-49bc-af9a-98c27357712f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Training on 16,392 samples...\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:602: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/16392 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "a42372c6535a43d6a4ee2a9fdd7057db"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/transformers/optimization.py:310: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
            "  FutureWarning,\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Epoch:   0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "bbc72b4bfc454bac87692fc1b4254a0a"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Epoch 0 of 1:   0%|          | 0/2049 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "906ddf324cfb48eeb526e6f343b7f5c7"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:136: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\n",
            "  \"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\", UserWarning)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vkOrqt-ywhrR"
      },
      "source": [
        "## Model Evaluation\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On Validation"
      ],
      "metadata": {
        "id": "sGXh1G4BwhrR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import f1_score, accuracy_score\n",
        "def f1_multiclass(labels, preds):\n",
        "    return f1_score(labels, preds, average='micro')\n",
        "\n",
        "def conf_matrix(labels, preds):\n",
        "    return confusion_matrix(labels, preds)\n",
        "    \n",
        "result, val_outputs, wrong_predictions = model.eval_model(validation_df, f1=f1_multiclass, cm=conf_matrix, acc=accuracy_score)\n",
        "print(result)"
      ],
      "metadata": {
        "id": "fF3T1fRjwhrS",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222,
          "referenced_widgets": [
            "13b2d831786446519c129e7d4f666f31",
            "f554f9a7d4d247dab406eb91cc9f6023",
            "5bd1532552164ecb9458bc4f6d731305",
            "ec35b3a9726a4ecaa1f31dd1c3f7a957",
            "1eebfb4cdda74dd9adc3c8224035b72d",
            "0913e468d4424ef19295dd7507106ca3",
            "279aca6d3d1f435faa3758284b5688b9",
            "71cd432b7b00477e9eb041926545fed9",
            "4dcd87b33ba04949925b5bcb3cd21361",
            "d014ed94b16b4d039b7d093487fa4320",
            "9e2e27f701174e89aa18bd742cbc9e18",
            "71d1dd450b4743fe9879a30f29874773",
            "74114d5762fc43b8bad775ae23d51bd9",
            "f5e037bf61954b608c2a6ab2bdbb0a06",
            "111da78fc4d740539ccf83f01e82ac77",
            "9e42fc3b5c7c41cf9f7af726ed6a994a",
            "66d7e1b298e1493cbbf7f1b919c2513f",
            "09e2f67f4caf4ef6ae0ae5de5dc98f65",
            "ea07e90f095a4a42b2edb4d574a61750",
            "65c0f393abde429496ee083ae85e5941",
            "833ce716c94343a182ab9bb96f18729d",
            "8df3665a670a4e31a50de8dae74f62a0"
          ]
        },
        "outputId": "0ebc4e42-07d7-422c-f4ea-0048524283cb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/2049 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "13b2d831786446519c129e7d4f666f31"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Evaluation:   0%|          | 0/257 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "71d1dd450b4743fe9879a30f29874773"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'mcc': 0.48638953925640704, 'f1': 0.6408003904343582, 'cm': array([[ 86,  42,   2,   1,   0],\n",
            "       [ 38, 106,  37,   5,   4],\n",
            "       [  3,  55, 109,  49,   5],\n",
            "       [  1,  20,  78, 306, 192],\n",
            "       [  0,   7,  12, 185, 706]]), 'acc': 0.6408003904343582, 'eval_loss': 0.8973847102910164}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(classification_report(validation_df.label,np.argmax(val_outputs,axis=1)))"
      ],
      "metadata": {
        "id": "EuapmZmNwhrS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2d46c73a-29c6-4eb8-f88a-1cd2322ff19d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.67      0.66      0.66       131\n",
            "           1       0.46      0.56      0.50       190\n",
            "           2       0.46      0.49      0.47       221\n",
            "           3       0.56      0.51      0.54       597\n",
            "           4       0.78      0.78      0.78       910\n",
            "\n",
            "    accuracy                           0.64      2049\n",
            "   macro avg       0.59      0.60      0.59      2049\n",
            "weighted avg       0.64      0.64      0.64      2049\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On Test"
      ],
      "metadata": {
        "id": "gFaKlixUwhrS"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LVbPdAq3whrT",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222,
          "referenced_widgets": [
            "a1c8be9751064b34b3233603e4314057",
            "09c50b21f3ac4ca4981bc05a00ba2b4e",
            "ee36a5abfe3945c29334c56655282b0c",
            "badfee1fa6b64ff2be734a4b946edf82",
            "a8181989cafe4aa1856f6579171c2d6e",
            "90583ee915574355b1f601b6eae60e04",
            "b7314e377c844d2794de8a87c2ec8d0f",
            "7d6a5a1560104a9291597899c2f83259",
            "1f997bfcb0944607a28cbce95e01597d",
            "673fcdca22e2426795ab208d7231a910",
            "f18ba6f2fd0d48598f49316090bf2a4d",
            "2873558c7d4c412da63cbc405138e931",
            "0d02db921be94c8caa0ac88bf9bc4562",
            "b4fdd48297604f898502cb3cbac6f8fa",
            "b986d48e633845d49696a0e341683174",
            "d9405cf9b2b0490ea20bee0d89acf343",
            "137be808ea1a42d08c10a80b86ce2bfd",
            "918dc1bde0f4492983130cd3c500b17f",
            "d6fd90336523437a8e61458404214268",
            "46c6bb5e63984ee880b9f623f877187c",
            "add311853c93405f9a72db999cec606d",
            "1bf17f34a8ea47e5bdc90f278612a4fd"
          ]
        },
        "outputId": "b6c6162b-e83d-4cd5-9e21-0c89cd791d01"
      },
      "source": [
        "# Evaluate the model\n",
        "result, test_outputs, wrong_predictions = model.eval_model(test_df, f1=f1_multiclass, cm=conf_matrix, acc=accuracy_score)\n",
        "print(result)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/simpletransformers/classification/classification_model.py:1443: UserWarning: Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\n",
            "  \"Dataframe headers not specified. Falling back to using column 0 as text and column 1 as labels.\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/2050 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "a1c8be9751064b34b3233603e4314057"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Running Evaluation:   0%|          | 0/257 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "2873558c7d4c412da63cbc405138e931"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'mcc': 0.5151746333765646, 'f1': 0.6580487804878049, 'cm': array([[119,  48,   3,   0,   0],\n",
            "       [ 25, 114,  26,  14,   1],\n",
            "       [  7,  46, 108,  50,   6],\n",
            "       [  1,  19,  79, 297, 174],\n",
            "       [  2,   9,  10, 181, 711]]), 'acc': 0.6580487804878049, 'eval_loss': 0.8944905155015827}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(classification_report(test_df.label,np.argmax(test_outputs,axis=1)))"
      ],
      "metadata": {
        "id": "UhhWmKx5whrT",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1410b5aa-83d2-4974-f0d7-a2d5c4ce5f5a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.77      0.70      0.73       170\n",
            "           1       0.48      0.63      0.55       180\n",
            "           2       0.48      0.50      0.49       217\n",
            "           3       0.55      0.52      0.53       570\n",
            "           4       0.80      0.78      0.79       913\n",
            "\n",
            "    accuracy                           0.66      2050\n",
            "   macro avg       0.62      0.63      0.62      2050\n",
            "weighted avg       0.66      0.66      0.66      2050\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### On raw text"
      ],
      "metadata": {
        "id": "sACsGeJ8whrU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "predictions, raw_outputs = model.predict([df.text.iloc[5]])\n",
        "predictions, raw_outputs, np.argmax(raw_outputs)"
      ],
      "metadata": {
        "id": "K9-r60G5whrU",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "referenced_widgets": [
            "db16723c07c14edea5b4901d0854fed2",
            "2fda7fc521554b6cbf3d93c647d80205",
            "772a02cc14944f1d8897364622ed2f60",
            "d133e8fa0d124577ac7b58eeb2531fa1",
            "5dad9097f7cf4abf83eb581b421390d5",
            "88a62497a4164c298dd0f2c8e933e469",
            "73e8a57addd349908c794ab1fdf9ae55",
            "6e367ae894684ff18a6d231d33eb7f87",
            "a6ffc48a17424bfa87612bb19f498735",
            "cf2317c3d27847f497f646e4b3037bbf",
            "4e84f2daba9843c2b569bed109d45111",
            "b4ee73f172b342bb890484ee88a25bfc",
            "4db9ed1da0cb424d94ef835fb1775185",
            "ac168ef961b942eda07a3099618b7195",
            "d6f5168f6a81485bacda91d3652c281e",
            "566ca8216c124bb5b7e3da3351c489e6",
            "d9f4108d6cee453280a8198e31f8643b",
            "be5aa403832a4bf39b2090d69f0225af",
            "71b87733d26341b08a0349916ef5a0ab",
            "4f7b7b5c80a24ce19e0b93769a9cd23f",
            "014a730add6646b58851f2d38972965e",
            "ce84359e2d7c4adaa015995dac9a79ad"
          ]
        },
        "outputId": "99eb23a1-b4ce-4ac4-d628-7ac858e4e348"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "db16723c07c14edea5b4901d0854fed2"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/1 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b4ee73f172b342bb890484ee88a25bfc"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([4]),\n",
              " array([[-1.98046875, -1.54199219, -0.68408203,  2.60546875,  4.45703125]]),\n",
              " 4)"
            ]
          },
          "metadata": {},
          "execution_count": 65
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#GPT2"
      ],
      "metadata": {
        "id": "Yr5JKDhBsAWS"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Installs"
      ],
      "metadata": {
        "id": "VCksRw4TL8xG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip3 install torch==1.10.2+cu113 torchvision==0.11.3+cu113 torchaudio==0.10.2+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html\n",
        "!pip3 install transformers"
      ],
      "metadata": {
        "id": "8Ys-_KCowyVp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a45355d0-001f-4d11-b033-3ff03780f37d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Looking in links: https://download.pytorch.org/whl/cu113/torch_stable.html\n",
            "Collecting torch==1.10.2+cu113\n",
            "  Downloading https://download.pytorch.org/whl/cu113/torch-1.10.2%2Bcu113-cp37-cp37m-linux_x86_64.whl (1821.4 MB)\n",
            "\u001b[K     |██████████████▋                 | 834.1 MB 1.3 MB/s eta 0:12:54tcmalloc: large alloc 1147494400 bytes == 0x3a4f8000 @  0x7f6f3d7a8615 0x592b76 0x4df71e 0x59afff 0x515655 0x549576 0x593fce 0x548ae9 0x51566f 0x549576 0x593fce 0x548ae9 0x5127f1 0x598e3b 0x511f68 0x598e3b 0x511f68 0x598e3b 0x511f68 0x4bc98a 0x532e76 0x594b72 0x515600 0x549576 0x593fce 0x548ae9 0x5127f1 0x549576 0x593fce 0x5118f8 0x593dd7\n",
            "\u001b[K     |██████████████████▌             | 1055.7 MB 1.2 MB/s eta 0:10:18tcmalloc: large alloc 1434370048 bytes == 0x7eb4e000 @  0x7f6f3d7a8615 0x592b76 0x4df71e 0x59afff 0x515655 0x549576 0x593fce 0x548ae9 0x51566f 0x549576 0x593fce 0x548ae9 0x5127f1 0x598e3b 0x511f68 0x598e3b 0x511f68 0x598e3b 0x511f68 0x4bc98a 0x532e76 0x594b72 0x515600 0x549576 0x593fce 0x548ae9 0x5127f1 0x549576 0x593fce 0x5118f8 0x593dd7\n",
            "\u001b[K     |███████████████████████▌        | 1336.2 MB 1.2 MB/s eta 0:06:49tcmalloc: large alloc 1792966656 bytes == 0x3980000 @  0x7f6f3d7a8615 0x592b76 0x4df71e 0x59afff 0x515655 0x549576 0x593fce 0x548ae9 0x51566f 0x549576 0x593fce 0x548ae9 0x5127f1 0x598e3b 0x511f68 0x598e3b 0x511f68 0x598e3b 0x511f68 0x4bc98a 0x532e76 0x594b72 0x515600 0x549576 0x593fce 0x548ae9 0x5127f1 0x549576 0x593fce 0x5118f8 0x593dd7\n",
            "\u001b[K     |█████████████████████████████▊  | 1691.1 MB 1.2 MB/s eta 0:01:51tcmalloc: large alloc 2241208320 bytes == 0x6e768000 @  0x7f6f3d7a8615 0x592b76 0x4df71e 0x59afff 0x515655 0x549576 0x593fce 0x548ae9 0x51566f 0x549576 0x593fce 0x548ae9 0x5127f1 0x598e3b 0x511f68 0x598e3b 0x511f68 0x598e3b 0x511f68 0x4bc98a 0x532e76 0x594b72 0x515600 0x549576 0x593fce 0x548ae9 0x5127f1 0x549576 0x593fce 0x5118f8 0x593dd7\n",
            "\u001b[K     |████████████████████████████████| 1821.4 MB 1.3 MB/s eta 0:00:01tcmalloc: large alloc 1821433856 bytes == 0xf40ca000 @  0x7f6f3d7a71e7 0x4a3940 0x4a39cc 0x592b76 0x4df71e 0x59afff 0x515655 0x549576 0x593fce 0x511e2c 0x549576 0x593fce 0x511e2c 0x549576 0x593fce 0x511e2c 0x549576 0x593fce 0x511e2c 0x549576 0x593fce 0x511e2c 0x593dd7 0x511e2c 0x549576 0x593fce 0x548ae9 0x5127f1 0x549576 0x593fce 0x548ae9\n",
            "tcmalloc: large alloc 2276794368 bytes == 0x1609d8000 @  0x7f6f3d7a8615 0x592b76 0x4df71e 0x59afff 0x515655 0x549576 0x593fce 0x511e2c 0x549576 0x593fce 0x511e2c 0x549576 0x593fce 0x511e2c 0x549576 0x593fce 0x511e2c 0x549576 0x593fce 0x511e2c 0x593dd7 0x511e2c 0x549576 0x593fce 0x548ae9 0x5127f1 0x549576 0x593fce 0x548ae9 0x5127f1 0x549576\n",
            "\u001b[K     |████████████████████████████████| 1821.4 MB 2.5 kB/s \n",
            "\u001b[?25hCollecting torchvision==0.11.3+cu113\n",
            "  Downloading https://download.pytorch.org/whl/cu113/torchvision-0.11.3%2Bcu113-cp37-cp37m-linux_x86_64.whl (24.5 MB)\n",
            "\u001b[K     |████████████████████████████████| 24.5 MB 1.3 MB/s \n",
            "\u001b[?25hCollecting torchaudio==0.10.2+cu113\n",
            "  Downloading https://download.pytorch.org/whl/cu113/torchaudio-0.10.2%2Bcu113-cp37-cp37m-linux_x86_64.whl (2.9 MB)\n",
            "\u001b[K     |████████████████████████████████| 2.9 MB 59.0 MB/s \n",
            "\u001b[?25hRequirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch==1.10.2+cu113) (4.1.1)\n",
            "Requirement already satisfied: pillow!=8.3.0,>=5.3.0 in /usr/local/lib/python3.7/dist-packages (from torchvision==0.11.3+cu113) (7.1.2)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torchvision==0.11.3+cu113) (1.21.6)\n",
            "Installing collected packages: torch, torchvision, torchaudio\n",
            "  Attempting uninstall: torch\n",
            "    Found existing installation: torch 1.11.0+cu113\n",
            "    Uninstalling torch-1.11.0+cu113:\n",
            "      Successfully uninstalled torch-1.11.0+cu113\n",
            "  Attempting uninstall: torchvision\n",
            "    Found existing installation: torchvision 0.12.0+cu113\n",
            "    Uninstalling torchvision-0.12.0+cu113:\n",
            "      Successfully uninstalled torchvision-0.12.0+cu113\n",
            "  Attempting uninstall: torchaudio\n",
            "    Found existing installation: torchaudio 0.11.0+cu113\n",
            "    Uninstalling torchaudio-0.11.0+cu113:\n",
            "      Successfully uninstalled torchaudio-0.11.0+cu113\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "torchtext 0.12.0 requires torch==1.11.0, but you have torch 1.10.2+cu113 which is incompatible.\u001b[0m\n",
            "Successfully installed torch-1.10.2+cu113 torchaudio-0.10.2+cu113 torchvision-0.11.3+cu113\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "torch",
                  "torchvision"
                ]
              }
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: transformers in /usr/local/lib/python3.7/dist-packages (4.20.1)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers) (4.11.4)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers) (2.23.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from transformers) (21.3)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.1.0 in /usr/local/lib/python3.7/dist-packages (from transformers) (0.8.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (2022.6.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers) (3.7.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from transformers) (6.0)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.13,>=0.11.1 in /usr/local/lib/python3.7/dist-packages (from transformers) (0.12.1)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers) (1.21.6)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers) (4.64.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.7/dist-packages (from huggingface-hub<1.0,>=0.1.0->transformers) (4.1.1)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->transformers) (3.0.9)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers) (3.8.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2022.6.15)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (3.0.4)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers) (1.25.11)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Imports"
      ],
      "metadata": {
        "id": "nSyd4V2OL_Eo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "from google.colab import drive\n",
        "from torch import nn\n",
        "from torch.optim import Adam\n",
        "from transformers import GPT2Model, GPT2Tokenizer\n",
        "from tqdm import tqdm\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import accuracy_score\n",
        "from sklearn.metrics import classification_report\n",
        "from sklearn.metrics import f1_score\n",
        "from sklearn.metrics import precision_score\n",
        "from sklearn.utils import class_weight\n",
        "from sklearn.metrics import confusion_matrix\n",
        "from sklearn.metrics import ConfusionMatrixDisplay"
      ],
      "metadata": {
        "id": "UQ1Q1W-NxCnr"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Loading Data and EDA"
      ],
      "metadata": {
        "id": "w0koK3CBLy19"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "drive.mount('/content/drive/')"
      ],
      "metadata": {
        "id": "1oGC1B6RIf4k",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "61df27cd-96f8-4340-bf1b-7945c5e70ed9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive/; to attempt to forcibly remount, call drive.mount(\"/content/drive/\", force_remount=True).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df = pd.read_csv(\"/content/drive/MyDrive/dataset/tripadvisor_hotel_reviews.csv\")\n",
        "df.sample(10)"
      ],
      "metadata": {
        "id": "OZUcStmUX3tL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bd1fa9f5-d976-4f94-8876-48a077c96170"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                  Review  Rating\n",
              "4810   london bridge london experience just returned ...       5\n",
              "19993  wonderful time, wonderful stay fusion suites, ...       5\n",
              "8970   class hotel booked short break condes barcelon...       4\n",
              "7740   staff putting ritz property fantastic, linens ...       5\n",
              "10129  excellent hotel stayed hotel twice, took mum 2...       5\n",
              "14884  stylish decor fab location, pulitzer great cen...       5\n",
              "5653   nice room good location great loaction 2 block...       4\n",
              "829    loud times just okay ladies beware bathrooms t...       3\n",
              "12143  standard 4* stuff stayed 4 nights july budget ...       3\n",
              "17080  great time bahia just returned bahia overall t...       4"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-df97497e-43f8-4e3a-899a-66fd1ec6fb28\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Review</th>\n",
              "      <th>Rating</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>4810</th>\n",
              "      <td>london bridge london experience just returned ...</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19993</th>\n",
              "      <td>wonderful time, wonderful stay fusion suites, ...</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8970</th>\n",
              "      <td>class hotel booked short break condes barcelon...</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7740</th>\n",
              "      <td>staff putting ritz property fantastic, linens ...</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10129</th>\n",
              "      <td>excellent hotel stayed hotel twice, took mum 2...</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14884</th>\n",
              "      <td>stylish decor fab location, pulitzer great cen...</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5653</th>\n",
              "      <td>nice room good location great loaction 2 block...</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>829</th>\n",
              "      <td>loud times just okay ladies beware bathrooms t...</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12143</th>\n",
              "      <td>standard 4* stuff stayed 4 nights july budget ...</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17080</th>\n",
              "      <td>great time bahia just returned bahia overall t...</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-df97497e-43f8-4e3a-899a-66fd1ec6fb28')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-df97497e-43f8-4e3a-899a-66fd1ec6fb28 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-df97497e-43f8-4e3a-899a-66fd1ec6fb28');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df[\"Review\"] = df[\"Review\"].apply(lambda x: x.replace(\"* \", \"\"))\n",
        "df[\"Review\"] = df[\"Review\"].apply(lambda x: x.replace(\"__Ç_é_\", \"s\"))\n",
        "df[\"Review\"] = df[\"Review\"].apply(lambda x: x.replace(\"__Ç\", \"s\"))\n",
        "df[\"Review\"] = df[\"Review\"].apply(lambda x: x.replace(\"///\", \" \"))"
      ],
      "metadata": {
        "id": "U9yYBSipY7ZH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9zKTKEkHY7ZK"
      },
      "source": [
        "# Rename 'sentence' column to 'text' (this is the name SimpleTransformers expects).\n",
        "df.columns = ['text', 'label']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Tramsformer assumes label from 0 to Num_labels."
      ],
      "metadata": {
        "id": "n0nWQ3-KY7ZL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Dict={1:0,2:1,3:2,4:3,5:4}\n",
        "# df['label']=df['label'].map(Dict)\n",
        "df['label'] = df['label'].apply(lambda x:x-1)"
      ],
      "metadata": {
        "id": "jisCYQERY7ZM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## EDA"
      ],
      "metadata": {
        "id": "GevTfVV6Y7ZN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check Missing values"
      ],
      "metadata": {
        "id": "TB9vVLm9Y7ZO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.isnull().sum()"
      ],
      "metadata": {
        "id": "2kJUB4HFY7ZP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b4bebc86-ad2d-48fa-da8b-61e67470ac04"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "text     0\n",
              "label    0\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check calss imbalancy"
      ],
      "metadata": {
        "id": "47_Zhd_hY7ZQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.label.value_counts()"
      ],
      "metadata": {
        "id": "pkizT-FgY7ZR",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ca4a6a24-7edf-4f96-e989-46c2ed1e8b21"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4    9054\n",
              "3    6039\n",
              "2    2184\n",
              "1    1793\n",
              "0    1421\n",
              "Name: label, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 74
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.groupby(\"label\").size().plot.bar()"
      ],
      "metadata": {
        "id": "nhzS7XyXY7ZS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2028cbc2-d840-46a7-e493-be336dcc9f82"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7f0b9454f650>"
            ]
          },
          "metadata": {},
          "execution_count": 75
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 864x432 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAt8AAAF+CAYAAABaonYSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3BU5cHH8V9CboSEi2GDKEW5KAFCuKkVdLBKioECCRoIKGiqIF5qIaASBKedqo2FwGAFFaJipIgKBgNFowJexpLRikhEFysIAoOQJbzmAmR3Q/b9g+bQJavJIj67Cd/PjDPNOc959kGfwtfjyUmIx+PxCAAAAMAvLjTQCwAAAADOF8Q3AAAAYAjxDQAAABhCfAMAAACGEN8AAACAIcQ3AAAAYAjxDQAAABgSFugFmPZ//3dMtbW82jwuLkZlZVWBXgaCDPsCvrAvcCb2BHxhX5wSGhqidu1a/ej58y6+a2s9xPd/8fcBvrAv4Av7AmdiT8AX9kXDeOwEAAAAMIT4BgAAAAwhvgEAAABDiG8AAADAEOIbAAAAMIT4BgAAAAwhvgEAAABDiG8AAADAEOIbAAAAMIT4BgAAAAwhvgEAAABDiG8AAADAkLBALwAAAAA/T2zrloqKDHzW2WyxgV6Cqp01qqw4Eehl/KjA/1MCAADAzxIVGaZRMwsDvYygsH5BqioDvYifwGMnAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCGNiu+9e/dq+vTpGjJkiPr166cRI0Zo2bJlcrlcXuM+++wzTZgwQX379tU111yjxx57TCdOnKg3n8vl0vz583XttdcqKSlJ48aNU3Fxsc/PbuycAAAAQLALa2jA4cOHNXbsWMXGxmrixIlq06aNPv30Uy1YsEDffPON5s+fL0my2+3KzMxU9+7dlZ2drUOHDumFF17QgQMH9Oyzz3rNmZ2drXfeeUe33XabLrnkEq1du1ZTpkzRihUr1L9/f2ucP3MCAAAAwa7B+C4sLFRFRYVefvllXXbZZZKkjIwMOZ1Ovfnmm/rrX/+q8PBwLVy4UG3bttWKFSvUqlUrSVKnTp00d+5cFRcXa9CgQZKkkpISbdiwQbNnz1ZmZqYkKS0tTSNHjlRubq5WrlxpfXZj5wQAAACaggYfOzl27JgkKS4uzut4+/btFRYWphYtWqiqqkpbtmxRWlqaFcmSlJqaqujoaL311lvWsaKiIoWHh2vs2LHWscjISKWnp2vr1q0qLS2VJL/mBAAAAJqCBuP7yiuvlCTNmTNHO3fu1Pfff69169ZZj4qEhobq66+/Vk1NjRITE72ujYiIUM+ePWW3261jdrtdXbp08QpqSUpKSpLH47HG+jMnAAAA0BQ0+NjJtddeq2nTpmnp0qXavHmzdfyPf/yj7rvvPkmSw+GQJNlstnrX22w2ff7559bXDodDHTp08DlOknXn2585/REXF3NW1zVHNltsoJeAIMS+gC/sC5yJPYFgFsz7s8H4lk49Z33VVVfpt7/9rdq2bav3339fTz31lC644AJNmDBB1dXVkk7dlT5TZGSkdV6SqqurFR4e7nOcJDmdTmtcY+f0R1lZlWprPWd1bXNis8XK4agM9DIQZNgX8IV9gTOxJ4JPMMdmIARyf4aGhvzkzd4G43vDhg3605/+pKKiIuuO9bBhw+TxeDRv3jyNGDFCUVFRklTv1YPSqZiuOy9JUVFRcrvdPsdJpyPcnzkBAACApqDBZ75ffvll9e7du96jIjfccIOOHz+unTt3Wo+G1D0q8r8cDofi4+Otr202m/VoyZnjJFlj/ZkTAAAAaAoajO8jR47o5MmT9Y7X3b0+efKkLr/8coWFhWnHjh1eY1wul+x2u3r27GkdS0hI0J49e6y3qNTZvn27dV6SX3MCAAAATUGD8d2lSxft2LFD+/bt8zq+YcMGtWjRQj169FBsbKwGDRqkwsJCr6guLCzU8ePHlZKSYh1LSUmR2+3W6tWrrWMul0sFBQUaMGCAdYfdnzkBAACApqDBZ77vvPNOffjhh5owYYJuvfVWtWnTRu+//74+/PBDjR8/3nr/d1ZWlsaPH69JkyZp7NixOnTokJYvX64hQ4Zo8ODB1nx9+/ZVSkqKcnNz5XA41LlzZ61du1YHDx5UTk6O12c3dk4AAACgKQjxeDwNvvqjpKRETz31lOx2u3744QddfPHFuvnmm3XnnXeqRYsW1rhPP/1Uubm5+uqrrxQTE6MRI0ZoxowZio6O9prP6XRq0aJFWr9+vcrLy9WjRw/NmDHDZ1A3ds7G4m0np/Cd6vCFfQFf2Bc4E3si+NhssRo1szDQywgK6xekBvXbThoV380J8X0Kv3HCF/YFfGFf4EzsieBDfJ8W7PHd4DPfAAAAAM4N4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADGl0fJeUlOiuu+7SlVdeqf79+2v06NEqKCjwGrNp0yaNGTNGffr00W9+8xstXrxYNTU19eaqqKjQI488oquvvlr9+vXTbbfdJrvd7vNzGzsnAAAAEOwaFd8ffPCBbrnlFtXU1GjatGmaNWuWBg8erO+//95rzH333ac2bdrokUceUXJyspYsWaKcnByvuWpra3XXXXdpw4YNmjhxoh588EGVlZVp0qRJ2rdvX73PbcycAAAAQFMQ1tCAyspKzZ49W+PHj9fcuXN/dNy8efPUq1cvPf/882rRooUkqVWrVlq2bJkmTZqkSy+9VJJUVFSkbdu2acmSJUpOTpYkDR8+XDfeeKMWL16sefPm+T0nAAAA0BQ0eOd7/fr1qqio0LRp0yRJVVVV8ng8XmN27dqlXbt2KSMjw4pkSbrllltUW1urd955xzr29ttvKz4+XkOHDrWOXXDBBRo+fLg2btwot9vt95wAAABAU9Dgne/i4mJ17dpVH3zwgebPn69Dhw6pdevWysjIUFZWllq0aKGvvvpKkpSYmOh1bYcOHXThhRda5yXJbrerd+/eCgkJ8Rrbp08fvfrqq9q3b5+6devm15wAAJwvYlu3VFRkg398/+JstthAL0HVzhpVVpwI9DIAvzT4/97vvvtOhw4dUnZ2tiZPnqxevXrpvffeU15enpxOp+bMmSOHwyFJstls9a632WwqLS21vnY4HLr66qvrjYuPj5cklZaWqlu3bn7N6Y+4uJizuq45CobfOBF82BfwhX0RXEbNLAz0EoLC+gWpimJvwodg/j2rwfg+fvy4ysvLNXPmTN11112SpGHDhun48eNatWqV7rnnHlVXV0uSIiIi6l0fGRmpEydO/1tpdXW1z3F1x+rm8mdOf5SVVam21tPwwGbOZouVw1EZ6GUgyLAv4Av7IrgEc1QEAnvzFPaFt0Dui9DQkJ+82dvgM99RUVGSpJEjR3odHzVqlNxut7744gtrjMvlqne90+m0ztfN52tc3bG6sf7MCQAAADQFDcZ33WMf7du39zpe93V5ebk1pu5Rkf/lcDisR0rq5vP1yEjdsbqx/swJAAAANAUNxnfv3r0lSYcPH/Y6fujQIUmn3lTSs2dPSdKOHTu8xhw+fFiHDh2yzktSQkKCvvzyy3pvTCkpKVF0dLQ6d+4sSX7NCQAAADQFDcZ3SkqKJGnNmjXWMY/Ho9WrVys6Olr9+vXTZZddpq5du+rVV1/VyZMnrXGrVq1SaGiohg0b5jVfaWmpNm3aZB07evSoioqKNHToUIWHh0uSX3MCAAAATUGD33CZmJiotLQ0LV26VGVlZerVq5c++OADffTRR3rwwQcVE3PqgfKHHnpI99xzj+68806NGDFC//nPf7Ry5UplZGSoS5cu1nw33nij+vXrp4ceekh33HGH2rVrp1WrVqm2tlb333+/12c3dk4AAACgKQjxnPn8hw8ul0tPP/203njjDR05ckSdOnVSZmamxo8f7zVu48aNWrx4sXbv3q0LLrhAN998s+69916FhXk3fnl5uebNm6eNGzfK6XSqT58+ys7Oth5xOZs5G4u3nZzC2wvgC/sCvrAvgovNFsurBv9r/YJU9uZ/sS9OC/S+aOhtJ42K7+aE+D6FP0zhC/sCvrAvgguRdVqgIyuYsC9OC/S++NmvGgQAAABwbhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCHENwAAAGAI8Q0AAAAYQnwDAAAAhhDfAAAAgCFnFd95eXnq0aOHUlNT65377LPPNGHCBPXt21fXXHONHnvsMZ04caLeOJfLpfnz5+vaa69VUlKSxo0bp+LiYp+f19g5AQAAgGDmd3w7HA4988wzio6OrnfObrcrMzNTTqdT2dnZSk9P16uvvqqsrKx6Y7Ozs5Wfn6/Ro0drzpw5Cg0N1ZQpU7Rt27aznhMAAAAIZmH+XrBgwQIlJibK4/GooqLC69zChQvVtm1brVixQq1atZIkderUSXPnzlVxcbEGDRokSSopKdGGDRs0e/ZsZWZmSpLS0tI0cuRI5ebmauXKlX7PCQAAAAQ7v+58l5SUaN26dZo9e3a9c1VVVdqyZYvS0tKsSJak1NRURUdH66233rKOFRUVKTw8XGPHjrWORUZGKj09XVu3blVpaanfcwIAAADBrtHx7fF49OijjyotLU09e/asd/7rr79WTU2NEhMTvY5HRESoZ8+estvt1jG73a4uXbp4BbUkJSUlyePxWGP9mRMAAAAIdo2O7zfeeEO7du3S9OnTfZ53OBySJJvNVu+czWaz7mbXjY2Pj/c5TpI11p85AQAAgGDXqGe+q6qqtGDBAt11110+o1mSqqurJZ26K32myMhI63zd2PDwcJ/jJMnpdPo9Z2PFxcX4fU1zZbPFBnoJCELsC/jCvkCwYm/Cl2DeF42K72eeeUbh4eH6/e9//6NjoqKiJJ16heCZnE6ndb5urNvt9jlOOh3h/szZWGVlVaqt9fh9XXNjs8XK4agM9DIQZNgX8IV9EVyCOSoCgb15CvvCWyD3RWhoyE/e7G0wvktLS5Wfn69p06bpyJEj1nGn0ym3260DBw4oNjbWejSk7lGR/3XmYyY/9shI3bV1Y/2ZEwAAAAh2DT7zXVZWJrfbrdzcXA0dOtT6a/v27dq9e7eGDh2qvLw8XX755QoLC9OOHTu8rne5XLLb7V7fpJmQkKA9e/bo2LFjXmO3b99unZfk15wAAABAsGvwznenTp20ZMmSescXLVqk48eP6+GHH9all16q2NhYDRo0SIWFhZo6dar1JpPCwkIdP35cKSkp1rUpKSl64YUXtHr1aus93y6XSwUFBRowYIA6dOggSX7NCQAAAAS7BuM7NjZWycnJ9Y7n5+erRYsWXueysrI0fvx4TZo0SWPHjtWhQ4e0fPlyDRkyRIMHD7bG9e3bVykpKcrNzZXD4VDnzp21du1aHTx4UDk5OV6f09g5AQAAgGDn94+X/ym9e/fW8uXLFRERoZycHK1evVrjxo3Tk08+WW/svHnzNGnSJBUWFuqxxx5TTU2Nli1bpoEDB571nAAAAEAwC/F4POfVqz9428kpvL0AvrAv4Av7IrjYbLEaNbMw0MsICusXpLI3/4t9cVqg90VDbzs5p3e+AQAAAPw44hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwhPgGAAAADCG+AQAAAEOIbwAAAMAQ4hsAAAAwJCzQCwAA+BbbuqWiIoPjt2mbLTagn1/trFFlxYmArgEAzoUGf1cvKSnR2rVr9fHHH+vgwYNq27at+vfvr+nTp+uSSy7xGvvZZ59p/vz5+uqrrxQTE6Phw4dr5syZatmypdc4l8ulJ598UoWFhaqoqFBCQoKysrI0aNCgep/f2DkBoLmJigzTqJmFgV5GUFi/IFWVgV4EAJwDDT528txzz+ndd9/V4MGDNWfOHI0bN06ffPKJ0tLStHv3bmuc3W5XZmamnE6nsrOzlZ6erldffVVZWVn15szOzlZ+fr5Gjx6tOXPmKDQ0VFOmTNG2bdu8xvkzJwAAABDsGrzznZmZqdzcXEVERFjHRowYoVGjRikvL09PPPGEJGnhwoVq27atVqxYoVatWkmSOnXqpLlz56q4uNi6q11SUqINGzZo9uzZyszMlCSlpaVp5MiRys3N1cqVK63PaeycAAAAQFPQ4J3vAQMGeIW3JF166aW67LLLrDvfVVVV2rJli9LS0qxIlqTU1FRFR0frrbfeso4VFRUpPDxcY8eOtY5FRkYqPT1dW7duVWlpqd9zAgAAAE3BWb3txOPx6MiRI2rXrp0k6euvv1ZNTY0SExO9xkVERKhnz56y2+3WMbvdri5dungFtSQlJSXJ4/FYY/2ZEwAAAGgKzurb6NetW6fDhw9bz147HA5Jks1mqzfWZrPp888/t752OBzq0KGDz3GSrDvf/szpj7i4mLO6rjkK9NsLEJzYFwhW7E34wr6AL8G8L/yO7927d+svf/mLBg4cqNTUVElSdXW1JNV7PEU69UhJ3fm6seHh4T7HSZLT6fR7Tn+UlVWpttZzVtc2JzZbrBwO3h0Ab+yL4BLMf3gEAnvzFPaFN/bFKewLb4HcF6GhIT95s9evx04cDoemTp2qNm3a6Mknn1Ro6KnLo6KiJJ16heCZnE6ndb5urNvt9jlOOh3h/swJAAAANAWNvvNdWVmpKVOmqLKyUqtWrfJ6HKTuf9c9KvK/HA6H4uPjvcbWPVpy5jhJ1lh/5gQAAACagkbd+XY6nbr77ru1d+9eLV26VF27dvU6f/nllyssLEw7duzwOu5yuWS329WzZ0/rWEJCgvbs2aNjx455jd2+fbt13t85AQAAgKagwfg+efKkpk+frs8//1xPPvmk+vXrV29MbGysBg0apMLCQq+oLiws1PHjx5WSkmIdS0lJkdvt1urVq61jLpdLBQUFGjBggPXNmP7MCQAAADQFDT528sQTT2jz5s26/vrr9cMPP6iw8PSPOm7VqpWSk5MlSVlZWRo/frwmTZqksWPH6tChQ1q+fLmGDBmiwYMHW9f07dtXKSkpys3NlcPhUOfOnbV27VodPHhQOTk5Xp/d2DkBAACApqDB+N65c6ck6b333tN7773nde7iiy+24rt3795avny5cnNzlZOTo5iYGI0bN04zZsyoN+e8efO0aNEiFRYWqry8XD169NCyZcs0cOBAr3H+zAk0ZbGtWyoq8qze/HnOBfo75qudNaqsOBHQNQAA8Etp8E/7FStWNHqyK664Qq+88kqD4yIjIzVr1izNmjXrnM0JNGVRkWEaNbOw4YHngfULUsWLwwAAzdVZ/YRLAAAAAP4jvgEAAABDiG8AAADAEOIbAAAAMIT4BgAAAAwhvgEAAABDiG8AAADAEOIbAAAAMIT4BgAAAAwhvgEAAABDiG8AAADAkLBAL+B8E9u6paIig+Nvu80WG9DPr3bWqLLiREDXAAAAYFJwVOB5JCoyTKNmFgZ6GUFh/YJUVQZ6EQAAAAbx2AkAAABgCPENAAAAGEJ8AwAAAIYQ3wAAAIAhxDcAAABgCPENAAAAGEJ8AwAAAIYQ3wAAAIAhxDcAAABgCPENAAAAGEJ8AwAAAIYQ3wAAAIAhxDcAAABgCPENAAAAGEJ8AwAAAIYQ3wAAAIAhxDcAAABgCPENAAAAGEJ8AwAAAIYQ3wAAAIAhxDcAAABgCPENAAAAGEJ8AwAAAIYQ3wAAAIAhxDcAAABgCPENAAAAGEJ8AwAAAIYQ3wAAAIAhxDcAAABgCPENAAAAGEJ8AwAAAIYQ3wAAAIAhxDcAAABgCPENAAAAGEJ8AwAAAIYQ3wAAAIAhxDcAAABgCPENAAAAGEJ8AwAAAIYQ3wAAAIAhxDcAAABgSJOIb5fLpfnz5+vaa69VUlKSxo0bp+Li4kAvCwAAAPBLk4jv7Oxs5efna/To0ZozZ45CQ0M1ZcoUbdu2LdBLAwAAABot6OO7pKREGzZs0AMPPKCHHnpIGRkZys/PV8eOHZWbmxvo5QEAAACNFvTxXVRUpPDwcI0dO9Y6FhkZqfT0dG3dulWlpaUBXB0AAADQeEEf33a7XV26dFGrVq28jiclJcnj8chutwdoZQAAAIB/wgK9gIY4HA516NCh3nGbzSZJft/5Dg0NOSfr+jni27UM9BKCRjD88wgW7IvT2BensS9OY1+cxr44jX1xGvvitEDui4Y+O8Tj8XgMreWsJCcnq3v37nr22We9ju/fv1/Jycl65JFHNHHixACtDgAAAGi8oH/sJCoqSm63u95xp9Mp6dTz3wAAAEBTEPTxbbPZfD5a4nA4JEnx8fGmlwQAAACclaCP74SEBO3Zs0fHjh3zOr59+3brPAAAANAUBH18p6SkyO12a/Xq1dYxl8ulgoICDRgwwOc3YwIAAADBKOjfdtK3b1+lpKQoNzdXDodDnTt31tq1a3Xw4EHl5OQEenkAAABAowX9206kU99cuWjRIq1fv17l5eXq0aOHZsyYocGDBwd6aQAAAECjNYn4BgAAAJqDoH/mGwAAAGguiG8AAADAEOIbAAAAMIT4BgAAwM9WUVGhhQsXavfu3YFeSlAL+lcN4uc7cuSI7Ha7SktLVV1draioKMXHxyshIUE2my3QywMANEEul0tVVVW64IILAr0UBInKykrl5eWpX79+6tatW6CXE7R420kztn37duXm5mrr1q3yeDw68x91SEiIBg4cqAceeED9+vUL0CoRrFauXKkXXnhBmzZtCvRSYMC3336rvLw8ffvtt2rXrp2GDx+u1NTUeuM2btyonJwc9sV5ori4WPn5+aqsrNSgQYM0depUud1uzZ49W++++648Ho86duyoWbNm6cYbbwz0cvELGzNmzE+ed7vd2rVrlzp16qTY2FiFhISooKDA0OqaDu58N1PFxcWaMmWKLrroIk2fPl19+vRRfHy8IiIi5HK5VFpaqu3bt2vt2rWaNGmS8vLydPXVVwd62QgiFRUVOnjwYKCXAQP27dun9PR01dTUqHv37rLb7Xr//fe1Zs0aLVq0SHFxcdbY48ePsy/OE9u2bdPkyZPVrl07tWvXTkuWLFFZWZlcLpf27t2ru+++WydOnNDbb7+trKwsvfTSS7riiisCvWz8gux2u6Kjo9W7d2+f510ulySpZcuWiomJMbm0JoU7381URkaGQkNDlZ+fr4iIiB8d53K5dNttt6m2tlavvfaawRUiEP797383euwbb7yhgoIC2e32X3BFCAYzZszQJ598opUrV+qSSy6RJBUWFurRRx9V27Zt9fzzz1vH161bp1mzZrEvzgOTJ0/W0aNH9fLLLysqKkoLFy7U8uXLNXDgQD333HMKCzt1/66qqkqjRo1Sz5499fTTTwd41fgl5eXl6dlnn1WfPn2UnZ2thIQEr/MHDhxQcnKylixZoqFDhwZolcGPO9/N1M6dOzV37tyfDG9JioiI0E033aTHH3/c0MoQSJMmTVJISEijxno8nkaPRdO2bds2TZw40QpsSUpNTVViYqKmTp2q8ePHa+nSpUpKSgrgKmHazp07NXnyZEVFRUk69cjBsmXLlJqaaoW3JMXExGjMmDF65ZVXArVUGDJlyhSNGTNGubm5Sk9P15gxYzR9+nTrv47xZ0bjEN/NVOvWrbVv375Gjd23b59at279C68IwSA6OkBmzyIAAAZYSURBVFoJCQm64447GhxbVFSkDRs2GFgVAu2HH35Q+/bt6x3v1q2bXnnlFU2ePFm33367/v73vwdgdQiUyspKxcbGWl+3bdtWknThhRfWG9uxY0dVVFQYWxsCp3379nriiSd0yy236LHHHtOwYcM0depUZWZmBnppTQbx3UyNHj1aL774ouLj45Wenq6WLVvWG3PixAmtXr1a+fn5uu222wKwSpiWmJiow4cPKzk5ucGx33zzjYEVIRhcdNFF+vrrr32ea9++vf7xj39o6tSpuueeezRkyBDDq0OgxMXFyeFwWF9HRERo2LBhPv9F7ciRI2rTpo3J5SHAkpKS9Nprr2nt2rVasGCBXnnlFd16663c/W4E4ruZmjZtmr7//ns9/vjjmjdvnrp27SqbzWZ9w6XD4dC3334rt9utlJQUTZs2LdBLhgFJSUl6/vnnVV5e3uAflL7ekIPm6aqrrlJRUZFmzZrl9ThBnZiYGC1fvlzTpk3T5s2b+cP1PNGrVy9t377d+rpVq1Y/+l8/tm3bxqvlzlNjxozRsGHD9PTTT2vRokX8udEIfMNlM1dSUqKioiLt3LlTDofDes+3zWZTQkKCUlJSeI7zPOJwOLRnzx4lJiYqOjo60MtBkPjiiy+Ul5enO+644ydfO1pbW6ucnBzt3LlTK1asMLhCBMKOHTu0b98+jRgx4ifHHT16VFlZWUpNTdVNN91kaHUIRg6HQ2VlZerUqRNvO/kJxDcAAABgCD9eHgAAADCE+AYAAAAMIb4BoBn5+OOP1aNHj7P6kc4HDhxQjx499NRTT53zdRUUFKhHjx76+OOPz/ncANCUEN8AAACAIcQ3AAAAYAjxDQAAABjCD9kBgGastrZWS5cu1UcffaS9e/eqvLxc7du313XXXafp06erXbt2Pq/75z//qaVLl2rv3r2Ki4vTzTffrHvuuafeD+EpLS3VkiVL9MEHH+jIkSNq27atrr/+ek2fPl1xcXEmfokA0KQQ3wDQjLndbj3//PMaNmyYhg4dqpYtW+qLL77Q66+/rs8++0yvv/66IiIivK7ZvHmz9u/fr1tvvVXt27fX5s2btXjxYh08eFA5OTnWuIMHDyojI0Nut1vp6enq3LmzvvvuO61atUoff/yxXn/9dcXGxpr+JQNAUCO+AaAZi4iI0EcffaSoqCjr2IQJE9S/f3/NnTtXGzdurPcTDHfu3Kk1a9aod+/ekqSJEyfqD3/4gwoKCpSRkWH9FMxHH31UNTU1euONN3ThhRda16ekpCgjI0Mvvvii7r//fgO/SgBoOnjmGwCasZCQECu8T548qYqKCh09elRXX321JKmkpKTeNYMHD7bCu26OyZMnS5LeffddSVJlZaXef/993XDDDYqIiNDRo0etvy6++GJ17txZ//rXv37pXx4ANDnc+QaAZu7NN9/U8uXLZbfb5Xa7vc6Vl5fXG9+tW7d6x7p37y5J2r9/vyRpz549qq2t1Zo1a7RmzRqfn/urX/3q5y4dAJod4hsAmrF33nlHWVlZSkpK0sMPP6yOHTsqMjJSJ0+e1OTJk+XxeM5q3rrrRo8erTFjxvgcExkZedbrBoDmivgGgGassLBQkZGReumll9SyZUvr+O7du3/0Gl/ndu3aJen03ezOnTsrJCREbrdbgwcPPserBoDmi2e+AaAZa9GihUJCQlRbW2sd83g8euaZZ370mi1btujLL7/0Gv/cc89JkpKTkyVJ7dq103XXXad3331Xn3/+eb05PB6Pjh49eq5+GQDQbHDnGwCasRtvvFFvv/22br/9dqWlpammpkYbN27UiRMnfvSahIQE3X777br11ltls9m0adMmbdmyRampqerfv7817s9//rNuueUWTZw4UampqerVq5dqa2u1f/9+bdq0SWlpabztBADOQHwDQDP2u9/9TseOHdOLL76ov/3tb2rTpo2uv/56zZw5U7/+9a99XnPDDTeoS5cuWrp0qfbs2aO4uDjde++9uvfee73GdezYUa+//rry8vK0efNmrVu3TpGRkerYsaOuv/56DR8+3MQvEQCalBDP2X63DQAAAAC/8Mw3AAAAYAjxDQAAABhCfAMAAACGEN8AAACAIcQ3AAAAYAjxDQAAABhCfAMAAACGEN8AAACAIcQ3AAAAYAjxDQAAABjy/xNtL7nl5iYtAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Data is imbalance, let's calculate the class_weights"
      ],
      "metadata": {
        "id": "38u8xzjHY7ZT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.utils.class_weight import compute_class_weight\n",
        "class_weights = compute_class_weight(class_weight = \"balanced\", classes= np.unique(df.label), y= df.label)\n",
        "class_weights"
      ],
      "metadata": {
        "id": "hTBlxYP-Y7ZU",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "108c9d37-2c87-4378-d7ad-de73687e50e8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([2.88402533, 2.28566648, 1.8764652 , 0.67862229, 0.45263972])"
            ]
          },
          "metadata": {},
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Dataset class"
      ],
      "metadata": {
        "id": "DTXvFqcOLqMr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer = GPT2Tokenizer.from_pretrained('gpt2')\n",
        "tokenizer.padding_side = \"left\"\n",
        "tokenizer.pad_token = tokenizer.eos_token\n",
        "\n",
        "\n",
        "class Dataset(torch.utils.data.Dataset):\n",
        "    def __init__(self, df):\n",
        "        self.labels = [label for label in df['label']]\n",
        "        self.texts = [tokenizer(text,\n",
        "                                padding='max_length',\n",
        "                                max_length=128,\n",
        "                                truncation=True,\n",
        "                                return_tensors=\"pt\") for text in df['text']]\n",
        "        \n",
        "    def classes(self):\n",
        "        return self.labels\n",
        "    \n",
        "    def __len__(self):\n",
        "        return len(self.labels)\n",
        "    \n",
        "    def get_batch_labels(self, idx):\n",
        "        # Get a batch of labels\n",
        "        return np.array(self.labels[idx])\n",
        "    \n",
        "    def get_batch_texts(self, idx):\n",
        "        # Get a batch of inputs\n",
        "        return self.texts[idx]\n",
        "    \n",
        "    def __getitem__(self, idx):\n",
        "        batch_texts = self.get_batch_texts(idx)\n",
        "        batch_y = self.get_batch_labels(idx)\n",
        "        return batch_texts, batch_y"
      ],
      "metadata": {
        "id": "LnO3svgrKFmj",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 113,
          "referenced_widgets": [
            "9e2e5b1d11594bc4b4b11c33b37280bf",
            "41df0c31e06f429db1c12abb601dd11b",
            "5b928194399f432aacec2f87cb1818ac",
            "b141a05434c04108bdd1a4c50657d214",
            "7678bc8db66d4baa852e750662680c5d",
            "1a9cd89607d84c819644165de1b16d5f",
            "87020c37c3d34c1b8102f309abc704be",
            "6f692b85b0f64942aadb85a805e76654",
            "239cfb006917429ba0d4ad358bf32269",
            "477ce14493fa4a7f8965913c194531f4",
            "1b5be88e200d4ea49aa9bb7c93aee2e7",
            "bad57053044b4366a06e8bee0c4793d3",
            "d73d7688b1244a3280308e2e8a047e60",
            "231fa6010d1445e0a98cbf3c2ee2f131",
            "536eaaf0aca84c19927c15f1ef07c183",
            "2705475481f945cbb9ad17afa1b9faa6",
            "346cb874f010436f990f2d1c462a5464",
            "84699c43650a4f749ed34acdc7187e83",
            "055c7460d6cc415eb6a3d79f91b6cf4d",
            "2a2dabe3b56642d383b17e2f9bd2a635",
            "fd275c5099ed405fb3ed649496dce615",
            "2dfbaf2036354878940814161fabe2f2",
            "cd064d23a39e435e810ff6cde9610c4c",
            "9fa38c5f23484104a7f89f91ee2b85c6",
            "f8d42feaa90547c9bf9e0fad71315515",
            "dabfa02dc3f64352ba4912ae06ae6c69",
            "bc6622e515db48dab5195e877ee7f671",
            "7959f779db344a089f1adf4827900315",
            "576516d057ee479d9b819ba1776197aa",
            "2b16d86eafbc4cd09659ffa058145941",
            "87209f2bc0454fbe9ce5b70147d699aa",
            "819a3dda4711485487e2f66e8475ae25",
            "28d1e7741f6d4f22831857c220edd7cf"
          ]
        },
        "outputId": "36e8e73e-ba05-4f74-b101-7e424c890f99"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/0.99M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "9e2e5b1d11594bc4b4b11c33b37280bf"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/446k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "bad57053044b4366a06e8bee0c4793d3"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/665 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "cd064d23a39e435e810ff6cde9610c4c"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Split training-test dataset\n",
        "\n"
      ],
      "metadata": {
        "id": "hb1YZcAPLimH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "np.random.seed(112)\n",
        "df_train, df_val, df_test = np.split(df.sample(frac=1, random_state=35),\n",
        "                                     [int(0.8*len(df)), int(0.9*len(df))])\n",
        "\n",
        "print(len(df_train), len(df_val), len(df_test))"
      ],
      "metadata": {
        "id": "ztSYe9ZuKJIj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6d4bf9f5-cc2a-423f-9595-7e73f491105a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "16392 2049 2050\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Model building\n",
        "\n",
        "\n",
        "Now comes to the most important part of this notebook. We need to buid a classifier model on top of a pre-trained GPT-2 model. The trick here is to add a linear layer on top of GPT-2's 12 layers of decoders with its output dimension equals our number of labels. In this way we can use GPT-2 to output 5 numbers which corresponds to our five news categories!\n",
        "\n",
        "For more information about how to build customized models in PyTorch, see [here](https://pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html)"
      ],
      "metadata": {
        "id": "-SqL7dLuLaqN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class SimpleGPT2SequenceClassifier(nn.Module):\n",
        "    def __init__(self, hidden_size: int, num_classes:int ,max_seq_len:int, gpt_model_name:str):\n",
        "        super(SimpleGPT2SequenceClassifier,self).__init__()\n",
        "        self.gpt2model = GPT2Model.from_pretrained(gpt_model_name)\n",
        "        self.fc1 = nn.Linear(hidden_size*max_seq_len, num_classes)\n",
        "\n",
        "        \n",
        "    def forward(self, input_id, mask):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "                input_id: encoded inputs ids of sent.\n",
        "        \"\"\"\n",
        "        gpt_out, _ = self.gpt2model(input_ids=input_id, attention_mask=mask, return_dict=False)\n",
        "        batch_size = gpt_out.shape[0]\n",
        "        #print(\"batch_size \",batch_size) to check the batch size\n",
        "        linear_output = self.fc1(gpt_out.view(batch_size,-1))\n",
        "        return linear_output"
      ],
      "metadata": {
        "id": "8FNQdXcQKonq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training loop\n",
        "\n",
        "\n",
        "Now it's time to train (fine-tune) our model! Here I build a standard PyTorch training loop following [this guide](https://pytorch.org/tutorials/beginner/basics/optimization_tutorial.html). Since this is a multi-class classification problem, I picked [cross-entropy-loss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) as our loss function (\"criterion\"), and [Adam](https://pytorch.org/docs/stable/generated/torch.optim.Adam.html) as the optimization algorithm."
      ],
      "metadata": {
        "id": "Z3XAE2JGLR3e"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def train(model, train_data, val_data, learning_rate, epochs):\n",
        "    train, val = Dataset(train_data), Dataset(val_data)\n",
        "    \n",
        "    train_dataloader = torch.utils.data.DataLoader(train, batch_size=2, shuffle=True)\n",
        "    val_dataloader = torch.utils.data.DataLoader(val, batch_size=2)\n",
        "    \n",
        "    use_cuda = torch.cuda.is_available()\n",
        "    device = torch.device(\"cuda\" if use_cuda else \"cpu\")\n",
        "    \n",
        "    criterion = nn.CrossEntropyLoss()\n",
        "    optimizer = Adam(model.parameters(), lr=learning_rate)\n",
        "    \n",
        "    if use_cuda:\n",
        "        model = model.cuda()\n",
        "        criterion = criterion.cuda()\n",
        "\n",
        "    for epoch_num in range(epochs):\n",
        "        total_acc_train = 0\n",
        "        total_loss_train = 0\n",
        "        \n",
        "        for train_input, train_label in tqdm(train_dataloader):\n",
        "            train_label = train_label.to(device)\n",
        "            mask = train_input['attention_mask'].to(device)\n",
        "            input_id = train_input[\"input_ids\"].squeeze(1).to(device)\n",
        "            \n",
        "            model.zero_grad()\n",
        "\n",
        "            output = model(input_id, mask)\n",
        "            \n",
        "            batch_loss = criterion(output, train_label)\n",
        "            total_loss_train += batch_loss.item()\n",
        "            \n",
        "            acc = (output.argmax(dim=1)==train_label).sum().item()\n",
        "            total_acc_train += acc\n",
        "\n",
        "            batch_loss.backward()\n",
        "            optimizer.step()\n",
        "            \n",
        "        total_acc_val = 0\n",
        "        total_loss_val = 0\n",
        "        \n",
        "        with torch.no_grad():\n",
        "            \n",
        "            for val_input, val_label in val_dataloader:\n",
        "                val_label = val_label.to(device)\n",
        "                mask = val_input['attention_mask'].to(device)\n",
        "                input_id = val_input['input_ids'].squeeze(1).to(device)\n",
        "                \n",
        "                output = model(input_id, mask)\n",
        "                \n",
        "                batch_loss = criterion(output, val_label)\n",
        "                total_loss_val += batch_loss.item()\n",
        "                \n",
        "                acc = (output.argmax(dim=1)==val_label).sum().item()\n",
        "                total_acc_val += acc\n",
        "                \n",
        "            print(\n",
        "            f\"Epochs: {epoch_num + 1} | Train Loss: {total_loss_train/len(train_data): .3f} \\\n",
        "            | Train Accuracy: {total_acc_train / len(train_data): .3f} \\\n",
        "            | Val Loss: {total_loss_val / len(val_data): .3f} \\\n",
        "            | Val Accuracy: {total_acc_val / len(val_data): .3f}\")\n",
        "            \n",
        "EPOCHS = 1\n",
        "model = SimpleGPT2SequenceClassifier(hidden_size=768, num_classes=5, max_seq_len=128, gpt_model_name=\"gpt2\")\n",
        "LR = 1e-5\n",
        "\n",
        "train(model, df_train, df_val, LR, EPOCHS)"
      ],
      "metadata": {
        "id": "mlF30ROhLKsN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Evaluation"
      ],
      "metadata": {
        "id": "doKtdR5qbiq0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def evaluate(model, test_data):\n",
        "\n",
        "    test = Dataset(test_data)\n",
        "\n",
        "    test_dataloader = torch.utils.data.DataLoader(test, batch_size=2)\n",
        "\n",
        "    use_cuda = torch.cuda.is_available()\n",
        "    device = torch.device(\"cuda\" if use_cuda else \"cpu\")\n",
        "\n",
        "    if use_cuda:\n",
        "\n",
        "        model = model.cuda()\n",
        "\n",
        "        \n",
        "    # Tracking variables\n",
        "    predictions_labels = []\n",
        "    true_labels = []\n",
        "    \n",
        "    total_acc_test = 0\n",
        "    with torch.no_grad():\n",
        "\n",
        "        for test_input, test_label in test_dataloader:\n",
        "\n",
        "            test_label = test_label.to(device)\n",
        "            mask = test_input['attention_mask'].to(device)\n",
        "            input_id = test_input['input_ids'].squeeze(1).to(device)\n",
        "\n",
        "            output = model(input_id, mask)\n",
        "\n",
        "            acc = (output.argmax(dim=1) == test_label).sum().item()\n",
        "            total_acc_test += acc\n",
        "            \n",
        "            # add original labels\n",
        "            true_labels += test_label.cpu().numpy().flatten().tolist()\n",
        "            # get predicitons to list\n",
        "            predictions_labels += output.argmax(dim=1).cpu().numpy().flatten().tolist()\n",
        "    \n",
        "    print(f'Test Accuracy: {total_acc_test / len(test_data): .3f}')\n",
        "    return true_labels, predictions_labels\n",
        "    \n",
        "true_labels, pred_labels = evaluate(model, df_val)"
      ],
      "metadata": {
        "id": "_E7GRAIMbj0X",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1564795b-c1a8-4b89-c9ff-77aa4d484167"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Test Accuracy:  0.645\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(classification_report(true_labels,pred_labels))"
      ],
      "metadata": {
        "id": "-QnXRbhki7bn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a243b1b2-adcf-4e3e-cd36-f0de112f63b3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.74      0.57      0.64       157\n",
            "           1       0.40      0.62      0.49       141\n",
            "           2       0.57      0.16      0.25       217\n",
            "           3       0.53      0.67      0.59       609\n",
            "           4       0.80      0.76      0.78       925\n",
            "\n",
            "    accuracy                           0.64      2049\n",
            "   macro avg       0.61      0.56      0.55      2049\n",
            "weighted avg       0.66      0.64      0.64      2049\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "true_labels, pred_labels = evaluate(model, df_test)\n",
        "print(classification_report(true_labels,pred_labels))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6h5NVT85qUg6",
        "outputId": "bcdeb489-42dc-40a2-ed8a-78642528383f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Test Accuracy:  0.640\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.66      0.64      0.65       157\n",
            "           1       0.46      0.59      0.51       193\n",
            "           2       0.59      0.18      0.27       226\n",
            "           3       0.54      0.67      0.60       610\n",
            "           4       0.79      0.75      0.77       864\n",
            "\n",
            "    accuracy                           0.64      2050\n",
            "   macro avg       0.61      0.57      0.56      2050\n",
            "weighted avg       0.65      0.64      0.63      2050\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(f1_score(true_labels,pred_labels,average='micro'))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_cQDjvVDq9Kc",
        "outputId": "bf880209-b837-4ac4-d52e-6bf0492e3ab8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.6404878048780488\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Plot confusion matrix.\n",
        "fig, ax = plt.subplots(figsize=(8, 8))\n",
        "cm = confusion_matrix(y_true=true_labels, y_pred=pred_labels, labels=range(len(df.label.unique())))#, normalize='true')\n",
        "disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=list(df.label.unique()))\n",
        "disp.plot(ax=ax)"
      ],
      "metadata": {
        "id": "vl4XiAC5bkUz",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 494
        },
        "outputId": "d7c96834-ebd4-4618-f61b-b5fe7d8ee8d6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<sklearn.metrics._plot.confusion_matrix.ConfusionMatrixDisplay at 0x7f0afaa68e90>"
            ]
          },
          "metadata": {},
          "execution_count": 86
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 576x576 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfIAAAHMCAYAAAAu4SoNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xT5f4H8E/Ske49KJRCyyhQKFNkyZVdkFFk7woCKiBDmYL6UxGEykUvgoKyCgKiQBFkg8p0sApYyobuvUeSJuf3R20kpqUppD097ed9X3ld+5znPPkmpP3mGec5MkEQBBAREZEkycUOgIiIiJ4eEzkREZGEMZETERFJGBM5ERGRhDGRExERSRgTORERkYQxkRMRkWQI2gyxQ6hyZLyOnIiIpESbOgrQJpimMXktyF13mKYtkZiLHQAREVF5aLVxgCbWVK1Jfmha6vETERHVaOyRExGRpGgELSBoTdOYoJV8ImSPnIiISMKk/kWEiIhqGOHv/5mCzETtiImJnIiIJEULAQJMM7ReHRI5h9aJiIgkjD1yIiKSFI0gwFRboMiqwVYq7JETERFJGHvkREQkKUUz5KbpScurwRw5EzkREUmKxoSJ3FSr38XEoXUiIiIJY4+ciIgkRWvCHjnYIyciIiIxsUdORESSohUEaEx12Vg1uPyMiZyIiCRF+/fDFGQmakdMHFonIiKSMPbIiYhIUjQQoOFiNx32yImIiCSMPXIiIpIUrQBoTNSRlkm/Q84eORERkZSxR05ERJJiylXrpmpHTEzkREQkKRrIoDHRhWOyanABGofWiYiIJIw9ciIikhStUPQwVVtSxx45ERGRhDGRExGRpGj/niM3xUP7lHPkERERmDJlCp577jm0bt0aAwcOxJ49e/TqnDhxAoMHD0aLFi3w4osvYs2aNSgsLDRoKysrC0uWLEGHDh3QqlUrjB8/HpGRkUbHwqF1IiKSFFMudpM/RTu//PILpk2bhvbt22PmzJkwNzfHgwcPEB8fb1CnQ4cOWLJkCW7duoUvvvgC6enpWLJkia6eVqvFlClTcOvWLUycOBHOzs749ttvMW7cOOzZswc+Pj5lxsNETkREZKTs7GwsXLgQI0eOxOLFi0utt2LFCjRr1gzffPMNzMzMAAC2trZYv349xo0bh/r16wMADh8+jMuXL+OLL75Az549AQB9+/ZFnz59sGbNGqxYsaLMmDi0TkREkqIVZCZ9lMePP/6IrKwszJw5EwCQk5MD4V+3Qr1z5w7u3LmDESNG6JI4AIwePRparRZHjx7VlR05cgQeHh7o0aOHrszFxQV9+/bF8ePHoVary4yJiZyIiMhI58+fh5+fH3755Rf85z//Qdu2bdG+fXuEhoZCo9EAAP766y8AQPPmzfXO9fT0RK1atXTHASAyMhIBAQGQyfS/ULRo0QK5ubl49OhRmTFxaJ2IiCRFa8I5crO/24mPj9cl4mIODg5wcHDQK3v48CESEhKwYMECvPrqq2jWrBlOnTqFDRs2QKlU4p133kFycjIAwN3d3eD53N3dkZSUpPs5OTkZHTp0MKjn4eEBAEhKSkKDBg2e+BqYyImISFI0kENjogHl4nbGjBmD2NhYvWPTp0/HjBkz9Mry8vKQmZmJt956C1OmTAEA9O7dG3l5edixYwdef/11FBQUAAAsLS0Nnk+hUCA/P1/3c0FBQYn1isuK23oSJnIiIqrxtm/fXmKP/N+srKwAAP3799crHzBgAA4fPoxr167p6qhUKoPzlUql7nhxeyXVKy57vG5pmMiJiEhSinZ2M83QevHObl5eXkbVd3d3x+3bt+Hm5qZXXvxzZmambkg9OTlZN0ReLDk5Ga1bt9Zr7/Gh9mLFZf8+vyRc7EZERGSkgIAAAEBiYqJeeUJCAoCiFedNmzYFAFy/fl2vTmJiIhISEnTHAaBJkya4ceOGwcr3iIgI2NjYGHUdORM5ERFJipg7uwUFBQEAvv/+e12ZIAjYvXs3bGxs0KpVKzRq1Ah+fn7YtWuX3nD9jh07IJfL0bt3b732kpKScOLECV1ZWloaDh8+jB49esDCwqLMmDi0TkREkqKBHBrBtIvdjNW8eXMEBwfjq6++QmpqKpo1a4ZffvkFZ86cwdy5c2FnZwcAmDdvHl5//XVMmjQJ/fr1w61bt7B9+3aMGDECvr6+uvb69OmDVq1aYd68ebqd3Xbs2AGtVmuw0K40MuHf/XmJmj15I1KSssQOAwAQFj4L4watFjsMAIA8pWq8J8W2nH0XEzp/IHYYOpq4BLFD0LPt/lqM9X1D7DCqJL43T1ZV3h+3Oi5YfeajCn2O32P+A2VhbNkVjaAwr4P23r+U6xyVSoW1a9di3759SElJgbe3N0JCQjBy5Ei9esePH8eaNWtw9+5duLi4YMiQIXjjjTdgbq7fh87MzMSKFStw/PhxKJVKtGjRAgsWLNAN45el2vTIU5KykBifKXYYOlUlFrPEDLFDMJAUky52CDqF0clih2Ag8WHVi6mq4HvzZDXl/dFCDq2JZoafph1LS0vMmjULs2bNemK9nj176rZdfRJHR0csXboUS5cuLXcsAOfIiYiIJK3a9MiJiKhmMOXObk97G9OqhD1yIiIiCWOPnIiIJEUjyEy3at1EG8uIiYmciIgkRfsU138/qS2p49A6ERGRhLFHTkREkqI14d3PTHUZm5ik/wqIiIhqMPbIiYhIUjSCCbdoNVE7YmIiJyIiSRF7Z7eqRvqvgIiIqAZjj5yIiCRFK5ju+m9tNbhtGHvkREREEsYeORERSYrGhJefmaodMTGRExGRpGgFObQmWm1uqnbEJP1XQEREVIOxR05ERJLCnd30Sf8VEBER1WDskRMRkaQU3cbUNJefVYfbmLJHTkREJGHskRMRkaQU3Y/cVHPk0u+RM5ETEZGkaE140xRefkZERESiYo+ciIgkpWho3UR7rVeDoXX2yImIiCSMPfJSjBh6HQ0bpKNhgzR41cpBYqItJkwOLrW+f+MUhIy9Cv/GKdAm7sNH7ztg45ZWuHffRa+ed50sBPW+g4YN0tDQLw12dmps29EC23YEVvRLqnAHfztYYnl+nhmGdgsq9bx+Lz/EtPnXAQCjevdCVqZlhcRXFclkAoJfTcFL41Lh6a2CNqkrpryrwZaVnlDmm4kdnqjq+CnRY0g62nTNhld9JbSJrbH2mAanf3TCng1uNf79qcmfHY0J58hN1Y6YpP8KKsgr46+iZYsExCfYITv7yYmliX8KVn58DJ6eOQj7NhAyuzdR2ysbocuOoX69dL26Tf2T8fKgSLi75eH2XZdSWpSu65ddEPpeS73HZ0tL/5Li4laAkGk3kZdbvf/wlGbq/8Xhtf+Lw8NbVli7uA5gFYRBk5LxwZYHkMmqwf0Vn0GfkakYPDkZ8Q8tsf2/npDZz0PMXQVCFiRg9f47sLTSih2iqGryZ0cLme7GKc/64ND6M7h27RqmTZuGbt26ITAwEJ07d8akSZNw6dIlsULSEzJ5IIaPHYZF7/ZAapr1E+u+PvlPqAvlmLuwF/bubwqZ7SuYu7AXBACTJ+q/nvO/e2PYmGGY9NpAbNzSugJfgTgSYm1w6rC33uP08dql1n997nXEx9jgwq+1KjHKqqFe4wIMmpiCMwcd8eGr9XHoW1fIHRbhq/+rjVZdcvBicIbYIYrq9AEnjGnbDJ9Mr4f9G90hsxmFj1+rj29Xe8AvoABBo9LEDlE0/OzQ40RL5NHR0dBoNBg2bBiWLFmCSZMmIS0tDWPHjsXZs2fFCksnIdHeqHpeXtnwb5yKM2frITXNRleemmaDM2froXXLBDg75evKc3IUyM2t3kPH5uZaWFkXllmv438S8PwLifjikxbQaqT/rbi8XgxOh1wO7N3gpld+aLsrCvLk6P5yeiln1gy3I2yQl204UvPLficAQH3/gsoOqcqo6Z8dLWTQCiZ6VIMeuWhz5P369UO/fv30ykaNGoWePXti69at6Ny5s0iRlY9/o1QAQORNN4NjkVFu6NPrLho1TMPvf9ap7NBE0bl7PLoFxcLMXEBGmiVOH/fC1i/9kZdroVfP2laN196+jsN7fXDrLye8NOShSBGLp3GrPGg0QNQVG71ytVKOuzes0LhVnkiRVW1uXmoAQHpKzV3iw88OPa5K/SZYW1vDxcUFWVlZYodiNBeXot52Sqrh8Hvq32WurjXjlyrquhPOnKyFuGhb2NgWol3nJAwY/hDNW6fh7cmdUJD/z8ftlWk3IZcDm9c2ETFicbl6FiIrzRxqleHAWGq8BQKey4O5hRaFai5lKSaXCxgzOxGFauDUXiexwxFNTf/saEx49zNTtSMm0RN5Tk4OVCoVMjIysG/fPty6dQvTpk0TOyyjWSmKhpDVhYZDgCp1UZlCoanUmMQyZ5L+KMrJQ954cPsOJrwRhUEj7mPX5kYAgKaBaeg7+BFC32tl0FOvSRTWWqhVJQ/rqZTyv+sIKFRXZlRV22sfxKJZuzxs/LgWYu5aiR2OaGr6Z0cQ5CbbkU3gqvVnt2jRInTs2BF9+/bFxo0bMXLkSLz22mtih2W0AmXRdyELc8NkbWlRVKZU1swV2QDwwzY/qFVyPNc5CQAgCCrMWHgNV/5wwy9Ha8Z0Q2mU+XJYWJa8uthSof27jvTn70xFm70agyam4mCYC3at8RQ7HFHxs0OPkwmCIOp1ClFRUUhJSUFCQgLCw8NRp04dLF68GLa2tmKGpUeb8hKgzYPc45TBMSH/AITMOZA5fASZzXD9Y3m7IGQtgczpK8isuhmeq74GIXUIYDsdcvs3Kyx+sWmTuwMwh9z9KITcTRCyP4HMeT1gVk9XR8heCSiPQub6PSBzgszcR7yAK4k2bSKgOgeZZwRkMv0FkNrUkUDhfcg9fxMpuqpFm/05kLsGsB4CmcPHkMlqdpKq6Z+d9XfGI0udaJK2HCw8MaXhVpO0JRbRE/nj1Go1hgwZgvr16+Pzzz8v17njBq1GYnxmhcT15f8OwNqqsMQNYby8srHpq/04cqwB/vu/DgCAo7+/h97t/w+zZ1xArx53MSbkZaRnGM6hN2qYiv+tOlyhG8KYJYp7GYqFpQbfnzqCm9edMH9qJxy85gbkbXniOWVtIGNKhdExlfI8JZkwLx6jZyXhreAGuP67HQDgmHY3+lkPwfc3buDaBVssHusnWnxVxdi3EjDurUTAajCCXO9CqAb3j35WVfmz41nPHdvur63Q52Ai1yf6HPnjLCws0KNHD6xbtw4FBQWwsqr6c2Dx8fa4ddsFXTo/xJbtgUj7+xI0F5c8dOn8EFcjapWYxKsbewcVsrMML6sbN/UWzM0F/H66aChUZv0yls68ZlCv/9AHCGybhv9+GIicrJoxb/7LfieMfDMJgyen6P4YA0DfMamwstHi5B5nEaOrGsbMLkrix3c7o9f0ZRCEEWKHVCXU9M9O0aVjprr7mfS/GFapRA4ABQUFEAQBubm5oibyHi/eg4dHLgDA0VEJc3MtRg0vSkBJSbY48fM/33bXbWiHT5YeR+iyY9h/0B9C7maELjsGuQxYv7GNXrs2NioM6h8F4J8V780DknRtX/jdG/cfSO+XcOTEO/Bvno6Ii65ITrCGtU0h2nVKRst2qbh53Qk/7q4PAJBZNMXZk14G57fvnAQgDb+f9qwxW7Q+uGmNHze7YtDEVCz5+gH+OGkPbdYyTH0vDlfP2dboVdkAMCAkBePnJiIxxgKXT9uh16s/6l0fnZFijku/GrffQ3XDzw49TrREnpaWBhcX/S1Kc3JycOTIEXh5ecHV1VWkyIr06XUXgS2S9MomjI0AAERc89BL5JE33TF/UU+MH3sVE8ZchZBzE/HxDvj4kxcMkrK9nUrXTrFWgYloFVg0TJSSYiPJRH7tkgvq+majR78YODiqodXKEBdtiy1r/bF3hy/Uqpq74O9Jvny3DhKjLdF3bBra98gCCg4ifKMbtq6sVeOHkBu3LLps09NbjbmfR0PInIv5a/45fvWcbY1N5EDN/uwUbdHKu58VE22OfPz48VAoFGjdujXc3d0RHx+PPXv2ICEhAatWrTLYLKYsFTlHXl7Fc+RVgdhz5P926OF/0bfebLHD0BFzjrwkx7S70Us+TOwwqiS+N09WVd6fypgjX3NrIjLVSWVXNIKjhQemN95okrbEIlqPfODAgQgPD0dYWBiysrJgb2+PVq1aYcWKFWjfvr1YYREREUmKaIl86NChGDp0qFhPT0REEsXbmOqT/isgIiKqwarcqnUiIqInEWC6u5YJ1WCxGxM5ERFJikaQmXBoXfqJnEPrREREEsYeORERSUrRzm4muo6cPXIiIiISE3vkREQkKVrIoTFRP1RbDfqzTORERCQpHFrXJ/2vIkRERDUYe+RERCQpWshNNiReHYbWpf8KiIiIajD2yImISFI0kJlsIxdT3Q5VTOyRExERGem3336Dv79/iY+7d+/q1b106RJGjRqFli1bonPnzvjoo4+Qn59v0KZKpcLKlSvRpUsXBAYGYvjw4Th//rzRMbFHTkREkiIIplttLghPd96ECRMQEBCgV+bp6an778jISISEhKBhw4ZYsGABEhISsHHjRsTExODLL7/UO2/BggU4evQoxo8fj3r16mHv3r2YPHkywsLC0Lp16zJjYSInIiJJ0QpyaE201/rTttO+fXv07Nmz1OOrVq2Ck5MTwsLCYGtrCwDw9vbG4sWLcf78eXTs2BEAEBERgYMHD2LhwoUICQkBAAQHB6N///4IDQ3F9u3by4yFQ+tERERPIScnB4WFhSWWnzt3DsHBwbokDgCDBg2CjY0NDh06pCs7fPgwLCwsMGzYMF2ZQqHA0KFDcfHiRSQlJZUZB3vkREQkKRrITLZI7WnbmTt3LvLy8mBubo7nn38e8+fPh7+/PwAgKioKhYWFaN68ud45lpaWaNq0KSIjI3VlkZGR8PX11Uv4ABAYGAhBEBAZGQkPD48nxsJETkRENV58fDw0Go1emYODAxwcHPTKLCws0KdPH3Tt2hXOzs6IiorCxo0bMXr0aHz//ffw9fVFcnIyAMDd3d3gedzd3XHlyhXdz8nJyXpz64/XA8AeORERVT+CCbdoFf5uZ8yYMYiNjdU7Nn36dMyYMUOvrE2bNmjTpo3u5x49eqB79+4YMmQI1qxZg08//RQFBQUAinrg/6ZQKHTHAaCgoAAWFhYl1gMApVJZ5mtgIiciIkmpiMVu27dvL7FHbowmTZqgY8eOuHDhAgDAysoKQNFlZf+mVCp1x4vrqtXqEusB/yT0J2EiJyKiGs/Ly+uZzy9O5MXD4sVD7I9LTk7Wm/N2d3cvcfi8+Nyy5scBrlonIiKJ0UJm0ocpREdHw9nZGQDQuHFjmJub4/r163p1VCoVIiMj0bRpU11ZkyZNcP/+feTm5urVvXr1qu54WZjIiYiIjJSWlmZQ9ueff+K3335Dly5dAAD29vbo2LEjwsPD9RJ0eHg48vLyEBQUpCsLCgqCWq3G7t27dWUqlQp79uxBmzZtSlwI928cWiciIknRCDDdXuvl3Nlt1qxZsLa2RuvWreHs7Izbt29j165dcHZ21lsYN3v2bIwcORLjxo3DsGHDkJCQgE2bNqFr167o1KmTrl7Lli0RFBSE0NBQJCcnw8fHB3v37kVcXByWLVtmVExM5EREJCmCCRe7CeVsp2fPnvjxxx+xadMm5OTkwMXFBf3798eMGTNQu3ZtXb2AgABs2rQJoaGhWLZsGezs7DB8+HDMmTPHoM0VK1Zg9erVCA8PR2ZmJvz9/bF+/Xq0bdvWqJiYyImIiIw0fvx4jB8/3qi67dq1w86dO8usp1AoMH/+fMyfP/+pYqo2iVwelwp5jOHchVjk0WVfxF8Z1A2ebSVmRVDXdRU7BB15QtX4d3qczMLw2lOxyKzKvvSlMsnt7cUOQUebnS12CDWW1oTXkZuqHTFxsRsREZGEVZseORER1QymvGzMVO2IiT1yIiIiCWOPnIiIJEULE86RV4MeORM5ERFJipiXn1VF0n8FRERENRh75EREJClawXSXjWnLubNbVcQeORERkYSxR05ERJLCy8/0MZETEZGkcGc3fRxaJyIikjD2yImISFIEE/bIBfbIiYiISEzskRMRkaRwjlwfEzkREUkKE7k+Dq0TERFJGHvkREQkKbyOXB975ERERBLGHjkREUkKLz/Txx45ERGRhLFHTkREkqKFCe9+ZpJWxMVETkREksLLz/RxaJ2IiEjC2CMnIiJJYY9cH3vkREREEsYeORERSYsgM91lY9WgR85ETkREksKd3fRxaJ2IiEjC2CN/BgorDdbuPg8v73z8uLMu1n3SRHds8LgHeL5rCrzr5cLeUY3sTAtEP7DF/h0+OH/KQ8Son93IwdfQ0C8NjfxS4eWZg4QkW4x/Y0iJddsExqFLh0do5JcKX590aBO2IjCgNyJu1DKo275NDF7qdQu+9dLh5FgAtdoMCUl2OP6LHw4c9YdabVbRL63CjHgjDg2b56Fhizx4+SiRGG2JCV1aGtQTBCWCRibj+R4Z8GuaByd3NdKSLBF12RbbP6+N6DvWIkRfserUz0f3gUlo0zkDXj75sFAIiH9khTOH3bB3S20o8//5d3/5lRg83y0N3r75sHcqRHaGOaLv22D/Vi+cO+4m4quoXHX8lOgxJB1tumbDq74SlgoB2pSBGPVmBvZscNN7z6ojLnbTx0T+DMa+fheOzqoSj/kHZCExzgp/nnFDZoYF7B3UeKFXIpasuoqwtQ2wY4NfJUdrOhPHXEZWtiXu3HOFrU3Jr79Y9xfuo1uX+3gQ7YRHsY5o6Jteal1fn3RotTIcOdkIqenWUFgWonnTJLz+yp94vk0sFnzYE5DoMNgr82ORlW6GO9dtYedQWHpFTQxmffIA13+3w5Hv3JGaaIFaPkr0H5uETkHpWDyhMSLOO1Re4JWg95BE9B8Tj99OuuDUj+4oLJQh8PlMTJj9EC/0Tcbs4S2hUhYlpsaBOUiMtcIfv7ogK90c9o6F6BKUgiVf3MTWz3ywY62PyK+mcvQZmYoBIam4cNQBJ/c6QaOW4c3PfBGy4BC6DsjAzAGNoCrggGtNIWoiT0pKwtatW3H16lVcv34deXl52Lp1K55//nkxwzJKgyZZCB79CBs/a4TJb90yOL58QaBB2b5vffD5t79hyIQH2PWNL7RaaSal8W8MRkKSPQBg/ar9sLJSl1p3047W+OyrDlAXmmHowBto6Hux1Lq79rUwKAs/1BSZWb9hYFAU/BumIuqONHtdIV1aICHaCgDw5dHrsLbRlFxR7oI3+gbg3l82esWn9rlizcEbeHVRNN4cEFDR4VaqM0dcsesrb+Tl/PPn6KedXoh7+BCjXo9Gn6GJ+HF7bQDA8tlNDM7fu6UO/rfnMoa+GotdX9aV7O9VeZw+4ISd//NEXvY/Pe9ZWz7DtgUXMXpWEoJGpWH/Jmn+rhhDMOFiN+61/ozu37+PDRs2IDExEf7+/mKGUi5yuYA3l/yFi+dccfaE8cPkWo0cqUkKWFlrYGYu3Y0Bi5O4MVLTbKAufLZhvsRkWwCAna3ymdoRU3ESL4tM7myQxAHg0W1rPLxljfqN800dmuhuX7fXS+LFfv2pKBHVa5z3xPO1GhlSE4t/r4QKibGquR1ho5fEi/2y3wkAUN+/oLJDqlTFQ+umekidqD3ygIAAXLhwAc7Ozjh+/DimTZsmZjhGCx7zEHV9c7H0bcM5zn+zc1BDbibA0UmFLr0S0bZTKiL+cIFaVb3nsJ6FtZUaFhYa2FirEdAkCSOCryMzS4Gbt93FDk00MpkAFw810lMsxA6l0rjVKvrillHCa7ZzVMNMDjg4q/FC3xS0fSEdEb85Qq2q2cPJbl5Fo2PpKZw1rUlE/de2s7MT8+mfimftfIx9/S6+Xe+HpHhreHg9uYe0Yd9ZODoX/XIVqmU4e8IDXyxrWhmhStbb087ihY6PdD9H3nLDmq+fR26epYhRieulsclw9VRj+2e1xQ6lUsjlAka9Ho1CtQynDhh+gfv6yEU4OhetNShUy3D2qCu+eL9hZYdZpQiCBmNmJ6JQDZza6yR2OBVLMOGQeDUYxOHXtnKa/k4kEmJssHdbPaPqL327JSwstXDzKECXXolQKLSwsSlEVnrNTUplCdvdEgeONoajgxKtmifAt146HOylO6z+rJq2zcaUxY9w94Y1dn7hJXY4lWLqonto1iYbmz6th9j7hlMNH01vCkuFFq6eKrwQlAJLhRbWthpkptecEYt/E7KXolm7PGz8uBZi7ho3lUPVQ80ehyqnbv3i0bpDKtZ83ASaQuPeuuuXnHH5giuO7a+D92a0QV6eGUI3/QE7+9IXiNV0Dx454/K12vj5rC9Wf9URR081xEeLTqCZf5LYoVW6hs1z8cGm20hNssS7rzSGWln9f2XHzXyIgePi8dPOWvhufd0S61z/0xGXzjrj2B5PvDslAPm5Zvh0x9UnXxFQjY2fGw/kbcPBMBfsWuMpdjgVTgsTzpFL9EqYx8kEQagSAwvFc+RVddW6IKggJHcFLAIhs3/nnwPaRAhpYwGrgZDZzQDkzpDJS788SFD+CiH9VcgclkJmM6wSIq9Y2pSXAG0e5B6nyqwr5H4DIfsTyJzDIFMY/28saPMgJLUBrPpD7hT6LOFWCca+Z4L6BoS0EEBmB5nrNsjM6lROgCLSZn8O5K4BrIdA5vAxZDLj/shWt9+r8nja90zKBv2yAvH5GSZpy8vaCeH/mWeStsRSbYbWJzy3BIkxaRXWvq2dGrtPpwHKnyEofzasULAfQsF+fL2qEaasOIggr5IX7nXsloQlq4Cv/28zfthaQjsmVtigYodi169KhpWVGuNf/qjMukMH3sCU8cBbb4Yh4sYxo5/DzlaJPVu0+O3sRSxZVvbzlIf8j0iTtmeML4/GwNpGgwl1xxgcO6rcjt6KMWjYPBfLtkchL8cM80bWQ2K0OH9oZFaKSnuuMdMfYuyMaBzb44H/LkqAIEzSO34kcyP6OE4s8dxOPVOw5Atgw+IN+OGbQ5URLrTZ2ZXyPE8y9q0EjHsrEUd3OaPPm0vR22y42CHBs547tt1fW8d2I/cAACAASURBVKHPIQhFD1O1JXXVJpFXtIICMyyda3htuKOzCtMX3cSfZ11xZF8dPLhlh8naPFhZF6IgX//tlcsF9B8RDQC4ec2xUuKWEmenfKRnGO5cFtzvJgDUmFXrDQJy8fG2KOTnmmH+SH8kRldeMhXL6GmPMHZGNI7vc8d/FzUqcSGToM2DlY0GBXn6V3zI5QL6j4kHANy8YvylkVI3ZnZREj++2xmr5tRF0MzqP+1SjHut62MiN5KmUI6zxw3nnopXrcdH2/xzXPMAYUdO48wJD8Q+sEV2ljlc3ZX4T1AC6vrm4dh+L9y47FyZ4ZtUj6534emeCwBwdCiAubkWo4dEACi65vvErw10dX3rpaNju6IvLwFNiua4e3a9h+Z///e+Q02Q9/dq9PWr9uPGTQ/cvueC1DQbODgo0SYwDm0CE3DvoRP2HJDuav8eg1Pg4V20C56jixrmlgJGzYgDACTFWOLE3qJrpgVNLD7edgt2jhqEb/JEs3Y5aNYuR6+ts4edq9UWnP1Hx2Hcm4+QGKvAlXNOeHFAst7xjBQLXD7nDGgeYNuvv+PMETfE3LdGdoY5XD1VeLF/Mur65ePYHg/cuFgzviAPCEnB+LmJSIyxwOXTdug2OANCfji6v1y0c2JGijku/VpzvtTUdEzkFcGsFk4e9EJAm3R06pYEaxsNcnPMcS/KHjs2+OHnQ4b7jEtJUI87aBmQqFcWMuoKAODqDU+9RN7QN1V37PHzi5341U+XyPf91BRtWsZhYFAU7O2UUKrMERPngI3bW2PfT01QoJTuiuQ+I1IQ2FF/KHbC27EAgIjz9rpEjsIYOLoULdgaNyeuxLYmdLZHYkz1SeSNWxR9UfGso8TbK24bHI/4zaEokZvVwolwDzRvl4lOPVNhbatBbo4Z7v5lhx1r6+LUjzVjxAYAGrcs2iTH01uNuZ8XfVEWMudi/pqi41fP2VbrRM6d3fSJvtht7dqiuZS7d+/iwIEDGDJkCLy9veHg4ICxY8ca3U5Fz5GXx+H4L0qdI69sFT1HXl7HzyxGzy6mned+FmLMkT9J8Rx5VVGZc+RledIcuRiqwhz5445pd6OXXPyFfpUxR/7SqVCTLnY72O1tk7QlFtF75J999pnezz/88AMAoE6dOuVK5EREVDMIJtxatTr0yEVP5FFRUWKHQEREEsJV6/pqzjJHIiKiakj0HjkREVF5cLGbPvbIiYiIJIw9ciIikhT2yPUxkRMRkaRoTbhq3VTtiIlD60RERBLGHjkREUmKABNefmaaZkTFHjkREZGEMZETEZGkFG0IIzPR49nj2bBhA/z9/TFo0CCDY5cuXcKoUaPQsmVLdO7cGR999BHy8/MN6qlUKqxcuRJdunRBYGAghg8fjvPnzxv1/EzkREQkLSZL4jLgGRe7JScnY926dbCxsTE4FhkZiZCQECiVSixYsABDhw7Frl27MHv2bIO6CxYswJYtWzBw4EC88847kMvlmDx5Mi5fvlxmDJwjJyIiekqffvopmjdvDkEQkJWVpXds1apVcHJyQlhYGGxtbQEA3t7eWLx4Mc6fP4+OHTsCACIiInDw4EEsXLgQISEhAIDg4GD0798foaGh2L59+xNjYI+ciIgkRTDx42lFRERg//79WLhwocGxnJwcnDt3DsHBwbokDgCDBg2CjY0NDh06pCs7fPgwLCwsMGzYP3evUygUGDp0KC5evIikpKQnxsFETkREVE6CIODDDz9EcHAwmjZtanA8KioKhYWFaN68uV65paUlmjZtisjIf26hHBkZCV9fX72EDwCBgYEQBEGvbkk4tE5ERJJSETu7xcfHQ6PR6B1zcHCAg4NDieft27cPd+7cwRdffFHi8eTkZACAu7u7wTF3d3dcuXJFr66np2eJ9QCU2SNnIiciohpvzJgxiI2N1SubPn06ZsyYYVA3JycHn376KaZMmQIPD48S2ysoKABQ1AP/N4VCoTteXNfCwqLEegCgVCqfGDsTORERScuzTm7/uy0A27dvL7FHXpJ169bBwsICr7zySqnNWllZASi6rOzflEql7nhxXbVaXWI94J+EXhomciIikpSKGFr38vIyqn5SUhK2bNmCmTNnIiUlRVeuVCqhVqsRExMDe3t73bB48RD745KTk/V68u7u7iUOnxefW1qvvxgXuxERERkpNTUVarUaoaGh6NGjh+5x9epV3L17Fz169MCGDRvQuHFjmJub4/r163rnq1QqREZG6i2Qa9KkCe7fv4/c3Fy9ulevXtUdfxL2yImISFKKdnYzXVvl4e3tXeICt9WrVyMvLw+LFi1C/fr1YW9vj44dOyI8PBxTp07VrUgPDw9HXl4egoKCdOcGBQVh48aN2L17t+46cpVKhT179qBNmzYlLoR7HBM5ERGRkezt7dGzZ0+D8i1btsDMzEzv2OzZszFy5EiMGzcOw4YNQ0JCAjZt2oSuXbuiU6dOunotW7ZEUFAQQkNDkZycDB8fH+zduxdxcXFYtmxZmTExkRMRkaRUxBx5RQgICMCmTZsQGhqKZcuWwc7ODsOHD8ecOXMM6q5YsQKrV69GeHg4MjMz4e/vj/Xr16Nt27ZlPg8TORERScyz75Gu15YJhIWFlVjerl077Ny5s8zzFQoF5s+fj/nz55f7uUtN5GvWrCl3YzKZDNOmTSv3eURERPR0mMiJiEhSxFzsVhWVmshPnDhRmXEQERHRUyg1kdepU6cy4yAiIjJOBezsJmVPtdhNpVIhPT0dzs7OJe4jS0REVFGksmq9spQrkd+4cQOffPIJLl26BI1Gg40bN6Jjx45ITU3FnDlzMHXqVL1r4yqTJi0DmuRUUZ67JFUlFgtrq7IrVTKLuHSxQ9ARnBzFDsGAvArFVNi4ao3MaQIbiB2CjuzcVbFDMCSrAkmpKsRQwxi9RWtkZCTGjBmD6OhoDBo0SO+Yq6srlEol9u7da/IAiYiI9Agmfkic0Yn8s88+g4eHBw4cOIC33noLwr+W+nXo0AEREREmD5CIiIhKZ3Qiv3jxIoYNGwZbW1vIShg6qV27dpk3PyciInpmf8+Rm+Jhuo1lxGN0IlcqlbC3ty/1eE5OjkkCIiIiIuMZvdjNx8cHN27cKPX4hQsX0LBhQ5MERUREVCpefqbH6B55//79ER4ejnPnzunKiofYN27ciNOnTxssgiMiIqoYMhM9pM/oHvnEiRNx9uxZTJo0CX5+fpDJZFi2bBnS0tKQkpKCTp06YfTo0RUZKxEREf2L0T1yS0tLbNq0CfPnz4dCoYBCocCDBw/g7OyMuXPn4quvvoJcbnRzRERET4eXn+kp14Yw5ubmCAkJQUhISAWFQ0REROXB+5ETEZH0VIOetKmUK5ErlUps3boVx48fR3R0NACgbt266NmzJ8aNGwcrq6q3HSgREVUzprz+uxpcR250Ik9LS8OECRNw+/Zt2NnZoW7dugCAu3fv4urVqwgPD8fWrVvh4uJSYcESERGRPqMT+YoVK3Dnzh0sWLAAo0eP1t31TKVS4dtvv8Unn3yCFStWYPny5RUWLBERkSAUPUzVltQZnchPnTqFoUOHGix0s7S0REhICG7fvo3jx4+bOj4iIiJ6AqOvF1OpVGjWrFmpx5s3bw6VSmWSoIiIiErFy8/0GN0jb9GiBf76669Sj9+4cQOBgYEmCYqIiKhUXOymx+ge+YIFC3DkyBGEhYWhsLBQV15YWIgtW7bg2LFjWLBgQYUESURERCUrtUc+fvx4gzInJyd8/PHH+Pzzz3Wr1qOjo5GTkwMfHx8sX74cW7ZsqbhoiYioxpMJRQ9TtSV1pSbymJiYEsu9vLwAABkZGQAAe3t72NvbQ61W664tJyIiospRaiI/efJkZcZBRERkHN7GVA/vckJERCRh3GudiIikhavW9ZQrkT969AibN2/G1atXkZWVBa1Wq3dcJpNxUxgiIqp41WBI3FSMHlqPiorC4MGDsXv3bt3CNhsbGyiVSsTGxsLMzEy3EI6IiIgqh9GJ/PPPP4eFhQXCw8OxefNmAMCiRYtw5swZfPDBB8jKysJ7771XUXESEREV4c5ueoxO5BcvXsSIESPg5+cHmUx/TmH48OHo2rUrQkNDTR4gERERlc7oRJ6bm6vbBMbCwgIAkJeXpzvepk0bXLp0ycThERER/Qt75HqMXuzm5uaGlJQUAICdnR2sra3x4MED3fGsrCxoNBqTB0hERKSHq9b1GJ3ImzRpguvXr+t+bt++PbZu3YrAwEBotVps27YNTZo0qZAgq7qxc+Iwbk6C7mdtQmMceWxjvEI18JJvGxEiq3x29iqMmHAbHbomwM29APl55nh4zx7bvvbHjauuunr+zdIxfupNNG6WDkCGyGvO2LyuKe7ddhQv+EqksNJg7Q8X4OWdjx93emPdMv3fnTr1cjFx1h00b5cOCwst7kQ6YPs6P1z93UWkiE1jZPA1NPRLRSO/VHh55iAhyRbjpw0tsW6bwDh0ef4hGvmlwtcnHdqELQhs1gcRf9UqtX0f7wyMHhKBlgEJsLdTIjPLCrfuuuKz9R2RkWldUS+r0nk3KMCYWYlo2CIPrp5qmFkI0Cb3wZT3cvD9Og+kJVmIHSJVIqMT+YABA7B9+3YUFBTAysoKM2fOxNixY3V7sltZWWH27NkVFmhVdvaQE+IeKHQ/L9gyHcsnrIFv03wMfz0JF47XjOTkXisPy9ecg7W1BkcP+CD2kS1s7dSo3zAbru4Funr+AelYvuYcUlOssP1rfwBA/6EP8Mnas3h7ahc8vOcg1kuoNGPfuAtH55Jv+1vLOw+fbv0TGo0MP2yqh9wccwQNicWHay/j3WmtcOU31xLPk4KJYy4hK1uBO/ddYGv75Nsed3/hHrp1uY8Hj5zwKNYJDX3Tnli/bctYvD/vFOIT7BH+U1OkZ1rBybEATRsnw8ZaXa0SuZuXGi6eapw77IjkeEtoCoFZazqj35hv8eLAdLze2x+ZqdU4mZtwr/UaNbTer18/9OvXT/dzs2bNcPDgQRw7dgxmZmbo2rWrbg7dGBEREdi7dy9+++03xMXFwcnJCa1bt8asWbNQr1698r0Kkd2PtMH9SBvdzwu/G4STe3bgzeWPAABHdriJFVqlevvdyzAzEzBt/H+QnmpVar2ps6+jsFCO+a93QmpK0R/X0ydq48sdP+PVN29gyayOlRWyKBo0yULwmGhsXN0Qk9++bXD8lTfvwNZejZmjnse9KHsAwIkfvfDlngt4Y1EUpgzqCECaw4Hjp72MhKSi17T+03BYWalLrbtpR2t89lVHqAvNMHTA9ScmcieHfCyceRoRN2rh3U+6Q6Op3ptWXjljjytn7PXK5mx9F5+OO4PFXz1A7+Fp2L3OU6ToqLI906fdy8sL48ePx5gxY8qVxAHg66+/xrFjx9CpUye88847GD58OH7//XcEBwfj7t27zxJWlaCw1uDFgWlIjrPAnz9X/x5mQKtUNG+Vhh+2N0R6qhXMzLRQKAoN6gmFD+HfLANnTnrpkjgApKZY48xJL7RqlwJnlwKD86oLuVzAm+9F4uI5V5w94WFwXNDm4fkXU3DtT2ddEgeAgnxzHNlbG97189C4eVZlhmxSxUncGKlptlAXmhlV96Xet+Bgr8SGbW2h0cihsCyEmZm27BOrmaSYol64nWM1X6/ExW56RNuiNSQkBKGhobC0tNSV9evXDwMGDMCGDRuwfPlysUIzia79M2DroMW+jR7QaqXZeyqP5zomAQCSE63x7orf0a5DEszMBcQ+ssWOTY1x6oh3UUX1NQBA5HVngzZu3nBG7wHRaNgkE3+cK71HL2XBYx+hrm8ulr4VWHKFwihYKrS4edVwOuZmRFFZ44As3LpeM6ZrjNW+dQxy8yxgZ6PCupX70aB+OjRaGf6KcsdXW57DrbvVc1TMQqGFtY0WllZaCMozePOTosU5f5ys/p0H+kepiXzhwoXlbkwmk+Hjjz82qm6bNoaLv+rXr49GjRpVix550MgUaLXAkV3Snc8sjzo+OQCAGQuuIi7aFqs+agVzCy0Gj7qHt9+7DDNzLY4f9AG0iQCA1GTDRF1c9vh8enXiWScfY9+4i2+/8kNSnDU8aucbVtIWfSFKSTJ8f1KSitZhuHooKzROKfKunQUzuYCP3zmOXy/Uw/bvW8LTPQejh0Qg9P0jmLGwHx7GGH55lLq+o1IxbWksAEBInwg7B0ssn+6D67/biRwZVaZSE/nevXvL3Vh5EnlJBEFASkqK5Fe/C4X30Pz5XFw+bY/EaEXZJ1QDNjZFw+j5eeZYOL0TCguLZm0u/FoL3+w+iQlTb+LET3UBoShJq9WGQ6YqVVGZQlE9hwWnL45EQow19ob5lF5JKEruarXhKI5aWfSeKqyq5/vzLGys1TAzE3DiVz+EftFFV377nitC/+8IxgyLwMf//Y+IEVaMc0ccEX3XClY2Gry/6yXkZG2Eo0v1/3zITLjYzWSL5kQkEwShyryM8PBwzJs3D8uXL8fgwYPFDuepabNXALlfQ+a4CjLr/mKHUym06VMB5SnA9nXI7fWvXtBmzAMK9kHm9hOg/AVC9ieQOW+ATKH/h1VQ/gwhfQpkDh9AZjOyMsOvcEJ+OITMeZC5bIfMsl1RWWEMhJTugM0YyB2KtjcWCo5AyJgBmcP7kNmM1m+j8A6ElH6A7VTI7d+q9NdgatqUlwBtHuQep8qsK+R+8/fnJgwyxfOGbSW2BoRcyJw3QaborH8s6UUASsg9zpso8qpLUN+EkDoEMrsZkNm9JnY4FeaFTRsQm22atSJ17B1w+pXJJmlLLFXmNqZ3797FBx98gLZt22LQoEHlPn9sgxlIfJhcAZGVj9xMwKHoaGSmmWGM31aoVdtEjcfcp06lPM8bb9/DSy8D6z44jQM/xOgdC3k9EsPGAW+9vBSf7hoCAFg97384+uN+vXp9Bj7EmwuA96YcwB/n/qiUuIXcEoa3TczcQoutR0/j1nVXfLVija7c1UOJFRuBk9+dwPYv7yMrwwLf//UGAGBH6GZs/eKMXjutO6Ri6VfA2nd+xoFdlTP9VNi44j4/6z9NhpWVGuOHLC2z7tAB1zFlPPDWm9sQ8ZfhHRa//q85fLyBya/sxYPon/WOfbZUhYZ+6XjphbKfpzxk566atL1ndUzzHXpbvYvVP1rArdb/MPa5k6LE4VnPHdvufVGxT8INYfRUiUSenJyMqVOnwtHREZ999hnkculeOtKhVyagTcHJPe5Qq6T7OsrrVqQTXsJDuHkYzm+7eRQly4x0BWDRAgDQtHk6jv6of5lhk4B0aLXAnZvVayGXwkoDJxc12ndNQfuuKQbHu/dPQPf+Cfj600aAeWOolHI0aZlpUK9JYFHZ7b+4kOnfou64wcc7E26ueXgQrT8X7uaah4zM6rl4siQKKy3snar/8Dr9Q/REnp2djcmTJyM7Oxs7duyAu7u72CE9k6CRRX+oD++snqtkS3Ph11rIm3UD3frEYOfmRijIL/poObsWoMMLCYh5aIv4WFvIzOshKtIRXbrHI2xDE6SlFP2BdXErQJfu8Yi46Ib0tOr1R7cg3wxL32phUO7orMb0xTfx5xlXHNlbGw9u22GK3Ba//eKGTj2S4Ns4G/dvFV2uZWVdiD6D4xD70AZR15jI/+34r37o9eJd9O8VhT+v/DOK0KFtNNxd8/DT8UYiRmd6zu5qpCcbbvjSslM26vkXIOJ8DVjsVmUmhcUnaiJXKpV47bXX8ODBA2zevBl+fn5ihvPMXDxVaPdiFmARiAc3Rf+OVKlysi3xzZpmmDE/Ap9uOINjB+rCwkKLfoMfwtxCiy//21xXd/1/m2PZmvNYse4sftztCwAYMOw+ZDIBX/+vmVgvocJoCuU4e9xwc47iVevxMdZ6xzd/3hCtnk/DR19exr5tPsjLMUPQkFi4eijx3oxWkOpmMADQo+tdeLoVXeHg6FAAc3MtRr9cNESdmGKHE7820NX19UlDx3bRAICAJkXTZj3/cxfNmxRd+bDvcFPk5RVdvnr5Wm2cPOOL7l3u46OFx/HbRW94uOdiUN9IpKZZY+t3rSrtNVaGGcti4OKpxtWzdkiMsYSlQoA2Yy6Wbr+H/Bw51n9QW+wQK5Ypr/+uBl8IRMs2Go0Gs2bNwpUrV7B27Vq0aiX9X7Tew9JgZg7IrIcBKP+qf6k7HF4PWRmWGDLmDsZNjoJWW3Rt+Ir32iDy2j97hEded8GC6R0xbkoUxk25CUEoKlv2Tlvcv1O9htWfRny0Dd6e8BxCZt7GsFcewNxCi7s37bHkDWlvzwoAQd1vo2VAol5ZyKgrAICrNzz1EnlDvzTdsX/Ov6P77xOnG+gSOQCs+F8X3HvgjD7d72BqyB/Iy7PE6Qv1sHlHG6Sl26A6+TncCT2HpqPHkHQ4uhRCEACoI/DTNlfsXueB5DjLMtug6kO0VetLly7F1q1b0a1bN/Tt21fvmK2tLXr27Fmu9qrKYjcAOFa4E73Mq8aq68pa7GasQ/c+RV+/qrPiujIWu5XH4cS1CPJ8Q+wwdCpysVt5HT/9DnqaeMHas6iKi916mQ0XO4xKWezW9euvEZtlolXrDg749dVXTdKWWETrkd+8eRMAcOrUKZw6pX/5SZ06dcqdyImIiGqicifymJgYnD9/HikpKRgwYAC8vb2hUqmQkpICNzc3vS1XnyQsLKzcwRIREXGOXF+5EvnKlSuxefNmaDQayGQytGrVSpfIX3rpJcycORMhISEVFCoRERGYyP/F6Audd+7ciW+++QajR4/Gxo0b8fjUup2dHbp3724wRE5EREQVy+ge+bfffotevXrhnXfeQXp6usFxf39//PFH5ezGRURENRf3WtdndI/8wYMH6NSpU6nHnZ2dS0zwRERE1cW1a9cwbdo0dOvWDYGBgejcuTMmTZqES5cuGdS9dOkSRo0ahZYtW6Jz58746KOPkJ9veKWMSqXCypUr0aVLFwQGBmL48OE4f974ewMYncgVCkWJARSLi4uDgwN3nCIiogpWvNe6qR7lEB0dDY1Gg2HDhmHJkiWYNGkS0tLSMHbsWJw9e1ZXLzIyEiEhIVAqlViwYAGGDh2KXbt2Yfbs2QZtLliwAFu2bMHAgQPxzjvvQC6XY/Lkybh8+bJRMRk9tB4YGIhjx45h4sSJBseUSiXCw8NLvMc4ERGRSYm42K1fv37o16+fXtmoUaPQs2dPbN26FZ07F919b9WqVXByckJYWBhsbW0BAN7e3li8eDHOnz+Pjh07AgAiIiJw8OBBLFy4ULdYPDg4GP3790doaCi2b99eZkxG98gnTZqEK1euYO7cuYiKigIApKSk4PTp0xg3bhwSExNLTPJERETVmbW1NVxcXJD19yY1OTk5OHfuHIKDg3VJHAAGDRoEGxsbHDp0SFd2+PBhWFhYYNiwYboyhUKBoUOH4uLFi0hKSirz+Y3ukXfq1Anvv/8+li5digMHDgAA5s2bBwCwsLDAhx9+iNatWxvbHBER0VORwYSL3f7+//j4eGg0+neNc3BwKHXKOCcnByqVChkZGdi3bx9u3bqFadOmAQCioqJQWFiI5s2b651jaWmJpk2bIjIyUlcWGRkJX19fvYQPFI2CC4KAyMhIeHh4PPE1lOs68hEjRqB79+44fPgw7t27B0EQUL9+ffTt2xeenoY3hSAiIpKCMWPGIDY2Vq9s+vTpmDFjRon1Fy1ahCNHjgAo6syOHDkSr732GoCiW3MDKPFunu7u7rhy5Z97CCQnJ5eYP4vPNWmP/PHGx40bV97TiIiITKMC5si3b99eYo+8NNOmTcOIESOQkJCA8PBwqFQqqNVqWFpaoqCgAABK3OlUoVDojgNAQUEBLCwMb0mrUCgAFK1BK0vNutcmERFRCby8vMpV39/fH/7+/gCAgQMHYsiQIVi4cCE+//xzWFlZASi6rOzflEql7jgAWFlZQa1Wl1gP+CehP4nRiXz8+PFl1pHJZNiyZYuxTRIREZVbVdsQxsLCAj169MC6detQUFCgGxYvHmJ/XHJyst6ct7u7e4nD58XnljU/DpQjkcfExBiUaTQaJCcnQ6vVwtnZGdbW1sY2R0RE9HSq4F7rBQUFEAQBubm5aNy4MczNzXH9+nX07t1bV0elUiEyMhIDBgzQlTVp0gRhYWHIzc3VW/B29epV3fGyGJ3IT548WWK5SqXCpk2bsGfPHt7RjIiIqrW0tDS4uLjoleXk5ODIkSPw8vKCq6srAKBjx44IDw/H1KlTdQk6PDwceXl5CAoK0p0bFBSEjRs3Yvfu3brryFUqFfbs2YM2bdoYtZD8mefILS0tMXXqVNy5cwfLly/HqlWrnrVJIiKi0onYI581axYUCgVat24Nd3d3xMfHY8+ePUhISNDLf7Nnz8bIkSMxbtw4DBs2DAkJCdi0aRO6du2qt915y5YtERQUhNDQUCQnJ8PHxwd79+5FXFwcli1bZlRMJlvs1rZtWyZxIiKq1gYOHIjw8HCEhYUhKysL9vb2aNWqFVasWIH27dvr6gUEBGDTpk0IDQ3FsmXLYGdnh+HDh2POnDkGba5YsQKrV69GeHg4MjMz4e/vj/Xr16Nt27ZGxWSyRB4TE1PiyjsiIiJTEnOx29ChQzF06FCj6rZr1w47d+4ss55CocD8+fMxf/788gXzN6MTeVxcXInlmZmZOHfuHMLCwvS+jRAREVHFMzqRd+/eHTJZyXeJEQQBvr6+WLx4sckCIyIiorIZncinTZtWYiJ3cnJC/fr10alTJ8jlRt+DhYiI6OlUwcvPxGR0Ii9tv1kiIiISj1Fd6NzcXPTs2RObN2+u4HCIiIierHixm6keUmdUj9zW1hYZGRkGt1kjIiISRTVIwKZi9KR2y5Ytce3atYqMhYiIiMrJ6ET+9ttv4/Dhw/jhhx8gCPwqREREIhFM/JC4Jw6tx8XFwcXFBVZWVli2bBkc3dU8jgAAIABJREFUHBywePFirFy5Ej4+Pnq3YgN49zMiIqLK9sRE3qNHD6xcuRL9+/fX3f2s+J6tKSkpFR8dERHRv5lykVp175ELgqAbRi/t7mdVhlZT9Kgqqkgs2qSq94WrKsUkaKrGv9PjtFlZYoegEzun7HshV6bYOVVnG2jnWlVvJ8vcl8WPKc/dQewQahyT7bVORERUKbghjB4mciIikhQxb5pSFZWZyP/8809oyjH8GBwc/EwBERERkfHKTOTfffcdvvvuuzIbEgQBMpmMiZyIiCoWh9b1lJnIhw8fjlatWlVGLERERFROZSbydu3aYcCAAZURCxERUdnYI9fDxW5ERCQpXOymjzcQJyIikjD2yImISHqqQU/aVJ6YyG/evFlZcRAREdFTYI+ciIikhYvd9DCRExGRpHCxmz4udiMiIpIw9siJiEhaOLSuhz1yIiIiCWOPnIiIJIVz5PrYIyciIpIw9siJiEhaOEeuh4mciIikhYlcD4fWiYiIJIw9ciIikhTZ3w9TtSV17JETERFJGHvkREQkLZwj18NETkREkiKDCa8jN00zouLQOhERkYSxR05ERNLCoXU97JETERFJGHvkJiCTCQh+NQUvjUuFp7cK2qSumPKuBltWekKZbyZ2eJVm+GuxaBiQg4YBufDyUSIxRoGQF9uUWr9d13QMeTUOPo3yYWOrQUqCJX476YzvN9RGRqplJUZe8Ua8HoeGzXPRsHne3++NJSa80KrEum+tvIdeQ1NKPPbRGw1x5pBLRYZaqWRKLTxn3YF5kho5QS7ImOyld9w8VgnHsEQo/srFAGEs3OuZIWuEO5Qt7AzaMktWwf6HFFhdy4VZmhpaOzOofK2QPcgNqgDbynpJT21sr8vwr5sC/7opqO2WjfhUOwx7f/QTz+nz3C0M6hKJBl5p0CZuw9aFCpy41ABbjuj/3jX2TsErfS8isEECrCwLEZfigP3nmuCHXwKgFSTYn2OPXA8TuQlM/b84DH41BWd+csQPX7pj1tqeGDRpMxo0z8eCEX4QhOqwnKJsr7z9CFnp5rhzwxZ2DoVPrBs0IhEzl97DrWu22L2+NpR5ZmgUmIPgkHh06p2G119qWa2+BL0yLwZZ6WZGvTfFVsz2MyiLulr1E1J5OOxMgjxLU+IxswQV3BfdB8yA7GA3zOw4EJ9+ug1uHz5Eyjv1oGz5TzKXp6nhMe8eZBoBOb1cUOhlCbN0NWyPZ8D9/QdIXeCDgrb2lfWynsprA/9AZq4Ct6LdYGetKrP+wtE/I+j52/jlii+O/tEIcyf3xrkbP6CWS7ZevZYN4rFq2k/IzbfE9780R0aOFZ7zj8XMIefhWysdK3Z2raiXVGF40xR9VSqRb9iwAaGhoWjSpAnCw8PFDsco9RoXYNDEFJw56IgPJ9cHAMzZtghfzTmKaR/F4cXgDJza6yxukJXklW6tkRBtBQBY99MVWNtoS607ZFIcUhMt8PaI5lCrinoEh3Z5IiPl/9u78/iY7v3x46+ZmOyLhCQ0KmILSoRoi+JnK6laUrsSlEsX3KJaUe393qKohlJaaxdFtU0rDXWv3W2rWlpUqg0q1kgkIyEh20wy5/fHNMOYWDtxZng/H495PORzPnPmfUYm7/NZR8egMWdp/lguP267d1qew9tFWN6bJZt+w8Or/OR1tR1fV63osFSlO16I9zfZ5MYGU3llps1xv9WZaAtKyZpTG2OYBz2f6kpc5V0Ej0+l8ooMMt+tCxrzTbLX/y7iklfK+ckPUvSIr+UcBW38qD72GF5bLzh8Iu//74GkZ5tj/2RKAh5uxuvWfbLlYZ5sdZTpn7Rn88/1AXjlxYEsWZ9mU3d8390oCjw3r5fl/InfP8TLA76jV5vDbNpbn+Tj1ex/QeKucZg+Fb1ez+LFi/H09FQ7lNvSPuYCWi0kLrf+o/vfNVUoKtDSsfcFlSK7+8oS1a3w9C7lcl4lSxIvk51l7lIvKnSYX027uJ335goFT+9SNPdCk+FapQr+i9MpauZNYUtfm8OaIhMev1yi+CEvjGEelnLFw4X8zpXRpRvQHSu8Ur/AfNNYGqCzOo+pciUULSjujv/7VJZkb04htsuvHDld1ZLEPdwMKIrt74mPRzH1amRz8Fh1m/P/Z084AN1aHvlbcatCsfPDyTnMb/fcuXNp3LgxjRs3VjuU21I/soDSUjjyq/UNiLFYS+rv7tSPLFApMse27/vKhNYr5B9TTvJgnQKqVi+mdZdsnh6TRvIeXw7+6Kd2iKr7Knkf637bR1LKL8xcdZjwyMtqh2Q33t9kU+lsMRf/Ub3c47pTRWiMCob6HjbHDPXNnzXXqxJ5WTe7/7J03A7lo802ojtWSMA7aSjuWi71rFIBV6GOmsG51AjM47cTwQzrup+Ns1eyNf5jlKwoJg34Hg/XKy15XSVzz0+R0bbztfivsodqZd2dwEWFcYiu9eTkZNavX89XX33FzJkz1Q7ntlQJLiEvx7ZlCZCdoeOhhwuopDNRYnSYeyaHsGR6Ldw8TMQMy6DPyAxL+ZaEQN59vTYm0/0xr6A8OXod61YE8+chL4oKXKjdsICYEeeI/zyFf42oz4EfnPsmxyXTgO/nWVzqF0hpkCsuWbbjwS455nkE17awzWWVrOoAFDfx4sKo6vh+lkXg/520lBuru5I1qzYlNdzsfBXqqRl0EYBOzVOp5GLik83NSM/24c1xvsS0WUfNoIv8c2F3QEPOJQ8uXHLnoVpZuOpKMFyV0JvVSwcgqLIT3iDacYz8XmiRq57IFUVh+vTpxMTE0LBhQ7XDuW1uHiaMhvKTjqFY+1cdhZLrD3fdl0pLNOjTXdm9NYA92wMoLtLSvO1FuvTNotSk4d2pddQOUTUfzXnQ6ucft/qzc30V3t94iLHTTzKyY1OVIrMP/6XplAa7cqnH9ecAaIrNXeWKzvazpei0VnXKmHxdMNbx4FKEFyUPuFIp3YBP0nmqvnkK/fQwSqva3hQ4I8+/xs79fYoYv6gbvxypAYDWbxLf/O8Q3R49SstGZ/jpj5qAhi/+14Rne/zMzH9sZcXGKHLz3WkRfpaR3X6hpFSDm+utTb4Ujkv1RP71119z7Ngx3nvvPbVDuSPFhVo8vMr/ILi6mf6qc/+2Lsuj0ShM/zAFl0oKL/VvTNkmibs2VeHSxUr0fzad7zZW4dfdldUN1IGkn3Tnu40BdOl3npCwQs6esO1ydgae317ELTkf/fRaUOn6nwvF7a9kbbRtLmmMJqs6AF5bc6i8PIPM+DqU1LwyH6Eo0pvgl1PxXZPJhRdr2Okq1FVsNK/myLrgZUniZf67pz7dHj1Ks7oZfyVyWL01EnddCQM7JrPi5TMAFBTpWJjYktHdf8ZF64RNUll+ZkWjlDdD4i65fPky0dHRDB48mOeffx6A2NhY8vLynGbWuilnBBh2owlORqOxXvtsyh4IJSfQBu9RKTr1mM4/CaYCtEE7bY4php9Rcgaj8ZmMxmuk9THjIZTs3uD1Alqf8Xcr3LvqRu/NjSiXF6JcXogm4DM0rtdfn++oDMVGBtd8jvBH6vL8O8Mt5efP5jCpw7/pNKQtsf/qh19VX06npPHiY6/x9Ku9eWbGIKvz7Nt6kLiuMxi7cCS9xkQDEFtnDO5ebixPnmfzuqMiJpJ/sYBPTy+p0Ouzpxt/fg6i5PSDSo3RVl1nfawkFeX8E+DRH63fDOtjpgIoOQoooGsAigklqznoItFW+bwiL8funpz6ARk5eXY5V/UAXza+OfLmFR2Yqi3yxYsXo9PpeOaZZ/72uYaEvUDmKb0doro9w17J4OnxJia26sWhveYJN1tNCXTz6MOXv//Obz958dqQfnc9rjJalVYBLP5PGh6eJobXHmZVvvnySmYN/Tdx82HFa2v5cvl3VsfDm15i/lfw+dyv+Tj+QIXHqZTefBmYvS3ZlIaHVynDQofYHNtSuJouHrblAK+8k0rHGBjeZCYZp+9kFvztO/NpXbudS5NfSog+jz0b97Nn436b49tXf8/21d9zcWgw+V38eUCn4cMNm5kTZZ5VnfLUv2mY+G98ErLwA/51eQdxiT8BEHJGT0k1Vxom/tvmvMEXMtHmm8o99nf4f2q7KY29fDLlPB5uRvqNibc55qYr4T+zXbiUm0rM81eO706YxIv/Xsj8sbAy8U+Wb7R97hXbaR95nBkjFZYnurBy843q3p5qgb6se3+03c4nbk61GVhZWVmsXLmSp59+mvPnz5OWlkZaWhrFxcUYjUbS0tLIzc1VK7xb9u36yphM8NQo6524nhicjbuniR3r7o815Lfj9DFzt3CHnudxqWQ9zvl4b/PN2NHkivsj6cgUUwE6V9v193Ua5dPmiRxO/el+15K4vSluWrIn1bB5XPhrN7eiZt5kT6pB0cM+KB4uFLbwwe33fHQniyzn0BSW4rXtIsbqrhjqXRleMD7oRqX0YlyPWq8ScT1SQKUMA4a6zjkUUZ5iYyX+dzCMqn6FtIs4YXUsps0fAPz4x4PlPdXC17OIZ3v8zIVL7ny9q1GFxVqhZOmZhWot8uzsbIxGI/Hx8cTH294NdurUiVGjRjFp0iQVort1Jw97sOHjKvQakc3rK07y8w4fTHmzePb/0jm424udiffPOG/HGD1BDxQD4BdQQiWdiYEvmDeoyEp3Y8fXgQCcOOzFrk0BtInO4d3E39iRVJXiIhei2l6kZacLpBzw5qd7aDMYgE5PnSco5K/3poqRSjqFQWPPApB11o3tiX9N/Co9ycrvD7J7iz/pJ90oKnAhrGEBXfvrMZk0vPtqmFqX8PdV0lDYynbGfdms9ZJgV6vjuYODcP8tn6rTTnK5exXWZ2wmcO5JXHKMnH811LIZDEBe/0CqvH2Gqm+cIr+LPyXVXamUYcBrywWopCGvf2DFX9/f1PXho1QLMM8gr+xdhK5SKcO6mnsuzuV4W9aMAyzb8Agtws/yf8N28OV3D3EuxwfThdG0jzzJf/fU49CJKxu8tGx0mqc7HeTnIzXIyfMgOOAyPVodxsezmLhlXcnNd84bQ3GFaom8Ro0a5U5wmz9/PgUFBbz66qvUqlXr7gd2B5b8K4TMM648MSSHRzrlQdFGkj6syidvV7tvtmcF6Novi4hHrcethk00T65J3uNrSeQAb02ox5GDGXToeZ7Y8WfQaiHzrBufLQ7hs/dD7rnlZ13764loab115rCXzIk8+SefK4lcG8iBH3xp2iqPDr0MuLkr5GTp+PabAD5//wHSjt87LcubKa3uRtabYfitzsQn8TzL1n2CElqJ869Zb88KUPSIL+f/FYpPUjaeOy6iLSjF5O1CUVMvLvULtNpUxlF1b3WEZvUyrMpGdf8FgAN/VrdK5JkXvHl2bgyju/9Mt0ePmrd0LanFosSWfL6zidU5zuX4YCipRJ92h/DzKiY3351fjjzAys3NOZPlpA0NmexmRdXJbuW508luao2Rl2erKYHHteqNi19NrTHy69l8eSVdvYfdvOJdosYY+Y3caIxcDfYcI/+7ysbIHUVFjpHfid0Jk2jdz35j3XfqboyRd5/yARnZdprsVsWXb2Y592Q32aVECCGEuEXJycm88cYbdOvWjcjISNq3b8+ECRM4deqUTd39+/czaNAgmjZtymOPPcaMGTMoLCy0qWcwGHj77bdp06YNERER9O/fnx9//PGWY1J9Hfm1Vq1apXYIQgghHJmKXesrVqxg//79REdHEx4ejl6vZ82aNcTExPDll19Sp455M6uUlBSGDx9O3bp1iYuL49y5c3z44YekpaWxZIn1Usi4uDi2bNnC0KFDCQ0NJTExkVGjRrFq1SqaNWt205gcLpELIYQQjmr48OHEx8fj6npl35Bu3brRo0cPli9fzuzZswGYN28elStXZtWqVXh5mb9+uEaNGrz22mv8+OOPtGrVCjC38Ddu3MiUKVMYPnw4ADExMXTv3p34+HjWrFlz05ika10IIYRT0SiKXR+3o3nz5lZJHKBWrVrUq1eP1NRUwLzZ2e7du4mJibEkcYBevXrh6enJf//7X0vZpk2b0Ol09Ot3ZV6Vm5sbffv2Zd++fWRl3fxLbSSRCyGEcC4O9jWmiqJw/vx5/P3N+4YcOXKEkpISm2/zdHV1pWHDhqSkpFjKUlJSCAsLs0r4ABERESiKYlX3eqRrXQghxH0vIyOD0mtWsfj6+uLre/PviV+/fj2ZmZlMmDABAL3evIIqMNB2/4LAwEB+/fVXy896vZ7g4OBy6wG31CKXRC6EEMKpaOz4NaZl5xk8eDBnz561OjZ27FjGjRt3w+enpqYybdo0oqKi6NWrFwBFRebdCK/tggdzt3nZ8bK6Op3tN/O5uZm/ere4uPim1yCJXAghxH1vzZo15bbIb0Sv1/Pss8/i5+fHggUL0GrNo9Xu7ubd8gwGg81ziouLLcfL6hqNtt9zXZbAyxL6jUgiF0II4VwqYPlZ9erVb+tply5dYtSoUVy6dIm1a9dadaOX/busi/1qer2eoKAgq7rldZ+XPffqutcjk92EEEI4FQ1Xutf/9uMOXr+4uJjnnnuOkydPsnTpUmrXrm11vH79+lSqVIlDhw5ZlRsMBlJSUmjYsKGlrEGDBpw4cYL8/HyrugcPHrQcvxlJ5EIIIcQtKi0tZfz48fz6668sWLCAyMhImzo+Pj60atWKpKQkqwSdlJREQUEB0dHRlrLo6GiMRiMJCQmWMoPBwLp162jevHm5E+GuJV3rQgghnIuKO7vNnj2bHTt20KFDBy5evGj1vSBeXl507twZgAkTJjBw4EBiY2Pp168f586d46OPPqJdu3a0bt3a8pymTZsSHR1NfHw8er2emjVrkpiYSHp6OrNmzbqlmCSRCyGEELfo8OHDAOzcuZOdO3daHQsJCbEk8oceeoiPPvqI+Ph4Zs2ahbe3N/3792fixIk255wzZw7z588nKSmJ3NxcwsPDWbZsGVFRUbcUkyRyIYQQTqUilp/dqtv5PpAWLVrw2Wef3bSem5sbkydPZvLkybcXzF9kjFwIIYRwYtIiF0II4VxUHCN3RJLIhRBCOBU1u9YdkXStCyGEEE5MWuRCCCGci6KYH/Y6l5OTFrkQQgjhxKRFLoQQwrnYcYxcJrsJIYQQd5vMWrciXetCCCGEE5MWuRBCCKeiUUBjst+5nJ20yIUQQggnJi1yIYQQzkXGyK1IIhdCCOFUZGc3a9K1LoQQQjgxaZELIYRwLrKzmxVJ5Pc4U2Gh2iHYcKiYHPBDrBQXqx2ChedGX7VDuOIpx4pn18LFaodwjUnsWrhU7SDAJQQYrXYU9xVJ5EIIIZyKjJFbkzFyIYQQwolJi1wIIYRzkeVnViSRCyGEcCrStW5NutaFEEIIJyYtciGEEE7GjsvP7oG+dWmRCyGEEE5MWuRCCCGcioyRW5NELoQQwrnIrHUr0rUuhBBCODFpkQshhHAq0rVuTVrkQgghhBOTFrkQQgjnYlLMD3udy8lJIhdCCOFcZLKbFelaF0IIIZyYtMiFEEI4FZnsZk1a5EIIIYQTkxa5EEIIJyN7rV9NWuRCCCGEE5MWuRBCCOdixzHye6BBLolcCCGEk5HlZ1aka10IIYRwYtIiF0II4VTMy8/s05SW5WdCCCGEUJW0yIUQQjgX018Pe53LyUkiF0II4VQ0imLHrnXn71uXrnUhhBDCiUmL3A5CahfTqc8Fmre7RPVaxZgym/H+1lK+31CZdcurUlzoonaIqqlRp4jB4zOp26SAKsFGTOciWPGtwt4dvny5OIicLJ3aIapqwNhM6jYppF5EIdVDDZiyOgBV1Q6rQtSscpEnIv/k0bpnqBGQh2ulUs7m+LLtUG3W7o6gyGj9uxBa9SJju/xE87B0dC4mTNk/06L2A/xyPMSq3pKRSUSFZVz3dfccC2Hsxz0q5JoqQt4FFz57N5jdm/04n6HD08tEaINChr58jiaP5gMQP74mW78IuOaZ/YBIXlt2grbdcy2lhflavloSyJ/Jnhw75MH5DFciWl3m7a+O3b2LsjdZfmZFErkddB2YTY/h2fy0xZcdiZUZ++6zpKXOY3jcOdr1uMiLPephKLo/Oz+qVjcSEGxk9yY/9Bmu/HPRaPZ/9y7dBmfTvucFnu8STm72/ZvMR7x6jrwcF44d8sDbtxSfKmpHVHF6Rh2m76O/8/3hUDYdrEdJqZYWtdN54fGf6dz4OCOWPkVxiflPUkhALitGJ1Jq0rLq+0guF7kyuXc2C4dt5MVPurE3tYblvB/+rzlJvxTavN7jTVJp2+AU3x+udbcu8W/LTNPxcp+6FOW70HVQNjVqF5N/yYXjf7iTnWH7OXll4SnLvzV+8Si5kwiPLLCqk5tTiVVzq+MfaKReRAEX9Pfv5+1epWoiNxgMLFiwgKSkJPLy8mjQoAETJkygVatWaoZ1277/pjKfLQym4JK55T1uxSBmPreOs8czeHp8FtGDclj/0b3ZyrqZX3f58OsuH8vPL344kPdfX8dve7x5belJuvTPIWFxsIoRqmtYywacO+0GwNIdR+7pRL79UG0++rYZ+cVulrJ1Pz/E6Ww/RrbfT8+owyTsaQzAmMf34ONuYOj7fTh6zvzZifvHs+iPtuWV7t/Td8FAQAPA3tQHy329Ee33U2x04b8H61XshdnRW2NDKS3RsHj7YaoEl9y0fqc+Fyz/1lZrh+ncBZs6AUFGVv/yO4EPGAHoVbeJ/QJWjey1fjVVm4lxcXGsXLmSnj17MnXqVLRaLaNGjeLAgQNqhnXb/kz2tCTxq327vjIAtcKL7nZIDi8rzdwq8PYrVTkSdZUl8ftBSnqQVRIvs/W3OgDUCc4BwF1npF2DU+w/8YAliQNotF4k/dKQ0MBcGoVk3fC1IkMzqBV4kf+lhJFX6G7Hq6g4v/3kxe97ven/QhZVgksoMUJRgeaGz1EUyL+kxXSDmdeuboolid8ryr7G1F4PZ6daizw5OZmNGzcyZcoUhg8fDkBMTAzdu3cnPj6eNWvWqBWa3VStbv7wXDgvIxg6NxMeniaU0nM0b5fHyKnmMc2fd/iqHJlQW7Cvedw357IHAPWqZeOmKyX5jG1PzW9nggB4qIaeP85evyenV1QKAEm/NLB3uBVm73bzZyEwxMC/hobx805fTKUaQmoXMXhCplXru0zv8CYUXHZB52qiSbtpDBvvSYPmBTb1xL1NtRb5pk2b0Ol09OvXz1Lm5uZG37592bdvH1lZN77jdnRarcLgCZmUGGFnYmW1w1HdE4OySTh0CEXfjllrj+PtW8rssTU5tNdb7dCEirQaEyM77KOkVMvmZHMXeFUfcyLS53nZ1C8rC/wr+ZfHy81Ap8bHOZvjw8/XTIxzZGmp5t6K+S8/yKWLLry84DQT552mkk5hzrhQNn92ZXKbf6CR3qOzGDc7jf/74AQDx2Vy9JfjvPRUXfZ/dx98phTFvg8np1oiT0lJISwsDC8v6w9rREQEiqKQkpKiUmT28dy0szRqUcAnb1cjLdU5uvYq0u7NfsQNrIOm8nusnhfM5TwX/ALu7251ARO77SaiZiZLt7fg1HnzDa+7q3ls2FBiO1xl+GsynLvu+uPHXSKO4eFawvr9DSgbR3cGBZfN1+vpbWLOl6l07H2BrgNzmJt4DG+/Ej6eXd3ShT5yagbP/judjr0v0PqJXIa8lMnCPbNwqaSwcEr5cwbEvUu1RK7X6wkKCrIpDwwMBHDqFrnp0nx6jchm46oAPl90/07kutr5DFcOfO+Dxv1xVs2tTvz4moycms6AsZlqhyZU8lynvQxodYh1Pzfk4++aW8qLDOZk7VrJ9kbPtZI5gRcZrz9c1SsqhZJSDRv2hds54orl5m7O0u1jLqBzvdJK9KlcSssueeRk6Syt9vLUqFeddj0ukn7C7Yb17gUak30fzk61wduioiJ0OttlEG5u5l/A4uLi2zrf6hPv2yWuv8t06V3IXwQefeg+aSY9XnaeFsHdsrX0CwBM2f0YMTWTfyz4QuWIHIPp/JNgKmCrKUHtUCqc+XOyHzz60Kf7TPr2uPI5UQwHUHK2EDeoMa+OnmD1vIR/P4Fy4QtGdO/ByAFDbM6rGI+gZC8Bt/ZseudfFXwVE+16tsA6y2DbVgLqvIq22hNWx6qErQaSyHf5Cm2169+gVG/4PPAFlzQ3qKcZAq5N0Fb7r/2CF6pSLZG7u7tjNNrOpCxL4GUJ/VYNCXuBzFN6u8R2p4a8dI7YlzLB/SmiA/5EUfqrGg8AGse6kdha+gWPu5jfl8Vbj/BALQO9qqv4PjnQ+NjSHWeo1agKj2v73bzyXZIzwv5LQUd1/JnRHffxzf76TEusiqK8Y3Xcw9XI1ikuJCev54WPrvz/7FsxkcUJ7/NcZxj29m/8njbP5twvddvFwNYw6UM3vk2xPW5PP89YbNfz1Q8PAGqiPzIL07l/Wh3TH6sJBOCr6Y3pnKHc52ur/Unab0uAAPy016+H0gQMezGdq6BleS4haAP/VzHnLmPPsW0H+htwp1TrWg8MDCy3+1yvNyfj8rrdHdngCeYkvi3BH43fLBTFsRKoWvwDy1/20rT1JULDi0jZ73mXIxJq+keHXxjdcR8bD9RnWmKHcj8nhQYd3x8JpXlYOvWqnbeUK6Z8erVI4dR5P35Ps/37oHMpJTryT7IvebDrSGiFXkdFaB2di6d3KTvW+VOYf+VPc3ZmJXZv8qNG7SJCwgwUFWgxFNm+b8cOnOD7bypTs14RD9S6ThK/Vyh2ftymrKws4uPjiY2NpVmzZoSHh7Nnz55y627fvp2nnnqKJk2a0L59exYtWkRJie0cj7y8PF5//XVatmxJZGQkQ4cOveW5Yqq1yBs0aMCqVavIz8+3mvB28OBBy3Fn0WP4eYa+nElmmo4D33vz+D820LH3laUiF89XYv93Pjc4w71r3Kw0AoKNHPzBm8w0V5T8lby84BT/r+c5+Hr5AAAXuUlEQVRFCi9rWTbtAbVDVFWnPjkE1TDf7PhVKQHlEoNeNI8NZ6Xp2P7VtdtwOq9+jx7i2U6/kHHRm72pIURH/Gl1PPuyh2Vzl/e2PMrDtc+ycPhG1u6OIL9Ih5LzNIE++UxY1Y3yJrG1b3iCyp7FrPwuklKT8+2k6FO5lFH/SmfBKw/yYvd6dB2YQ4lBwzefVKXEqOGFGWcBOHvcjdeG1KZVdC4hYcW4e5o4/ocHmz+bilYLL845Y3PupA+rkp9nnkxXWqIhM03Hp/PN83dqNyqkZZe8u3eh94ATJ06wfPlyQkNDCQ8Pv+7eJ99++y1jxoyhZcuWvP766xw9epT33nuPCxcu8Prrr1vqmUwmRo8ezdGjRxkxYgT+/v58+umnxMbGsm7dOmrWrHnDeFRL5NHR0Xz44YckJCRY1pEbDAbWrVtH8+bNCQ52nkli9Zual8sE1zDy8rtnUHJfZvKiK8cP7va6bxP5/5Iq07nvBTr1uYBfQAnKpXjCI+E/q6uQsDgIfbqr2iGqquugHJq2vmoplZLH8MnmP6oHd3vdU4m8bBOX6pUv80bfnTbH952obknkaTl+/GN5DGO77GFY2wPoXEygieDFT5602p71aj1bHAZg/T7naQRcq9uQbHwDSkh4P4iVc6qh1ULDqHzi3svkoUfMvyf+QUaatb3EwR+82bnOn+IiLQFBRtoPaE3/kUuoWc92ftFXS4LITLvyWcs848bKOdUBeLx/jtMlcrW//eyhhx7ip59+wt/fn23btjFmzJhy682ZM4dGjRrxwQcf4OJivpHy8vJi2bJlxMbGUqtWLcC8HPvAgQO89957dO7cGYAnnniCrl27smjRIubMmXPDeFRL5E2bNiU6Opr4+Hj0ej01a9YkMTGR9PR0Zs2apVZYd2TuhJrMnXDljmmrKcGhxjnV9N0Gf77b4G/5eWvpF/wjxAHmDjiIV/rWtfr5Xv7deWNdR95Y1/GW65/U+zNpTbTl530rJrI39frj3uM+7v634nMUbbrl0qZb7nWPBwSV8MrC0zbl2mpJmM4tKPc5n+z9w27xCfD2vvla/WPHjnHs2DGmTZtmSeIATz/9NEuWLGHLli2MHj0agM2bNxMUFESnTp0s9QICAnjiiSf45ptvMBqN5U4OL6Nq/9OcOXOIjY0lKSmJGTNmUFJSwrJly4iKilIzLCGEEA7NnpvBVMxktz/+MN88NW7c2Ko8ODiYatWqWY6DeV+Vhx56CM01k5ObNGlCfn4+p0/b3rhdTdW9Q93c3Jg8eTKTJ09WMwwhhBDOxPTXw17nAjIyMigttd67wNfXF1/fO9tGumzidtneKFe7drK3Xq+nZcuWNvXKJn1nZWVRp06d676WbAIuhBDivjd48GDOnj1rVTZ27FjGjRt3R+crKjJ/WZarq+08IDc3NwoLC63qllevrKzsXNcjiVwIIYRTqYjJbmvWrCm3RX6n3N3NW3MbDLZLAYuLiy3Hy+qWV6+s7Oq65ZFELoQQ4r5XvXp1u56vrEu9vO3I9Xo9zZo1s6pb3r4qZWU321fF+RZbCiGEuL8p2PHbzyomxIYNGwJw6NAhq/LMzEzOnTtnOQ7mfVN+//13lGt6GZKTk/H09LzpOnJJ5EIIIZyLE3yNab169ahduzaff/65VZf92rVr0Wq1dOnSxVIWHR1NVlYW27dvt5Tl5OSwadMmOnXqdMOlZyBd60IIIcRte/998xd1paamApCUlMS+ffvw9fVlyBDzF/q88sorPP/884wcOZJu3bpx9OhR1qxZw4ABAwgLC7Ocq2vXrkRGRvLKK69YdnZbu3YtJpPplibbSSIXQgjhXCpg+dntWrDAevOdr776CoCQkBBLIu/QoQOLFi1i0aJFTJ8+nYCAAJ5//nleeOEFq+e6uLiwbNky5syZw6pVqyguLqZJkya89dZbhIbe/HsDJJELIYQQt+nIkSO3VK9z586WbVdvxM/PjzfffJM333zztmORRC6EEMKpaLDj8rOKmu12F8lkNyGEEMKJSYtcCCGEc7HnbPMKmrV+N0kiF0II4VwkkVuRrnUhhBDCiUmLXAghhHORFrkVaZELIYQQTkxa5EIIIZyLA2wI40gkkQshhHAudvwaU+laF0IIIYSqpEUuhBDCuchkNyvSIhdCCCGcmLTIhRBCOBcFMNmrRW6f06hJErkQQgjnIl3rVqRrXQghhHBi0iIXQgjhXKRFbkVa5EIIIYQTkxa5EEII5yItcivSIhdCCCGc2D3TIq8aEqB2CFaCQwPVDsFMo1E7AhsO896AQ96NO9L741bFV+0QrFR3pHhcQtSOwJYjxKStVvGvYVLst/zMXudRkUZRHPAvmRBCCHEdw9pMJ+vsBbucKyjEn5W7XrfLudQiXetCCCGEE7tnutaFEELcL+w42e0e2NpNWuRCCCGEE5MWuRBCCOdiwo6T3exzGjVJIhdCCOFcZB25FelaF0IIIZyYtMiFEEI4F2mRW5EWuRBCCOHEpEUuhBDCuUiL3IokciGEEM7FZDI/7HUuJydd63bw22+/MWbMGDp06EBERASPPfYYI0eOZP/+/WqH5hCysrKIj48nNjaWZs2aER4ezp49e9QOyyEkJyfzxhtv0K1bNyIjI2nfvj0TJkzg1KlTaofmkJYvX054eDi9evVSOxTVGQwG3n77bdq0aUNERAT9+/fnxx9/VDssoQJJ5HZw5swZSktL6devH6+//jojR44kJyeHIUOG8MMPP6gdnupOnDjB8uXLyczMJDw8XO1wHMqKFSvYunUrrVu3ZurUqfTv35+9e/cSExNDamqq2uE5FL1ez+LFi/H09FQ7FIcQFxfHypUr6dmzJ1OnTkWr1TJq1CgOHDigdmh3gXKle/3vPu6Bnd3kS1MqSGFhIZ07d6Zx48YsXbpU7XBUdfnyZYxGI/7+/mzbto0xY8bwySef8Oijj6odmur2799P48aNcXV1tZSdPHmSHj168OSTTzJ79mwVo3MscXFxpKenoygKeXl5JCUlqR2SapKTk+nXrx9Tpkxh+PDhABQXF9O9e3eCgoJYs2aNugFWsGEPv0bWmRy7nCvowQBW/jzDLudSi7TIK4iHhwcBAQHk5eWpHYrqvL298ff3VzsMh9S8eXOrJA5Qq1Yt6tWrJy3yqyQnJ7N+/XqmTJmidigOYdOmTeh0Ovr162cpc3Nzo2/fvuzbt4+srCwVo7sL7NUat+ekORVJIrejy5cvk5OTw/Hjx5k3bx5Hjx6lVatWaoclnIyiKJw/f15ufv6iKArTp08nJiaGhg0bqh2OQ0hJSSEsLAwvLy+r8oiICBRFISUlRaXIhBpk1rodvfrqq2zevBkAnU7HwIEDee6551SOSjib9evXk5mZyYQJE9QOxSF8/fXXHDt2jPfee0/tUByGXq8nODjYpjwwMBDg3m+Ry17rViSR29GYMWMYMGAA586dIykpCYPBgNFotOk6FeJ6UlNTmTZtGlFRUTIzG3Mv19y5cxk9ejRBQUFqh+MwioqK0Ol0NuVubm6Aebz8nqaYUBQ7ZWB7nUdF0rVuR+Hh4Tz22GP06dOHDz74gN9//13G9MQt0+v1PPvss/j5+bFgwQK0Wvl4Ll68GJ1OxzPPPKN2KA7F3d0do9FoU16WwMsSurg/SIu8guh0Ojp16sTixYspKirC3d1d7ZCEA7t06RKjRo3i0qVLrF271tJFej/Lyspi5cqVvPjii5w/f95SXlxcjNFoJC0tDR8fH/z8/FSMUh2BgYHldp/r9XqAe7/3wqTYsWtdJruJGygqKkJRFPLz89UORTiw4uJinnvuOU6ePMnSpUupXbu22iE5hOzsbIxGI/Hx8XTq1MnyOHjwIKmpqXTq1Inly5erHaYqGjRowIkTJ2z+thw8eNByXNw/pEVuBzk5OQQEBFiVXb58mc2bN1O9enWqVKmiUmTC0ZWWljJ+/Hh+/fVX3n//fSIjI9UOyWHUqFGj3Alu8+fPp6CggFdffZVatWrd/cAcQHR0NB9++CEJCQmWdeQGg4F169bRvHnzcifC3VNkr3UrksjtYPz48bi5udGsWTMCAwPJyMhg3bp1nDt3jnnz5qkdnkN4//33ASxro5OSkti3bx++vr4MGTJEzdBUNXv2bHbs2EGHDh24ePGi1SYnXl5edO7cWcXo1OXj41Pu9a9cuRIXF5f7+r1p2rQp0dHRxMfHo9frqVmzJomJiaSnpzNr1iy1w6t4ih33Wr8HJrtJIreDnj17kpSUxKpVq8jLy8PHx4fIyEjmzJnDI488onZ4DmHBggVWP3/11VcAhISE3NeJ/PDhwwDs3LmTnTt3Wh0LCQm5r5OVuLE5c+Ywf/58kpKSyM3NJTw8nGXLlhEVFaV2aOIuky1ahRBCOJVhjSeReTrbLucKrlmFlYfi7XIutchkNyGEEMKJSde6EEIIp6KYFBQ7jZEr98DyM0nkQgghnIvMWrciXetCCCGEE5MWuRBCCOciO7tZkRa5EEII4cSkRS6EEMLJmOy4kYvzbwgjLXIhrpGWlkZ4eDgLFy68YZkjiYuLIzw8/JbqduzYkdjY2Dt+rdjYWDp27HjHz7+R8PBw4uLiKuTcQtyrpEUuHMKePXsYOnSoVZmnpydhYWH06tWLIUOG4OLiolJ0f09aWhqJiYl07tyZhg0bqh2OEE7PvPzMPmPbd3Ieg8HAggULSEpKIi8vjwYNGjBhwgRatWpll5hulyRy4VC6d+9Ou3btUBSFrKwsEhMTmTlzJseOHWP69OmqxRUSEkJycvId3UycPXuWRYsWERISIolcCHtQFPt1rd/B8rO4uDi2bNnC0KFDCQ0NJTExkVGjRrFq1SqaNWtmn7hug3StC4fSqFEjevXqRUxMDKNHjyYhIYGgoCASEhKsvpP6WpcvX67QuDQaDW5ublSqJPe+QtzPkpOT2bhxI5MmTeKVV15hwIABrFy5kurVqxMfr85Wr5LIhUPz9vamWbNmKIrCmTNngCtjvH/88QcjR44kKiqKnj17Wp5z8uRJXn75Zdq0aUPjxo3p2LEjb731FgUFBTbn/+WXXxg4cCARERG0bt2aadOmlVvvRmPkmzdvJjY2lhYtWtC0aVO6du3KjBkzLF8rWTZkMGXKFMLDwwkPD7cao1YUhU8//ZTevXvTtGlTmjVrRmxsLD/99JPNaxUXF/PWW2/Rpk0bIiIi6Nu3L7t27br9N/Yau3btYvz48XTq1ImIiAhatGjBiBEj2Lt373Wfc+bMGZ5//nmioqJo3rw5Y8aMsfwfXe12rk+IW1HWtW6vx+3YtGkTOp2Ofv36Wcrc3Nzo27cv+/btIysry96Xe1PSvBAOTVEUTp06BYC/v7+lPD09nWHDhhEdHU2XLl0syffQoUMMGzYMX19fBgwYQHBwMIcPH2bVqlUcOHCAVatWodPpADh48CDPPPMMXl5ejBo1Ch8fH/7zn/8wefLkW47vnXfeYcmSJdStW5fhw4cTGBjI6dOn2bJlC//85z95+OGHee6551iyZAkDBgywfDNV1apVLed4+eWX2bhxI127dqV3794YDAY2bNjAiBEjWLhwIZ06dbLUnThxItu2baNDhw60bduW06dPM27cOGrUqHHnbzKQmJhIbm4uMTExVKtWjczMTMt3XX/yySe0aNHCqn5BQQGxsbFEREQwceJETp06xaeffsrBgwdJTEwkMDDwjq5PCEeXkpJCWFgYXl5eVuUREREoikJKSgpBQUF3NSZJ5MKhFBYWkpOTA0BWVharV6/m8OHDREZGUqtWLUu9tLQ0ZsyYYXVXDPDqq68SGBjIl19+ibe3t6W8VatWjB07lg0bNtC7d28AZs2ahaIorF27lrCwMACefvppnn766VuKNTk5mSVLlvDoo4+yfPly3NzcLMcmTZoEgK+vL61bt2bJkiVERkbSq1cvq3Ns3bqVDRs2MG3aNAYMGGApHzp0KP379+fNN9+kY8eOaDQadu3axbZt23jqqaeYPXu2pe7DDz/MmDFjbinm65k+fTqenp5WZQMHDuTJJ59k6dKlNon8woULDB06lKlTp1rFMXbsWBYuXMi0adNu+/qEuFVVQ/ztNkZeNcTcQMjIyKC0tNTqmK+vL76+vlZler2e4OBgm/OU3bxKi1zc9xYuXGjVfa3VaunYsaPNRLfKlStbEnKZI0eOcOTIEcaNG4fBYLDcEABERUXh6enJDz/8QO/evcnOzubAgQN07drVksQBXF1dGT58OC+99NJNY12/fj0AL730klUSB245Ma1fvx4vLy86d+5sFS+YhxAWLlzIyZMnCQsLY9u2bQCMHDnSql7nzp0JCwvjxIkTt/Sa5bk6iefn52MwGNBqtTRt2pSDBw+W+5zRo0db/fz4448TFhbG9u3bLYn8dq5PiFs1/3v7TnwtKiqiV69e5ObmWpWPHTuWcePG2dQt69W7WtnfgOLiYrvGdiskkQuHMmDAAKKjo9FoNHh4eFCrVi0qV65sU+/BBx+0mUGempoK2N4MXK1swlzZWG7t2rVt6tStW/eWYj116hQajYYGDRrcUv3ypKamkp+fT+vWra9bJzs7m7CwMM6cOYNWq7XqmShTp06dv5XIT58+zTvvvMOuXbvIy8uzOlbeTYmvr69V9/nVcWzbto2CggI8PT1v6/qEUEvZfJZrXdsaB3B3d8doNNqUlyXwa2/q7wZJ5MKhhIaG3vCPfhkPD4/rHhsxYgRt27Yt91h5H8y/Q6PR/K1uYUVRCAgIYO7cudetU69evTs+/63Iz89n8ODBFBYWMmzYMOrXr4+XlxdarZalS5f+rUlpjnB9QtxMeV3o1xMYGFhu97lerwe46+PjIIlc3ENCQ0MBc3f8zW4GyiaHHT9+3ObYsWPHbun1atWqxXfffcfhw4eJiIi4br0bJfrQ0FBOnjxJ06ZNbSbPXOvBBx/EZDJx8uRJm+RX1htxJ3788UeysrKYOXMmffr0sTo2f/78cp+Tl5eHXq+3aZWnpqZSpUoVS1f97VyfEM6gQYMGrFq1ivz8fKvf6bIhqL/TQ3enZPmZuGc0atSI+vXr89lnn5W7DKqkpISLFy8C5lnjkZGR7Nixw6pL2mAw8PHHH9/S6/Xo0QOAefPmYTAYbI4rf200UZbUrh1/A4iJicFkMjFv3rxyX+PqtfNls7s/+OADqzrbtm37W93qZUMUyjUbY+zateu64+MAy5Yts/p569atnDhxgs6dO1vKbuf6hHAG0dHRGI1GEhISLGVlXfPNmzcvdyJcRZMWubhnaDQa5syZw7Bhw+jZsyd9+vShbt26FBUVcerUKbZu3crEiRMtk+Ti4uKIjY1l0KBBDB482LL87NqZq9cTERHBqFGjWL58Ob179+aJJ54gMDCQtLQ0Nm/eTEJCAr6+vtStWxcvLy8+/fRT3N3d8fX1JSAggFatWhEdHU3v3r1ZvXo1v//+Ox06dMDf359z587x66+/curUKbZv3w5A27Zt6dChA4mJiVy8eJG2bdty5swZPv/8c+rXr8/Ro0fv6H2LiooiMDCQt956i7Nnz1KtWjVSUlJISkq67nn9/f3ZunUrWVlZPPLII5blZ1WrVmXs2LGWerdzfUI4g6ZNmxIdHU18fDx6vZ6aNWuSmJhIeno6s2bNUiUmSeTintKwYUMSExNZunQpO3bs4LPPPsPLy4uQkBCeeuopq72QmzVrxkcffcTcuXNZtmwZPj4+dO3alUGDBlla2zczadIkGjRowOrVq1mxYgWKolCtWjXatWuHu7s7YJ4c88477zB//nxmzpyJwWDgkUcescQya9YsHn30Ub744guWLl2K0WgkMDCQRo0a2cyenz9/PvPnz2fDhg3s3r2b+vXrs3DhQr755ps7TuS+vr6sWLGCt99+m9WrV1NSUkLjxo1Zvnw5X375Zbnn9fT0ZOXKlcycOZO5c+eiKApt27YlLi7OZozwdq5PCGcwZ84c5s+fT1JSErm5uYSHh7Ns2TLLPhF3m0a5tj9NCCGEEE5DxsiFEEIIJyaJXAghhHBiksiFEEIIJyaJXAghhHBiksiFEEIIJyaJXAghhHBiksiFEEIIJyaJXAghhHBiksiFEEIIJyaJXAghhHBi/x/4YN0uRelpvQAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Saving and loading the model"
      ],
      "metadata": {
        "id": "jYGBIz7qb5Py"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# save trained model\n",
        "torch.save(model.state_dict(), \"gpt2-text-classifier-model.pt\")"
      ],
      "metadata": {
        "id": "ojda1PNcbuoN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# load trained model\n",
        "model_new = SimpleGPT2SequenceClassifier(hidden_size=768, num_classes=5, max_seq_len=128, gpt_model_name=\"gpt2\")\n",
        "model_new.load_state_dict(torch.load(\"gpt2-text-classifier-model.pt\"))\n",
        "model_new.eval()"
      ],
      "metadata": {
        "id": "DkOUYVRUbuGT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Model Inference"
      ],
      "metadata": {
        "id": "ruEMXcq5cKZv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.text.iloc[5], df.label.iloc[5]"
      ],
      "metadata": {
        "id": "FbmWiVHXcTeX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "example_text = df.text.iloc[5]\n",
        "fixed_text = \" \".join(example_text.lower().split())\n",
        "print(fixed_text)\n",
        "\n",
        "tokenizer = GPT2Tokenizer.from_pretrained('gpt2')\n",
        "tokenizer.padding_side = \"left\"\n",
        "tokenizer.pad_token = tokenizer.eos_token\n",
        "\n",
        "model_input = tokenizer(fixed_text, padding='max_length', max_length=128, truncation=True, return_tensors=\"pt\")"
      ],
      "metadata": {
        "id": "da0ufn_ccGwO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "mask = model_input['attention_mask'].cpu()\n",
        "input_id = model_input[\"input_ids\"].squeeze(1).cpu()\n",
        "\n",
        "output = model_new(input_id, mask)\n",
        "print(\"output \", output)\n",
        "prob = torch.nn.functional.softmax(output, dim=1)[0]\n",
        "print(\"prob \", prob)\n",
        "pred_label = prob.argmax()\n",
        "print(\"pred_label \", pred_label)"
      ],
      "metadata": {
        "id": "YLg0MXancHUj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Models Comparison\n",
        "\n",
        "\\begin{array}{ccc}\n",
        "&Bert&Bert(clsImb)&DistillBert&Albert&Roberta&XLNet&GPT2\\\\\n",
        "Epochs&4&1&1&1&1&1&1\\\\\n",
        "BatchSize&32&8&8&8&8&8&2\\\\\n",
        "LearnRate&2*10^{-5}&4*10^{-5}&4*10^{-5}&4*10^{-5}&4*10^{-5}&4*10^{-5}&10^{-5}&\\\\\n",
        "F1ScoreEval&0.646&0.65&0.64&0.57&0.49&0.64&0.64\\\\\n",
        "F1ScoreTest&0.646&0.667&0.64&0.58&0.51&0.66&0.64\\\\\n",
        "TrainTime&5min&4min&2min&3min&4min&5min&14 min\n",
        "\\end{array}"
      ],
      "metadata": {
        "id": "7khnuPW-f010"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "From above table, we can say that, Bert(clsImb) and XLNet are best. I have trained for one epoch only just to avoid the colab memory and limited runtime limitation. The above results might varry for higher epochs and tunning of other parameters. "
      ],
      "metadata": {
        "id": "ikOPuE-mwD2A"
      }
    }
  ]
}