Sentiment Analysis using Bag of Words model_PyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem Scenario\n",
    "\n",
    "We need to train a model to recognise the emotion of the movie review as positive or negative. This is a binary classification problem. The dataset used here is known as 'Polarity Dataset'. This dataset has following properties:\n",
    "\n",
    "* It only comprises of English reviews.\n",
    "* The text is in lowercase.\n",
    "* There is whitespace around punctuations like periods, commas, and brackets.\n",
    "* Text has been split into one sentence per line.\n",
    "\n",
    "This problem is approached in three stages:\n",
    "\n",
    "&nbsp; &nbsp; **A. Data Preprocessing**\n",
    "\n",
    "&nbsp; &nbsp; **B. Bag Of Words Representation**\n",
    "\n",
    "&nbsp; &nbsp; **C. Sentiment Analysis Model**\n",
    "\n",
    "\n",
    "# A. DATA PREPROCESSING\n",
    "\n",
    "This stage involves the following steps:\n",
    "\n",
    "* Separation of data into training and test set.\n",
    "* Loading and cleaning the data.\n",
    "* Defining a vocabulary of preferred words.\n",
    "\n",
    "## A.1. Training and Testing set\n",
    "\n",
    "The dataset contains 1000 positive and 1000 negative reviews. Among them, 900 positive and 900 negative reviews are used as training sets and the remaining 100 positive and 100 negative reviews are used as test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_set_pos_path = \"Data/Training Data/pos/\"\n",
    "train_set_neg_path = \"Data/Training Data/neg/\"\n",
    "\n",
    "test_set_pos_path = \"Data/Test Data/pos/\"\n",
    "test_set_neg_path = \"Data/Test Data/neg/\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A.2. Load the file\n",
    "\n",
    "The **load_file** method helps to load the content of a file into memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_file(filepath):\n",
    "    file = open(filepath, 'r')  # open the file in the read only mode\n",
    "    text = file.read()          # read the contents of the file\n",
    "    file.close()                # close the file \n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A.2. Clean the File\n",
    "\n",
    "The **clean_file** method  helps to turn the contents of the file into clean tokens. The following methods are employed in this project to generate clean tokens.\n",
    "\n",
    "* Split on white space.\n",
    "* Remove all the punctuations.\n",
    "* Remove all words that does not purely consist of alphabetic characters.\n",
    "    (after this step, we are only left with alphabetic tokens)\n",
    "* Remove all stop words.\n",
    "* Remove all words with small length (length <= 1 character)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "import string\n",
    "\n",
    "def clean_file(file):\n",
    "    tokens = file.split()                                 # split into tokens on whitespace\n",
    "    \n",
    "    table = str.maketrans('' , '', string.punctuation)    # remove punctuation\n",
    "    tokens = [w.translate(table) for w in tokens]\n",
    "    \n",
    "    tokens = [word for word in tokens if word.isalpha()]  # remove non-alphabetic tokens\n",
    "    \n",
    "    set_of_stop_words = set(stopwords.words('english'))   # remove stop words\n",
    "    tokens = [word for word in tokens if not word in set_of_stop_words]\n",
    "    \n",
    "    tokens = [word for word in tokens if len(word) > 1]   # remove tokens of length <= 1\n",
    "    \n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "************************************************PRE CLEANING(ORIGINAL TEXT)**********************************************\n",
      "\n",
      "plot : derek zoolander is a male model . \n",
      "he is also very dumb and impressionable . \n",
      "for that reason , he is secretly hired and trained ( so secret , that even he doesn't know about it ) by an underground fashion syndicate to assassinate the prime minister of malaysia , who wants to abolish child labor in his country . \n",
      "will zoolander fulfill the dirty deed ? \n",
      "will zoolander ever grace the world his new \" look \" ? \n",
      "is this a funny movie ? \n",
      "find out below . . . \n",
      "critique : there's a place in our world for \" stupid comedies \" . \n",
      "films which don't pretend to take themselves seriously , are based on idiotic premises and filled with dumb jokes . \n",
      "there's also a time for them and many would argue that this difficult period of our history , might be an ideal circumstance during which to \" relax \" by watching something so completely frivolous . \n",
      "well , if you're in the mood for some seriously mindless entertainment , ben stiller and his cast of many have assembled one of the more original dumb comedies in some time . \n",
      "of course , comedies as such are usually very subjective and i could see how some will see this film simply as stupid and unfunny , while others will grant them the \" stupid \" , but consider it funny instead . \n",
      "i personally enjoyed it for the most part , cracked up loudly during a couple of specific sequences and loved the derek zoolander character and the unrestrained whipping they released upon the fashion industry . \n",
      "snap ! \n",
      "it was also nice to see several real-life models with small roles in the movie , not taking themselves too seriously . \n",
      "and if you're the type of person who likes celebrity cameos in films , well , don't look any further because dozens of famous faces show up here including vince vaughn , billy zane , winona ryder , christian slater , david duchovny , natalie portman and many , many others . \n",
      "i especially liked andy dick's complete make-over as the masseuse . \n",
      "hi-larious ! \n",
      "but with all films of this type , there is bound to be some stuff that simply doesn't work . \n",
      "a few particulars which didn't strike my fancy included the bulimia and \" orgy \" sequence with christine taylor , the break-dance fighting , and i also never get why they use real countries in plots like this ( why not just \" make up \" a country , instead of zeroing in on a certain people ? ) . \n",
      "i also could have done with less of the taylor character in general , since she wasn't all that interesting and seemed to slow things down every now and then ( more zoolander , dude ! ) . \n",
      "but those few missteps were nothing compared to some of the more memorable scenes which definitely did work for me ! \n",
      "i almost pissed myself during the \" gas station \" disaster , absolutely adored the \" walk-off \" contest ( with david bowie as the judge , no less ) and appreciated many of zoolander's moronic one-liners ( \" i was a merman . . . a \n",
      "merman ! ! \" ) . \n",
      "and i dare you to get his \" monkey \" photo shoot out of your head after seeing this movie ( \" you're a monkey , derek . . . a \n",
      "monkey ! \" ) \n",
      "i also really liked the soundtrack and the pace of the film , both of which zipped and zagged , and established a nice rhythm to it all . \n",
      "again , it's to note that this movie is dumb and not for everyone , but my guess is that if you laughed at the trailer , you will likely enjoy many of the quips in the actual picture as well . \n",
      "if you thought the trailer was dumb , skip this dodo bird and go rent austin powers again or something , a film from which there is an obvious influence here . \n",
      "blue steel , baby . . . yeah ! ! ! \n",
      "where's joblo coming from ? \n",
      "austin powers ( 7/10 ) - austin powers 2 ( 7/10 ) - deuce bigalow ( 7/10 ) - dude , where's my car ( 7/10 ) - freddy got fingered ( 5/10 ) - jay & silent bob strike back ( 8/10 ) - joe dirt ( 5/10 ) - meet the parents ( 8/10 ) - say it isn't so ( 3/10 ) \n",
      "\n",
      "*************************************************POST CLEANING(TOKENS)***************************************************\n",
      "\n",
      "['plot', 'derek', 'zoolander', 'male', 'model', 'also', 'dumb', 'impressionable', 'reason', 'secretly', 'hired', 'trained', 'secret', 'even', 'doesnt', 'know', 'underground', 'fashion', 'syndicate', 'assassinate', 'prime', 'minister', 'malaysia', 'wants', 'abolish', 'child', 'labor', 'country', 'zoolander', 'fulfill', 'dirty', 'deed', 'zoolander', 'ever', 'grace', 'world', 'new', 'look', 'funny', 'movie', 'find', 'critique', 'theres', 'place', 'world', 'stupid', 'comedies', 'films', 'dont', 'pretend', 'take', 'seriously', 'based', 'idiotic', 'premises', 'filled', 'dumb', 'jokes', 'theres', 'also', 'time', 'many', 'would', 'argue', 'difficult', 'period', 'history', 'might', 'ideal', 'circumstance', 'relax', 'watching', 'something', 'completely', 'frivolous', 'well', 'youre', 'mood', 'seriously', 'mindless', 'entertainment', 'ben', 'stiller', 'cast', 'many', 'assembled', 'one', 'original', 'dumb', 'comedies', 'time', 'course', 'comedies', 'usually', 'subjective', 'could', 'see', 'see', 'film', 'simply', 'stupid', 'unfunny', 'others', 'grant', 'stupid', 'consider', 'funny', 'instead', 'personally', 'enjoyed', 'part', 'cracked', 'loudly', 'couple', 'specific', 'sequences', 'loved', 'derek', 'zoolander', 'character', 'unrestrained', 'whipping', 'released', 'upon', 'fashion', 'industry', 'snap', 'also', 'nice', 'see', 'several', 'reallife', 'models', 'small', 'roles', 'movie', 'taking', 'seriously', 'youre', 'type', 'person', 'likes', 'celebrity', 'cameos', 'films', 'well', 'dont', 'look', 'dozens', 'famous', 'faces', 'show', 'including', 'vince', 'vaughn', 'billy', 'zane', 'winona', 'ryder', 'christian', 'slater', 'david', 'duchovny', 'natalie', 'portman', 'many', 'many', 'others', 'especially', 'liked', 'andy', 'dicks', 'complete', 'makeover', 'masseuse', 'hilarious', 'films', 'type', 'bound', 'stuff', 'simply', 'doesnt', 'work', 'particulars', 'didnt', 'strike', 'fancy', 'included', 'bulimia', 'orgy', 'sequence', 'christine', 'taylor', 'breakdance', 'fighting', 'also', 'never', 'get', 'use', 'real', 'countries', 'plots', 'like', 'make', 'country', 'instead', 'zeroing', 'certain', 'people', 'also', 'could', 'done', 'less', 'taylor', 'character', 'general', 'since', 'wasnt', 'interesting', 'seemed', 'slow', 'things', 'every', 'zoolander', 'dude', 'missteps', 'nothing', 'compared', 'memorable', 'scenes', 'definitely', 'work', 'almost', 'pissed', 'gas', 'station', 'disaster', 'absolutely', 'adored', 'walkoff', 'contest', 'david', 'bowie', 'judge', 'less', 'appreciated', 'many', 'zoolanders', 'moronic', 'oneliners', 'merman', 'merman', 'dare', 'get', 'monkey', 'photo', 'shoot', 'head', 'seeing', 'movie', 'youre', 'monkey', 'derek', 'monkey', 'also', 'really', 'liked', 'soundtrack', 'pace', 'film', 'zipped', 'zagged', 'established', 'nice', 'rhythm', 'note', 'movie', 'dumb', 'everyone', 'guess', 'laughed', 'trailer', 'likely', 'enjoy', 'many', 'quips', 'actual', 'picture', 'well', 'thought', 'trailer', 'dumb', 'skip', 'dodo', 'bird', 'go', 'rent', 'austin', 'powers', 'something', 'film', 'obvious', 'influence', 'blue', 'steel', 'baby', 'yeah', 'wheres', 'joblo', 'coming', 'austin', 'powers', 'austin', 'powers', 'deuce', 'bigalow', 'dude', 'wheres', 'car', 'freddy', 'got', 'fingered', 'jay', 'silent', 'bob', 'strike', 'back', 'joe', 'dirt', 'meet', 'parents', 'say', 'isnt']\n",
      "*************************************************************************************************************************\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# an example to understand the effects of load_file and clean_file methods\n",
    "\n",
    "filename = 'Data/Training Data/pos/cv034_29647.txt'\n",
    "text = load_file(filename)\n",
    "tokens = clean_file(text)\n",
    "\n",
    "print(\"\\n************************************************PRE CLEANING(ORIGINAL TEXT)**********************************************\\n\")\n",
    "print(text)\n",
    "print(\"*************************************************POST CLEANING(TOKENS)***************************************************\\n\")\n",
    "print(tokens)\n",
    "print(\"*************************************************************************************************************************\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A.3. Define a Vocabulary\n",
    "\n",
    "It is necessary to define a vocabulary of words to vectorise the document of reviews. The below method defines a vocabulary of words and maintains the count of the occurrence of each word in the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The length of the vocab:  44276\n",
      "\n",
      "Top 10 frequently occuring words: [('film', 7983), ('one', 4946), ('movie', 4826), ('like', 3201), ('even', 2262), ('good', 2080), ('time', 2041), ('story', 1907), ('films', 1873), ('would', 1844)]\n"
     ]
    }
   ],
   "source": [
    "from os import listdir\n",
    "from collections import Counter\n",
    "\n",
    "def add_words_to_vocab_and_update_count(directory, vocab):\n",
    "    for filename in listdir(directory):\n",
    "        filepath = directory + '/' + filename\n",
    "        text = load_file(filepath)  # load the file\n",
    "        tokens = clean_file(text)   # clean the file\n",
    "        vocab.update(tokens)        # update count of the word in the vocab\n",
    "    \n",
    "vocab = Counter()   # to hold tokens and their respective counts. Eg: [('tok1',tok1_count), ('tok2',tok2_count),...]\n",
    "\n",
    "add_words_to_vocab_and_update_count('Data/Training Data/pos', vocab)\n",
    "add_words_to_vocab_and_update_count('Data/Training Data/neg', vocab)\n",
    "\n",
    "print('The length of the vocab: ',len(vocab))\n",
    "print('\\nTop 10 frequently occuring words:',vocab.most_common(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Intuitively** we can say that most infrequently occuring words may not contribute much to the training of model. So only those words occuring more than two times are considered in the token list. \n",
    "\n",
    "Note: It is not a hard rule to use the threshold of two. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Tokens (Before):  44276\n",
      "Total Tokens (After) :  25767\n"
     ]
    }
   ],
   "source": [
    "min_occurrence = 2\n",
    "\n",
    "print('Total Tokens (Before): ',len(vocab))\n",
    "tokens = [token for token,count in vocab.items() if count >= min_occurrence]    # list of tokens with count >= 2\n",
    "print('Total Tokens (After) : ',len(tokens))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_list(lines, filename):\n",
    "    data = '\\n'.join(lines)\n",
    "    file = open(filename, 'w')\n",
    "    file.write(data)\n",
    "    file.close()\n",
    "\n",
    "save_list(tokens, 'vocab.txt')   # the vocabulary is saved in a text file for later use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# B. BAG OF WORDS REPRESENTATION\n",
    "\n",
    "This stage focuses on preparing the data for the training model. It involves following steps:\n",
    "\n",
    "* Converting reviews to lines of tokens.\n",
    "* Encoding reviews with a bag-of-words model representation.\n",
    "\n",
    "## B.1. Reviews to lines of tokens (Reviews -> Tokens)\n",
    "\n",
    "Here, each review file is loaded to memory and cleaned for tokens. The obtained tokens are further cleaned by retaining only those tokens that are also in vocabulary we defined previously. The resultant tokens are joined by whitespace. \n",
    "**reviews_to_lines** method returns the list of reviews in a specified location. For example, let's say we have 100 reviews in a folder. Then the **reviews_to_lines** method returns a list of length 100. Each index of the list stores a single review(cleaned review) with the words/tokens separated by whitespace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reviews_to_lines(directory, vocab):\n",
    "    lines = []\n",
    "    for filename in listdir(directory):\n",
    "        filepath = directory + filename\n",
    "        text = load_file(filepath)  # load the file\n",
    "        tokens = clean_file(text)   # clean the file\n",
    "        tokens = [word for word in tokens if word in vocab]   # filter by vocab\n",
    "        line = ' '.join(tokens)     # single review -> tokens -> filter -> single line with tokens spaced by whitespace\n",
    "        lines.append(line)          # list of reviews. Single review is stored at each index of the list\n",
    "    return lines\n",
    "\n",
    "# load the vocabulary\n",
    "vocab = load_file(\"vocab.txt\")\n",
    "vocab = vocab.split()\n",
    "vocab = set(vocab)\n",
    "\n",
    "# Training Data : reviews to lines\n",
    "train_pos_reviews_to_lines = reviews_to_lines(train_set_pos_path, vocab)\n",
    "train_neg_reviews_to_lines = reviews_to_lines(train_set_neg_path, vocab)\n",
    "\n",
    "# Test Data : reviews to lines\n",
    "test_pos_reviews_to_lines = reviews_to_lines(test_set_pos_path, vocab)\n",
    "test_neg_reviews_to_lines = reviews_to_lines(test_set_neg_path, vocab)\n",
    "\n",
    "# Total training and testing data\n",
    "train_reviews = train_pos_reviews_to_lines + train_neg_reviews_to_lines\n",
    "test_reviews  = test_pos_reviews_to_lines  + test_neg_reviews_to_lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## B.2. Reviews to Bag-Of-Words Vectors (Reviews -> Tokens -> Vectors)\n",
    "\n",
    "The data that is fed to the training model should be encoded into numerical values and all the training examples should be of uniform length. So far what we have is the training and test data in the texual form and of non-uniform length. We use the Bag-Of-Words model to encode the data in order to make it suitable for training/learning. In this model each document/review is transformed to an encoded vector where each word/token is assigned a score. The length of the vector corresponds to the length of the vocabulary. There are different methods for scoring the words like **freq, binary, count** and **tfidf**. In this project we use 'freq' for scoring the words. Let's understand how it works.\n",
    "\n",
    "**Example:** vocab = {this, that, is, mine, not, cat, dog} and text = \"This this is mine\". \n",
    "\n",
    "The encoded text using the 'freq' scoring method = [0.5, 0.0, 0.25, 0.25, 0.0, 0.0, 0.0]\n",
    "\n",
    "* the number of words/tokens in the text is 4.\n",
    "* the length of the vector = length of the vocabulary = 7\n",
    "* score of 'this' = (total occurence of 'this' in the text) / (total words in the text) = 2/4 = 0.5\n",
    "* the score of 'this' is stored in the index corresponding to 'this' in the vector( for now, we can assume it to be in the same index as in vocabulary).\n",
    "* the words 'that', 'not', 'cat', 'dog' do not occur in the text so their corresponding indexes in the vector is assigned a score of zero(because 0/4 = 0)\n",
    "\n",
    "We use **Tokenizer** class provided by Keras to achieve this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Shape of xtrain:  (1800, 25768)\n",
      " Shape of xtest :  (200, 25768)\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "\n",
    "def prepare_data(train_reviews, test_reviews, mode):\n",
    "    tokenizer = Tokenizer()\n",
    "    tokenizer.fit_on_texts(train_reviews) # fit the tokenizer on the texts\n",
    "\n",
    "    xtrain = tokenizer.texts_to_matrix(train_reviews, mode = mode)  # encode the training set\n",
    "    xtest  = tokenizer.texts_to_matrix(test_reviews, mode = mode)   # encode the test set\n",
    "\n",
    "    return xtrain, xtest\n",
    "\n",
    "xtrain, xtest = prepare_data(train_reviews, test_reviews, mode = 'freq')\n",
    "\n",
    "print(\" Shape of xtrain: \", xtrain.shape)\n",
    "print(\" Shape of xtest : \", xtest.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the shape of the xtrain and xtest we can understand the following:\n",
    "\n",
    "* each review is encoded in the vector of 25768 items/values.\n",
    "* there are total of 1800 training examples and 200 test examples.\n",
    "\n",
    "Now, we need the class labels for these reviews. 'Positive' is encoded as 0 and 'Negative' is encoded as 1. Thus, we create ytrain and ytest as shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "train_pos_limit = int(xtrain.shape[0]/2) # upper limit of pos training reviews\n",
    "train_neg_limit = xtrain.shape[0]        # upper limit of neg training reviews\n",
    "test_pos_limit  = int(xtest.shape[0]/2)  # upper limit of pos test reviews\n",
    "test_neg_limit  = xtest.shape[0]         # upper limit of neg test reviews \n",
    "\n",
    "ytrain = np.array([0 for i in range(0, train_pos_limit)] + [1 for i in range(train_pos_limit, train_neg_limit)])\n",
    "ytest  = np.array([0 for i in range(0, test_pos_limit)]  + [1 for i in range(test_pos_limit, test_neg_limit)])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# C. Sentiment Analysis Model (PyTorch)\n",
    "\n",
    "In this project we develop a multi-layered perceptron to predict the sentiment/emotion of the reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "\n",
    "# converting numpy arrays to tensors\n",
    "\n",
    "xtrain = torch.from_numpy(xtrain)\n",
    "xtrain = xtrain.type(torch.float)\n",
    "\n",
    "xtest = torch.from_numpy(xtest)\n",
    "xtest = xtest.type(torch.float)\n",
    "\n",
    "ytrain = torch.from_numpy(ytrain) # torch.Size([1800])\n",
    "ytrain = ytrain.view(-1,1)        # torch.Size([1800, 1])\n",
    "ytrain = ytrain.type(torch.float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_size = xtrain.size(1)\n",
    "hidden_size = 50\n",
    "output_size = 1\n",
    "num_epochs = 25\n",
    "learning_rate = 0.001"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "class Net(nn.Module):\n",
    "    \n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size)\n",
    "        self.relu = nn.ReLU()\n",
    "        self.fc2 = nn.Linear(hidden_size, output_size)\n",
    "        self.sigmoid = nn.Sigmoid()\n",
    "        \n",
    "    def forward(self, xtrain):\n",
    "        y = self.fc1(xtrain)\n",
    "        y = self.relu(y)\n",
    "        y = self.fc2(y)\n",
    "        y = self.sigmoid(y)     \n",
    "        return y\n",
    "    \n",
    "net = Net()\n",
    "opt = torch.optim.Adam(net.parameters(), lr=learning_rate)\n",
    "criterion = nn.BCELoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch = 0/25, loss = 0.052988581359386444\n",
      "epoch = 1/25, loss = 0.05649341270327568\n",
      "epoch = 2/25, loss = 0.04446496069431305\n",
      "epoch = 3/25, loss = 0.0415322482585907\n",
      "epoch = 4/25, loss = 0.041134681552648544\n",
      "epoch = 5/25, loss = 0.0424233116209507\n",
      "epoch = 6/25, loss = 0.04483966901898384\n",
      "epoch = 7/25, loss = 0.04829555004835129\n",
      "epoch = 8/25, loss = 0.05203211307525635\n",
      "epoch = 9/25, loss = 0.055232200771570206\n",
      "epoch = 10/25, loss = 0.05616191402077675\n",
      "epoch = 11/25, loss = 0.053085245192050934\n",
      "epoch = 12/25, loss = 0.045952897518873215\n",
      "epoch = 13/25, loss = 0.03638116270303726\n",
      "epoch = 14/25, loss = 0.026192672550678253\n",
      "epoch = 15/25, loss = 0.017530256882309914\n",
      "epoch = 16/25, loss = 0.010714988224208355\n",
      "epoch = 17/25, loss = 0.00620630057528615\n",
      "epoch = 18/25, loss = 0.0034002589527517557\n",
      "epoch = 19/25, loss = 0.0018232984002679586\n",
      "epoch = 20/25, loss = 0.0009501320309937\n",
      "epoch = 21/25, loss = 0.0004938868223689497\n",
      "epoch = 22/25, loss = 0.0002534114464651793\n",
      "epoch = 23/25, loss = 0.00012815819354727864\n",
      "epoch = 24/25, loss = 6.532882252940908e-05\n"
     ]
    }
   ],
   "source": [
    "cost = []\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    \n",
    "    for i, example in enumerate(xtrain):  \n",
    "        \n",
    "        example = example.reshape(-1, input_size)\n",
    "                \n",
    "        # Forward pass\n",
    "        outputs = net(example)\n",
    "        loss = criterion(outputs[0], ytrain[i])\n",
    "        \n",
    "        # Backward and optimize\n",
    "        opt.zero_grad()\n",
    "        loss.backward()\n",
    "        opt.step()\n",
    "      \n",
    "    print(\"epoch = {}/{}, loss = {}\".format(epoch, num_epochs, loss.item()))\n",
    "    cost.append(loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f248b56eef0>]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl81NW9//HXJztJIJCQsCSQhE1WWRICuFe0YtWCCopV3KhLlba33ntbbX+97bWbbW2tta6tWqBVi7gQV6rghqIQdiIgYQ9rIBDWQJI5vz8ycDGyTGCS7yzv5+ORx8x85wzz+T6GvOfkfL/fc8w5h4iIRI8YrwsQEZHmpeAXEYkyCn4RkSij4BcRiTIKfhGRKKPgFxGJMgp+EZEoo+AXEYkyCn4RkSgT53UBDbVt29bl5eV5XYaISFiZN2/edudcZiBtQy748/LyKCkp8boMEZGwYmbrAm2roR4RkSij4BcRiTIKfhGRKKPgFxGJMgp+EZEoo+AXEYkyCn4RkSgT1cG/umIvM5Zt9boMEZFmFbXBv6S8iqse/4TxE0uo2HPQ63LkFDnnWF2xl7lrK9H60SKBCbkrd5vDvHU7ufnZOSTE1n/vvbd8G9cM7uRxVRKoDZX7mb16B7NX7eCTVdvZurv+i/uawhx+OaofCXFR258RCUjUBf9nq3dw69/n0rZlIs/dNpQxj3/CO8u2KvhD2Nbd1cxe5Q/61dvZUHkAgLapCQztksFZXdtSvnM/j72/ig2VB3jihgLSkuM9rlokdEVV8M9auZ1vT5pLdusWPHfbUNq1SmJ4r3ZMnVdOdU0dSfGxXpcowM59h5i9ur43P3vVDlZV7AOgVVIcw7pmMP7sfM7q1pbuWamY2ZHXdctK5d6XlnDlYx/zzM2DyWub4tUuiIS0qAn+95Zv445/zKNL2xQmjx9CZstEAIb3ymLyp+uYvWoHX+uZ5XGV8u/SLXzvhQVU1/hISYhlcH461w7uxFld29KrQytiY+y4r71qUA6d0pO5fVIJox77mCdvKGBIl4xmrF4kPERF8E8v3cKE5+ZzRvuWTL51CG1SEo48N7RLBskJsby7bKuC32OTZ6/lZ8WlnJnTmp9e3oszc1oTH9u48frBeem8evfZ3PL3udzw9Gc8cNWZXF2Q0zQFi4SpiD8K9tqiTdz1z/n06ZjGP7899EuhD5AUH8u53dsyc/k2nRXiEZ/P8cBby/nptFIu7NmO528bSkFueqND/7DcjBRe+c7ZDM5L5z9fXMSD01fg8+mzFTksooP/pXnlfP+FBRR0bsM/vj2EtBbHPuA3vFc7NldVU7ppdzNXKAdr6/jBlIU88cEqbhjamSduGESLhNM/1pKWHM/EW4u4trATf3mvjO++sIDqmrogVCwS/iJ2qOf5Oev58StLOKtrBn+9sZDkhOPv6oU9szCDGcu20Tc7rRmrjG67q2u4Y9I8Zq/ewQ9HnMF3zu/6pYO1pys+NoYHru5Hl8wUHnh7ORt3HuCvNxYeOb4jEq0issc/8ZO13PfyEs7vkcnTNw0+YegDtE1NZECn1sxYrqt4m8vmqgNc88RsStZV8tC1/bnrgm5BDf3DzIw7zu/K49cXsHzLbkY9+jErtuwJ+vuIhJOIC/6nPlzFz4pLubh3O54cVxDwKZoX9WrH4vIqtu6ubuIKZfmW3Vz56Cds3HmAv99SxJUDm/7g64i+7XnxjrOoqfNx9eOf8P6KbU3+niKhKqKC/5EZK/n1m8u57MwOPHb9IBLjAh8rvqhXOwBmLlcgNKVPVm1nzOOzcTim3DmMs7u1bbb37peTxrQJZ9MpPZlb/z6XKXM3NNt7i4SSiAn+j1ZW8Id3vuCqgdk8fO2ARp8R0qNdKjltWmjStiY0beFGbnpmDh1aJ/HKXWfTq0OrZq+hQ1oLpt45jKFdMvhZcanmaZKoFDHBf063tjz6rUE8OKY/cadwGqCZcVGvdswq266zP4LMOccTH6zi+y8sZFDnNrx451l0bN3Cs3pSEuP45ai+HKyt48kPVnlWh4hXIib4zYzLzuxAzAmu7DyZ4b2yqK7x8XHZ9iBWFt3qfI6fFZfywFvLuaJ/RyaNLzruabXNqUtmKlcOzGHyp+vYpuM6EmUiJviDYUh+BqmJcby7TOP8wVBb5+N7zy9g0ux13HFeFx6+dkCjjrs0te8N70atz/HY++r1S3RR8B8lIS6G83q0ZebyrbqK9zTV+Rz3TFnEG0s285Nv9OK+b/Q6rb/GmkJuRgqjB+Xw3Jz1bK464HU5Is1Gwd/A8J7t2Lr7IEs36ireU+XzOX700mKKF23i3kt7ctt5Xbwu6bgmXNgNn8/x2Hvq9Uv0UPA38LWeWcQYvKOze06Jc47/N20pU+eV84OLenDn+V29LumEOqUnc83gTrwwdz0bd6nXL9EhoOA3sxFmtsLMyszs3mM8n2hm//I//5mZ5fm355nZATNb6P95IrjlB196SgKDOrfRaZ2nwDnH/a9/znOfreeuC7ryveHdvC4pIHd/rRuG8ZeZZV6XItIsThr8ZhYLPApcCvQGrjOz3g2ajQd2Oue6AQ8Bvz3quVXOuQH+nzuDVHeTGt6rHaWbdmvctxGcczzw9nKe/Xgt48/J578vOaNJpmBoCtmtWzC2qBMvlmxgQ+V+r8sRaXKB9PiLgDLn3Grn3CHgBWBkgzYjgYn++1OB4RYuv/XHcFGv+nn5Z+jsnoD96d2VPPnBam4Y2pn/d1mvsAn9w+66oBsxMer1S3QIJPizgaOvbS/3bztmG+dcLVAFHF76KN/MFpjZB2Z27mnW2yy6ZaXSOT1Zwz0BevS9Mh6esZJrCnO4/5t9wy70AdqnJXH9kM5MnV/Ouh37vC5HpEkFEvzH+i1ueK7j8dpsBjo75wYC9wDPmdlXrtM3s9vNrMTMSioqKgIoqWmZGcN7ZfHxqh3sP1TrdTkh7W8freb301cwakBHfnPVmSF3ymZjfOeCrsTHGn+eoV6/RLZAgr8c6HTU4xxg0/HamFkckAZUOucOOud2ADjn5gGrgB4N38A595RzrtA5V5iZmdn4vWgCF/Vqx6FaH7NW6ire45k8ey2/fGMZ3+jXngfH9D/herjhIKtlEuOG5vLKgnJWV+z1uhyRJhNI8M8FuptZvpklAGOB4gZtioGb/PdHAzOdc87MMv0HhzGzLkB3YHVwSm9ag/PSaZkYp3H+45gydwM/nVbKRb3a8fDYgac0P1IouuP8riTGxfLnGSu9LkWkyZz0t9U/Zj8BmA4sA6Y450rN7H4z+6a/2dNAhpmVUT+kc/iUz/OAxWa2iPqDvnc65yqDvRNNISEuhvPOyGTG8m1ar7WBVxds5EcvL+b8Hpk8ev3AU14bNxS1TU3kprPymLZoE2XbtGCLRKaAfmOdc28653o457o6537l3/Y/zrli//1q59wY51w351yRc261f/tLzrk+zrn+zrlBzrnXmm5Xgu+iXlls33uQxRurvC4lZLyxeDP3TFnI0PwMnhxXEFJz7wTL7ed1ITk+lj+9q16/RKbI6ao1gQt61F/Fq7N76r29dEv94vW5bXj65sKAVzcLN+kpCdx8dh5vLNnM8i2aukMij4L/BNqkJFCYm67ZOqkf3rn7ufmcmZPGMzeffB3jcHfbuV1ISYjjYfX6JQIp+E9ieK8slm3eHdXzuDw/Zz0/mLKQorx0Jo8fQssk7+fTb2qtkxO49Zx83lq6hdJNGuqTyKLgP4nhh9fijdLhnqdnreG+l5dwQY9Mnr1lMCmJkd3TP9r4c/JpmRSnsX6JOAr+k+iamUJeRnJUDvf8ZeZKfvH651zatz1PjovcMf3jSWsRz23nduGdz7eypFy9fokcCv6TqL+Ktx2zV+1g38HouIrXOcfv3l7Og/+uX7z+kesGkhAXnf9Vbjk7j7QW8Tz07hdelyISNNH529xIw3tlcajOx0crvZ9Ooqn5fI7/fe1zHnt/FdcP6XzKi9dHipZJ8dx+XhdmLt/GgvU7vS5HJCii9ze6EQbnpdMyKfLX4q3zOe59eTF//2Qt3z4nn1+O6hvWc+8Ey01n5ZGeksBDGuuXCKHgD0B8bAwXnJHFe8u3URehV/HW1Pn4j38tZEpJOd8f3p2fhOHUyk0lNTGOO87rwodfVFCyNiwuPBc5IQV/gC7qlcWOfYdYuGGX16UEXXVNHd/5x3xeW7SJ+y7tyQ8u7qHQb2DcsFxaJ8czcfY6r0sROW0K/gBd0COL2BiLuKt4Dxyq47ZJJby7bCu/GNmHO0J8jVyvJCfEMbJ/R6aXbqFqf43X5YicFgV/gNKS4ynMbRNRs3Xuqa7hpmfm8HHZdh4c059xw/K8LimkjS7oxKFaH68tbjgruUh4UfA3wkW92rFi656IWJd1Q+V+rv/bZ8xfv5M/XzeQ0QU5XpcU8vpmt6Jn+5a8OK/c61JETouCvxGG+9fifTeMh3t8Psfk2Wu55E8fsrpiH0+OK+DyMzt6XVZYMDNGF+SwaMMuVm7VlM0SvhT8jdAlM5We7Vvyh39/wTufh1/4H+7l/3RaKQW5bZj+g/OOTEkhgblyYDZxMaZev4Q1BX8jPXPzYPLbpnDbpBL+PGNlWCzS4vM5Jn+6jkv+9CFLNlbxm6v6MenWIrJbt/C6tLCTkZrIhT2zeHn+RmrqfF6XI3JKFPyN1LF1C168cxhXDszmj+98wd3PzQ/pqRyO9PJfXXqkl39dUWedrnkaRhfksH3vQT78IvKv5JbIFD1TLQZRUnwsf7ymP707tOI3by1jzfZ9PDWukM4ZyV6XdoTP5/jnnPX85s1lxJjxm6v6MXZwJwV+EHytZxZtUxN4saRcQ2USltTjP0Vmxm3ndeHvtxSxadcBvvnoLD4u2+51WUB9L/+Gp9XLbyrxsTGMGpDNjOVbqdx3yOtyRBpNwX+azuuRSfGEc8hMTeTGZ+bw9Kw1OOfNuP/RY/mLNuzi11dqLL+pjCnsRE2d49UFG70uRaTRFPxBkNc2hVfuPpvhPbP4xeuf818vLqa6pq5Za1i3Y9+RXv6gzvW9/G8NUS+/qZzRviVn5qTp7B4JSxrjD5LUxDieuKGAh2es5OEZKymr2MuTNxTQPi2pyd6zan8NbyzZzKsLNzJnTSUpCbH8+sp+XFeksfzmMKYgh59OK2Xpxir6Zqd5XY5IwBT8QRQTY/zg4h706tCKe6Ys5Iq/zOKJGwooyG0TtPeorqlj5vJtvLpgI++t2EZNnaNLZgr3XNyDMYU5dEjTsE5zuaJ/R37x+jKmzitX8EtYUfA3gRF925Pf9mxum1TCdU99yv0j+zC6IOeUFzSp8zk+Xb2DVxds5O2lW9hzsJbMloncOCyPUQOy6ZvdSj18D7ROTuDiPu2YtnAjP/5Gr6hdpUzCj3l1IPJ4CgsLXUlJiddlBMWu/Yf47vML+GjldmIMslom0T4tiQ5pR9+2qL9tlUS7VklHwsM5R+mm3UxbuJHiRZvYuvsgqYlxXNKnPaMGduSsrm2J1SIpnnt/xTZufnYuj18/iEv7dfC6HIliZjbPOVcYSFv1+JtQ6+QEnr15MK8v3szqir1srqpmy+5qVm7by4dfVLDv0FcPALdNTaRDWhL7DtWyumIf8bHG+T2y+OnlHbmoV7uoW/A81J3bPZP2rZJ4cV65gl/CRkDBb2YjgIeBWOBvzrkHGjyfCEwCCoAdwLXOubVHPd8Z+Bz4uXPuweCUHh7iYmMYNTD7mM/tqa5hS1U1m6qq2VJ1oP6LoaqazVXVtE6O59az87msXwfapCQ0c9USqNgY46pB2TzxwSq27a4mq1XTHcwXCZaTBr+ZxQKPAhcD5cBcMyt2zn1+VLPxwE7nXDczGwv8Frj2qOcfAt4KXtmRoWVSPC2T4unerqXXpchpGF2Qw2Pvr+KVBRu1kI2EhUCORhUBZc651c65Q8ALwMgGbUYCE/33pwLDzX+00cxGAauB0uCULBJaumSmUpDbhhfnlXt28Z5IYwQS/NnAhqMel/u3HbONc64WqAIyzCwF+BHwv6dfqkjoGlOQQ9m2vRG5JrNEnkCC/1injjTs1hyvzf8CDznn9p7wDcxuN7MSMyupqNCMhxJ+LjuzA0nxMbqSV8JCIMFfDnQ66nEO0HDR0SNtzCwOSAMqgSHA78xsLfAfwI/NbELDN3DOPeWcK3TOFWZmZjZ6J0S81jIpnm/07cBrizY1+3QdIo0VSPDPBbqbWb6ZJQBjgeIGbYqBm/z3RwMzXb1znXN5zrk84E/Ar51zfwlS7SIhZXRhDnuqa5leusXrUkRO6KTB7x+znwBMB5YBU5xzpWZ2v5l909/saerH9MuAe4B7m6pgkVA1ND+DnDYtmKrhHglxAZ3H75x7E3izwbb/Oep+NTDmJP/Gz0+hPpGwERNjXD0ohz/PXMnGXQc0HbaELE0uIhJEowtycA5eVq9fQpiCXySIOqUnM6xLBlPn65x+CV0KfpEgG1OYw7od+5mzptLrUkSOScEvEmQj+rYnNTFOB3klZCn4RYIsOSGOy/p14I0lm9l3sNbrckS+QsEv0gTGFOaw/1Adby7Z7HUpIl+h4BdpAgW5bejSNkVTOEhIUvCLNAEz4+qCHOasqWT9jv1elyPyJQp+kSZyeAGeaQs3elyJyJcp+EWaSHbrFhTlp/Pqwo06p19CioJfpAmNGpDNqop9lG7a7XUpIkco+EWa0Df6tSc+1nh1gYZ7JHQo+EWaUOvkBC44I4viRZuo82m4R0KDgl+kiY0akM22PQf5dPUOr0sRART8Ik1ueK8sUhPjNNwjIUPBL9LEkuJjGdG3PW8v3aJlGSUkKPhFmsGoAdnsOVjLzOXbvC5FRMEv0hyGdc0gq2WihnskJCj4RZpBbIxxRf+OvL+igqr9NV6XI1FOwS/STEYNyOZQnY83l2rGTvGWgl+kmfTNbkWXzBQN94jnFPwizcTMGDUgm8/WVLJp1wGvy5EopuAXaUYjB3QEoHjRJo8rkWim4BdpRrkZKQzs3FrDPeIpBb9IMxs1IJvlW/awYsser0uRKKXgF2lml53ZgdgY41Ut0CIeCSj4zWyEma0wszIzu/cYzyea2b/8z39mZnn+7UVmttD/s8jMrgxu+SLhp21qIud2b0vxwk34NGOneOCkwW9mscCjwKVAb+A6M+vdoNl4YKdzrhvwEPBb//alQKFzbgAwAnjSzOKCVbxIuBo1IJuNuw5Qsm6n16VIFAqkx18ElDnnVjvnDgEvACMbtBkJTPTfnwoMNzNzzu13ztX6tycB6t6IABf3bkeL+FgN94gnAgn+bGDDUY/L/duO2cYf9FVABoCZDTGzUmAJcOdRXwQiUSslMY6v92nHm0s2c6jW53U5EmUCCX47xraGPffjtnHOfeac6wMMBu4zs6SvvIHZ7WZWYmYlFRUVAZQkEv5GDchm1/4aPvhC/+eleQUS/OVAp6Me5wANrz450sY/hp8GVB7dwDm3DNgH9G34Bs65p5xzhc65wszMzMCrFwlj53RvS3pKgoZ7pNkFEvxzge5mlm9mCcBYoLhBm2LgJv/90cBM55zzvyYOwMxygTOAtUGpXCTMxcfGcPmZHXj3863sqdaMndJ8Thr8/jH5CcB0YBkwxTlXamb3m9k3/c2eBjLMrAy4Bzh8yuc5wCIzWwi8AtzlnNse7J0QCVcjB2RzsNbH9NKtXpciUcScC60TbQoLC11JSYnXZYg0C+cc5//+fXIzkpk8fojX5UgYM7N5zrnCQNrqyl0RD5kZIwd05OOy7WzbU+11ORIlFPwiHhs5IBufg9cWaYEWaR4KfhGPdctKpW92K6bp7B5pJgp+kRAwakA2i8urWF2x1+tSJAoo+EVCwBX9O2IGry7UAi3S9BT8IiGgXaskzuqawbSFGwm1M+0k8ij4RULEyP7ZrNuxn4UbdnldikQ4Bb9IiBjRrz0JcTFM03CPNDEFv0iIaJUUz0W9spi2cCPVNXVelyMRTMEvEkKuH5LLzv01vLFY5/RL01Hwi4SQs7pm0DUzhUmfrvO6FIlgCn6REGJmjBuay6INu1hcroO80jQU/CIh5qqCHJITYpk8W71+aRoKfpEQ0yopnisHZlO8aBM79x3yuhyJQAp+kRB047A8Dtb6eHHehpM3FmkkBb9ICDqjfUuK8tP5x6fr8fl0Ja8El4JfJETdOCyX9ZX7tRi7BJ2CXyREXdKnPVktE5k0e63XpUiEUfCLhKj42BiuK+rM+19UsH7Hfq/LkQii4BcJYd8a0plYM/7xmU7tlOBR8IuEsHatkrikT3umlGzQ/D0SNAp+kRA3blguu/bXULxIs3ZKcCj4RULckPx0erRLZfLsdVqkRYJCwS8S4g7P37NkY5UWaZGgUPCLhIErB+WQmhin+XskKBT8ImEgNTGOqwZl8/rizezYe9DrciTMBRT8ZjbCzFaYWZmZ3XuM5xPN7F/+5z8zszz/9ovNbJ6ZLfHfXhjc8kWix7ihuRyq8/GvEs3fI6fnpMFvZrHAo8ClQG/gOjPr3aDZeGCnc64b8BDwW//27cAVzrl+wE3A5GAVLhJturdrybAuGfzz0/XUaf4eOQ2B9PiLgDLn3Grn3CHgBWBkgzYjgYn++1OB4WZmzrkFzrnD56CVAklmlhiMwkWi0Y3Dctm46wAzl2/zuhQJY4EEfzZw9N+W5f5tx2zjnKsFqoCMBm2uBhY45zRAKXKKLu7djvatkjR/j5yWQILfjrGt4d+ZJ2xjZn2oH/6545hvYHa7mZWYWUlFhWYiFDmeuNgYvjWkMx+t3M7qir1elyNhKpDgLwc6HfU4B2h4CeGRNmYWB6QBlf7HOcArwI3OuVXHegPn3FPOuULnXGFmZmbj9kAkyowt6kR8rPGPT9d7XYqEqUCCfy7Q3czyzSwBGAsUN2hTTP3BW4DRwEznnDOz1sAbwH3OuY+DVbRINMtqmcSIvh14cd4G9h+q9bocCUMnDX7/mP0EYDqwDJjinCs1s/vN7Jv+Zk8DGWZWBtwDHD7lcwLQDfipmS30/2QFfS9EosyNw3LZU13LtIWav0caz0Jt7o/CwkJXUlLidRkiIc05x6UPf4SZ8eb3zsHsWIfZJJqY2TznXGEgbXXlrkgYMjNuHJbHss27mbdup9flSJhR8IuEqVEDO9IyKY5Jmr9HGknBLxKmkhPiGF2Qw1tLN1OxR5fHSOAU/CJhbNzQXGrqHJM/Va9fAqfgFwljXTJTubRve56ZtUazdkrAFPwiYe4/v34G+w/V8tj7x7w+UuQrFPwiYa5bVipjCjoxefY6Nu464HU5EgYU/CIR4PsXdQeDP73zhdelSBhQ8ItEgI6tW3DTsFxeml/Oyq17vC5HQpyCXyRC3HVBN1IS4njw3yu8LkVCnIJfJEK0SUng9vO6ML10KwvW62peOT4Fv0gEufWcfNqmJvDbt5cTavNwSehQ8ItEkJTEOCZ8rRufrq7ko5XbvS5HQpSCXyTCXDekMzltWvC76cvxaVF2OQYFv0iESYyL5Z6Le7B0427eXLrZ63IkBCn4RSLQyAHZnNGuJX/49xfU1Pm8LkdCjIJfJALFxhj/fckZrNm+jxdLyr0uR0KMgl8kQg3vlUVBbhsenvEFBw7VeV2OhBAFv0iEMjN+NKInW3cfZOLstV6XIyFEwS8SwYry0/naGZk89l4ZVftrvC5HQoSCXyTC/fclPdldXcuTH2raZqmn4BeJcL07tmLkgI488/Eatu2u9rocCQEKfpEocM/FPaitc/x55kqvS5EQoOAXiQK5GSlcV9SZF+ZsYO32fV6XIx5T8ItEie9e2I342Bj+qMVaop6CXyRKZLVK4tZz8ihetInSTVVelyMeCij4zWyEma0wszIzu/cYzyea2b/8z39mZnn+7Rlm9p6Z7TWzvwS3dBFprNvP60pai3h+P12LtUSzkwa/mcUCjwKXAr2B68ysd4Nm44GdzrluwEPAb/3bq4GfAv8VtIpF5JSltYjnrgu68v6KCt5YrAncolUgPf4ioMw5t9o5dwh4ARjZoM1IYKL//lRguJmZc26fc24W9V8AIhICbjk7n0GdW/NfLy5i+ZbdXpcjHggk+LOBDUc9LvdvO2Yb51wtUAVkBKNAEQmuhLgYnrihgJZJcdw+aR679h/yuiRpZoEEvx1jW8PVHQJpc/w3MLvdzErMrKSioiLQl4nIKcpqlcQT4wrYUlXNd59fQJ0WbIkqgQR/OdDpqMc5wKbjtTGzOCANqAy0COfcU865QudcYWZmZqAvE5HTMKhzG+4f2YePVm7nd9OXe12ONKNAgn8u0N3M8s0sARgLFDdoUwzc5L8/GpjptNKzSMgbW9SZG4Z25skPVlO8qGF/TiJV3MkaOOdqzWwCMB2IBZ5xzpWa2f1AiXOuGHgamGxmZdT39Mcefr2ZrQVaAQlmNgr4unPu8+Dvioiciv+5vA8rtuzhh1MX0TUzhT4d07wuSZqYhVrHvLCw0JWUlHhdhkhUqdhzkCsemUVcrFE84RzSUxK8LkkayczmOecKA2mrK3dFhMyWiTw5roBtew4y4bn51Gqd3oim4BcRAPp3as2vRvXlk1U7eOAtHeyNZCcd4xeR6DGmsBOlm3bzt1lr6JPdiisH5nhdkjQB9fhF5Et+clkvhuSnc+9LS1i6UZO5RSIFv4h8SXxsDI9eP4iMlATumDyPHXsPel2SBJmCX0S+om1qIk+OK2T73oPc/dx8anSwN6Io+EXkmPrlpPHA1f34dHUlv3pjmdflSBDp4K6IHNeVA3NYunE3T89aQ5+OrRhT2OnkL5KQpx6/iJzQfZf25OxuGfzwpcU8MmMlPk3oFvYU/CJyQnGxMfz1xkJGDcjmD+98wfiJczWVc5hT8IvISSUnxPHHa/rzi1F9mVW2ncsfmaVTPcOYgl9EAmJmjBuay5Q7hlHnc1z1+Cf8a+56r8uSU6DgF5FGGdi5Da9/9xyK8tL50UtL+OHURVTX1HldljSCgl9EGi0jNZGJtxbx3Qu7MaWknKsf/4T1O/Z7XZYESMEvIqckNsb4z6+fwTM3F7Khcj+XP/IRM5Zt9bosCYCCX0ROy4U92/H6d88lp00y4yeW8Id/r9AaviFOwS8ip61zRjIv33UWYwqTL7D1AAAHc0lEQVRyeGRmGTc9M0dz/IQwBb+IBEVSfCy/H9OfB67qx5y1lVz+yCxmLNuqC75CkIJfRIJqbFFnXrrzLOJjYxg/sYSL/vgBk2avZd/BWq9LEz+tuSsiTeJQrY+3lm7mmVlrWFReRcukOMYO7sSNw/LolJ7sdXkRpzFr7ir4RaRJOeeYv34Xz368hreWbsE5x9d7t+eWs/Moyk/HzLwuMSI0Jvg1O6eINCkzoyC3DQW5bdi06wCTP13Hc5+t5+3SLfTp2Ipbzs7niv4dSIyL9brUqKEev4g0uwOH6nhlwUae/XgNK7ftpW1qAtcPyeWGoblktkz0urywpKEeEQkLzjlmlW3n2Y/XMnP5NuJijH45aRTlpzMkP52C3HTSWsR7XWZYUPCLSNhZXbGXqfPK+WxNJYvLd1FT5zCDXu1bUZSfTlF+OoPz0vUXwXEo+EUkrB04VMeCDTuZs6aSOWsqmb9+J9U19ev+dslMoSgv/ciXQU4bnSEETRD8ZjYCeBiIBf7mnHugwfOJwCSgANgBXOucW+t/7j5gPFAHfM85N/1E76XgF5GGDtX6WLqpijlrKpm7ppI5ayvZU11/XUDb1EQ6p7cgp00yOW3qbzv5H3dsnRQ1B42DGvxmFgt8AVwMlANzgeucc58f1eYu4Ezn3J1mNha40jl3rZn1Bp4HioCOwLtAD+fccedwVfCLyMnU+RwrtuxhzpodlG7azcZdByjfeYBNuw5Qe9SVwmaQ1TKRTkd9KeS0aUFmy0TSWsQf+WnVIp6k+PD+ggj26ZxFQJlzbrX/H38BGAl8flSbkcDP/fenAn+x+pNzRwIvOOcOAmvMrMz/780OpDgRkWOJjTF6d2xF746tvrS9ts7H1j0HKa/cT/nO+i+DDTv3U75zPyXrdvLa4s3HnUAuMS7mS18GR38ptGoRT4v4WJLiY0g6fBsXS+KR26OfiyUxLobEuBjiY2OIjTHiYiykrlcIJPizgQ1HPS4HhhyvjXOu1syqgAz/9k8bvDb7lKsVETmBuNgYslu3ILt1i6+EFNR/MWyuqqZy3yGqDtR86Wf3gRp27f+/x5urqlm+ZQ+7D9SwJwjTTRz+AoiLMeJiY/y3RlxMDHGxRmyMMbxnFj+5rPdpv9fJBBL8x/qaaviVebw2gbwWM7sduB2gc+fOAZQkItJ4cbExdEpPbvSUEXU+x8HaOqprfEduq2vq/D++Bs/V3z9U66PG56OuzlHrc9T6fPW3dY46n6Omzue/ddT5fNT4HO3TWjTRnn9ZIMFfDnQ66nEOsOk4bcrNLA5IAyoDfC3OuaeAp6B+jD/Q4kVEmkNsjJGcEEdygteVBEcgs3POBbqbWb6ZJQBjgeIGbYqBm/z3RwMzXf1R42JgrJklmlk+0B2YE5zSRUTkVJy0x+8fs58ATKf+dM5nnHOlZnY/UOKcKwaeBib7D95WUv/lgL/dFOoPBNcCd5/ojB4REWl6uoBLRCQCNOZ0Ti3EIiISZRT8IiJRRsEvIhJlFPwiIlFGwS8iEmVC7qweM6sA1p3GP9EW2B6kcsKN9j16RfP+R/O+w//tf65zLjOQF4Rc8J8uMysJ9JSmSKN9j859h+je/2jedzi1/ddQj4hIlFHwi4hEmUgM/qe8LsBD2vfoFc37H837Dqew/xE3xi8iIicWiT1+ERE5gYgJfjMbYWYrzKzMzO71up7mZmZrzWyJmS00s4ie5c7MnjGzbWa29Kht6Wb2jpmt9N+28bLGpnSc/f+5mW30f/4LzewbXtbYVMysk5m9Z2bLzKzUzL7v3x7xn/8J9r3Rn31EDPUEsiB8pDOztUChcy7iz2c2s/OAvcAk51xf/7bfAZXOuQf8X/xtnHM/8rLOpnKc/f85sNc596CXtTU1M+sAdHDOzTezlsA8YBRwMxH++Z9g36+hkZ99pPT4jywI75w7BBxeEF4ikHPuQ+rXfTjaSGCi//5E6n8hItJx9j8qOOc2O+fm++/vAZZRv453xH/+J9j3RouU4D/WgvDRtqi7A/5tZvP8axhHm3bOuc1Q/wsCZHlcjxcmmNli/1BQxA11NGRmecBA4DOi7PNvsO/QyM8+UoI/oEXdI9zZzrlBwKXA3f7hAIkejwNdgQHAZuAP3pbTtMwsFXgJ+A/n3G6v62lOx9j3Rn/2kRL8AS3qHsmcc5v8t9uAV6gf/oomW/1joIfHQrd5XE+zcs5tdc7VOed8wF+J4M/fzOKpD75/Oude9m+Ois//WPt+Kp99pAR/IAvCRywzS/Ef7MHMUoCvA0tP/KqIUwzc5L9/EzDNw1qa3eHQ87uSCP38zcyoX+N7mXPuj0c9FfGf//H2/VQ++4g4qwfAfwrTn/i/BeF/5XFJzcbMulDfyweIA56L5P03s+eBC6iflXAr8DPgVWAK0BlYD4xxzkXkAdDj7P8F1P+p74C1wB2Hx7wjiZmdA3wELAF8/s0/pn6sO6I//xPs+3U08rOPmOAXEZHARMpQj4iIBEjBLyISZRT8IiJRRsEvIhJlFPwiIlFGwS8iEmUU/CIiUUbBLyISZf4/YmSrezyjFIUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "plt.plot(cost)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}