Model_1.ipynb

{"metadata":{"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":31436,"sourceType":"datasetVersion","datasetId":19447}],"dockerImageVersionId":19668,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Beginners Guide to Text Generation using LSTMs\n\nText Generation is a type of Language Modelling problem. Language Modelling is the core problem for a number of of natural language processing tasks such as speech to text, conversational system, and text summarization. A trained language model learns the likelihood of occurrence of a word based on the previous sequence of words used in the text. Language models can be operated at character level, n-gram level, sentence level or even paragraph level. In this notebook, I will explain how to create a language model for generating natural language text by implement and training state-of-the-art Recurrent Neural Network. \n\n### Generating News headlines \n\nIn this kernel, I will be using the dataset of [New York Times Comments and Headlines](https://www.kaggle.com/aashita/nyt-comments) to train a text generation language model which can be used to generate News Headlines\n\n\n## 1. Import the libraries\n\nAs the first step, we need to import the required libraries:","metadata":{"_uuid":"20c011dd401be7b6448c43f965e5d0bf548c53b9","_cell_guid":"e084e610-8128-4769-ab64-6aa194044892"}},{"cell_type":"code","source":"# keras module for building LSTM \nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Embedding, LSTM, Dense, Dropout\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.callbacks import EarlyStopping\nfrom keras.models import Sequential\nimport keras.utils as ku \n\n# set seeds for reproducability\nfrom tensorflow import set_random_seed\nfrom numpy.random import seed\nset_random_seed(2)\nseed(1)\n\nimport pandas as pd\nimport numpy as np\nimport string, os \n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nwarnings.simplefilter(action='ignore', category=FutureWarning)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Load the dataset\n\nLoad the dataset of news headlines","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0"}},{"cell_type":"code","source":"curr_dir = '../input/'\nall_headlines = []\nfor filename in os.listdir(curr_dir):\n    if 'Articles' in filename:\n        article_df = pd.read_csv(curr_dir + filename)\n        all_headlines.extend(list(article_df.headline.values))\n        break\n\nall_headlines = [h for h in all_headlines if h != \"Unknown\"]\nlen(all_headlines)","metadata":{"_uuid":"87836e3adbe046dd0db62013491ba62bae93b6be","_cell_guid":"b8ef1429-ff19-4a6c-92d7-af8cc61c55f7","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Dataset preparation\n\n### 3.1 Dataset cleaning \n\nIn dataset preparation step, we will first perform text cleaning of the data which includes removal of punctuations and lower casing all the words. ","metadata":{"_uuid":"fda5d4868631d3618d4d9a9a863541b2faf121c0","_cell_guid":"9dbd8bc9-fb61-43b9-b0c4-98bd7f3f8150"}},{"cell_type":"code","source":"def clean_text(txt):\n    txt = \"\".join(v for v in txt if v not in string.punctuation).lower()\n    txt = txt.encode(\"utf8\").decode(\"ascii\",'ignore')\n    return txt \n\ncorpus = [clean_text(x) for x in all_headlines]\ncorpus[:10]","metadata":{"_uuid":"2a07365a27a7ba2f92fc9ba4d05d8e6254a68d8c","_cell_guid":"b8bf84ed-da11-4f89-a584-9dceea677420","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 3.2 Generating Sequence of N-gram Tokens\n\nLanguage modelling requires a sequence input data, as given a sequence (of words/tokens) the aim is the predict next word/token.  \n\nThe next step is Tokenization. Tokenization is a process of extracting tokens (terms / words) from a corpus. Python’s library Keras has inbuilt model for tokenization which can be used to obtain the tokens and their index in the corpus. After this step, every text document in the dataset is converted into sequence of tokens. \n","metadata":{"_uuid":"6fd11859fd71aa5c7ce10bdbbd31c8eb6d1b3118","_cell_guid":"9d83cc08-19ba-4b00-9ca6-dcf5ff39c8af"}},{"cell_type":"code","source":"tokenizer = Tokenizer()\n\ndef get_sequence_of_tokens(corpus):\n    ## tokenization\n    tokenizer.fit_on_texts(corpus)\n    total_words = len(tokenizer.word_index) + 1\n    \n    ## convert data to sequence of tokens \n    input_sequences = []\n    for line in corpus:\n        token_list = tokenizer.texts_to_sequences([line])[0]\n        for i in range(1, len(token_list)):\n            n_gram_sequence = token_list[:i+1]\n            input_sequences.append(n_gram_sequence)\n    return input_sequences, total_words\n\ninp_sequences, total_words = get_sequence_of_tokens(corpus)\ninp_sequences[:10]","metadata":{"_uuid":"9129a8b773feb72eff91aa0025157a173d10c625","_cell_guid":"896543c9-7944-4748-b8ef-ef8cbc2a84f0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the above output [30, 507], [30, 507, 11], [30, 507, 11, 1] and so on represents the ngram phrases generated from the input data. where every integer corresponds to the index of a particular word in the complete vocabulary of words present in the text. For example\n\n**Headline:** i stand  with the shedevils  \n**Ngrams:** | **Sequence of Tokens**\n\n<table>\n<tr><td>Ngram </td><td> Sequence of Tokens</td></tr>\n<tr> <td>i stand </td><td> [30, 507] </td></tr>\n<tr> <td>i stand with </td><td> [30, 507, 11] </td></tr>\n<tr> <td>i stand with the </td><td> [30, 507, 11, 1] </td></tr>\n<tr> <td>i stand with the shedevils </td><td> [30, 507, 11, 1, 975] </td></tr>\n</table>\n\n\n\n### 3.3 Padding the Sequences and obtain Variables : Predictors and Target\n\nNow that we have generated a data-set which contains sequence of tokens, it is possible that different sequences have different lengths. Before starting training the model, we need to pad the sequences and make their lengths equal. We can use pad_sequence function of Kears for this purpose. To input this data into a learning model, we need to create predictors and label. We will create N-grams sequence as predictors and the next word of the N-gram as label. For example:\n\n\nHeadline:  they are learning data science\n\n<table>\n<tr><td>PREDICTORS </td> <td>           LABEL </td></tr>\n<tr><td>they                   </td> <td>  are</td></tr>\n<tr><td>they are               </td> <td>  learning</td></tr>\n<tr><td>they are learning      </td> <td>  data</td></tr>\n<tr><td>they are learning data </td> <td>  science</td></tr>\n</table>","metadata":{"_uuid":"f22aa5e0c04620ca5034ab9389322eee543060c6","_cell_guid":"a22c88f5-f2a3-457c-835b-63341e657e3f"}},{"cell_type":"code","source":"def generate_padded_sequences(input_sequences):\n    max_sequence_len = max([len(x) for x in input_sequences])\n    input_sequences = np.array(pad_sequences(input_sequences, maxlen=max_sequence_len, padding='pre'))\n    \n    predictors, label = input_sequences[:,:-1],input_sequences[:,-1]\n    label = ku.to_categorical(label, num_classes=total_words)\n    return predictors, label, max_sequence_len\n\npredictors, label, max_sequence_len = generate_padded_sequences(inp_sequences)","metadata":{"_uuid":"ca588b414e70e21bebcead960f6632805d37dd8c","_cell_guid":"73254551-40bd-45b1-a7a5-88fe4cbe0b20","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Perfect, now we can obtain the input vector X and the label vector Y which can be used for the training purposes. Recent experiments have shown that recurrent neural networks have shown a good performance in sequence to sequence learning and text data applications. Lets look at them in brief.\n\n## 4. LSTMs for Text Generation\n\n![](http://www.shivambansal.com/blog/text-lstm/2.png)\n\nUnlike Feed-forward neural networks in which activation outputs are propagated only in one direction, the activation outputs from neurons propagate in both directions (from inputs to outputs and from outputs to inputs) in Recurrent Neural Networks. This creates loops in the neural network architecture which acts as a ‘memory state’ of the neurons. This state allows the neurons an ability to remember what have been learned so far.\n\nThe memory state in RNNs gives an advantage over traditional neural networks but a problem called Vanishing Gradient is associated with them. In this problem, while learning with a large number of layers, it becomes really hard for the network to learn and tune the parameters of the earlier layers. To address this problem, A new type of RNNs called LSTMs (Long Short Term Memory) Models have been developed.\n\nLSTMs have an additional state called ‘cell state’ through which the network makes adjustments in the information flow. The advantage of this state is that the model can remember or forget the leanings more selectively. To learn more about LSTMs, here is a great post. Lets architecture a LSTM model in our code. I have added total three layers in the model.\n\n1. Input Layer : Takes the sequence of words as input\n2. LSTM Layer : Computes the output using LSTM units. I have added 100 units in the layer, but this number can be fine tuned later.\n3. Dropout Layer : A regularisation layer which randomly turns-off the activations of some neurons in the LSTM layer. It helps in preventing over fitting. (Optional Layer)\n4. Output Layer : Computes the probability of the best possible next word as output\n\nWe will run this model for total 100 epoochs but it can be experimented further.","metadata":{"_uuid":"8b8a64b96011f427c48d5b0819e3e74af604ce43","_cell_guid":"8b5d80ff-54a8-4380-8a3c-149be880551d"}},{"cell_type":"code","source":"def create_model(max_sequence_len, total_words):\n    input_len = max_sequence_len - 1\n    model = Sequential()\n    \n    # Add Input Embedding Layer\n    model.add(Embedding(total_words, 10, input_length=input_len))\n    \n    # Add Hidden Layer 1 - LSTM Layer\n    model.add(LSTM(100))\n    model.add(Dropout(0.1))\n    \n    # Add Output Layer\n    model.add(Dense(total_words, activation='softmax'))\n\n    model.compile(loss='categorical_crossentropy', optimizer='adam')\n    \n    return model\n\nmodel = create_model(max_sequence_len, total_words)\nmodel.summary()","metadata":{"_uuid":"76ef6d9352002d333a7c75e8aed7ce996015f527","_cell_guid":"60d6721e-e40e-4f2b-8f63-c06459d68f26","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets train our model now","metadata":{"_uuid":"f0b16b471969dbb831cb0024e303341e11b63de4","_cell_guid":"1826aa1a-cb77-4379-a69d-e9b180945dce"}},{"cell_type":"code","source":"model.fit(predictors, label, epochs=100, verbose=5)","metadata":{"_uuid":"156f3303b8120cc6932e6db985cbea4a7ceb08bf","_cell_guid":"07d5cf03-d171-4993-9f8b-18446649ecb0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5. Generating the text \n\nGreat, our model architecture is now ready and we can train it using our data. Next lets write the function to predict the next word based on the input words (or seed text). We will first tokenize the seed text, pad the sequences and pass into the trained model to get predicted word. The multiple predicted words can be appended together to get predicted sequence.\n","metadata":{"_uuid":"448bf43b123060dfe4e27cb9f12889e4fe0ed2a7","_cell_guid":"61e99cfe-7395-4d61-8d1a-8539103d3db5"}},{"cell_type":"code","source":"def generate_text(seed_text, next_words, model, max_sequence_len):\n    for _ in range(next_words):\n        token_list = tokenizer.texts_to_sequences([seed_text])[0]\n        token_list = pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')\n        predicted = model.predict_classes(token_list, verbose=0)\n        \n        output_word = \"\"\n        for word,index in tokenizer.word_index.items():\n            if index == predicted:\n                output_word = word\n                break\n        seed_text += \" \"+output_word\n    return seed_text.title()","metadata":{"_uuid":"e71e56543b7065f115a05e3fd062262b3b94ad46","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6. Some Results","metadata":{"_uuid":"c49bf4ea0e54f3145149e164e243d897f545b84c","_cell_guid":"ea0bddb6-acc6-4592-a2e0-ffc4129a582f"}},{"cell_type":"code","source":"print (generate_text(\"united states\", 5, model, max_sequence_len))\nprint (generate_text(\"preident trump\", 4, model, max_sequence_len))\nprint (generate_text(\"donald trump\", 4, model, max_sequence_len))\nprint (generate_text(\"india and china\", 4, model, max_sequence_len))\nprint (generate_text(\"new york\", 4, model, max_sequence_len))\nprint (generate_text(\"science and technology\", 5, model, max_sequence_len))","metadata":{"_uuid":"a21548224c9e661a29e3d369e348aada0599bdc9","_cell_guid":"e38dd280-093b-4091-b82b-9aa90045b107","_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Improvement Ideas \n\nAs we can see, the model has produced the output which looks fairly fine. The results can be improved further with following points:\n- Adding more data\n- Fine Tuning the network architecture\n- Fine Tuning the network parameters\n\nThanks for going through the notebook, please upvote if you liked. ","metadata":{"_uuid":"279f2e20c482b40d707413d0b1842f179a0d3d7b","_cell_guid":"b2cfe563-974a-4e05-ad60-233d409d3de1"}}]}