data-preparation-notebooks/02_Train-a-GPT-2-Text-Generating-Model-w-GPU.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"02_Train a GPT-2 Text-Generating Model w/ GPU","version":"0.3.2","provenance":[{"file_id":"1VLG8e7YSEwypxU-noRNhsv5dW4NfTGce","timestamp":1557696317187},{"file_id":"1mMKGnVxirJnqDViH7BDJxFqWrsXlPSoK","timestamp":1555602712120}],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"H7LoMj4GA4n_","colab_type":"text"},"source":["#  Train a GPT-2 Text-Generating Model w/ GPU For Free \n","\n","by [Max Woolf](http://minimaxir.com)\n","\n","*Last updated: May 8th, 2019*\n","\n","Retrain an advanced text generating neural network on any text dataset **for free on a GPU using Collaboratory** using `gpt-2-simple`!\n","\n","For more about `gpt-2-simple`, you can visit [this GitHub repository](https://github.com/minimaxir/gpt-2-simple).\n","\n","\n","To get started:\n","\n","1. Copy this notebook to your Google Drive to keep it and save your changes. (File -> Save a Copy in Drive)\n","2. Make sure you're running the notebook in Google Chrome.\n","3. Run the cells below:\n"]},{"cell_type":"code","metadata":{"id":"KBkpRgBCBS2_","colab_type":"code","colab":{}},"source":["!pip install -q gpt_2_simple\n","import gpt_2_simple as gpt2\n","from datetime import datetime\n","from google.colab import files"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Bj2IJLHP3KwE","colab_type":"text"},"source":["## Verify GPU\n","\n","Colaboratory now uses an Nvidia T4 GPU, which is slightly faster than the old Nvidia K80 GPU for training GPT-2, and has more memory allowing you to train the larger GPT-2 models and generate more text. However sometimes the K80 will still be used.\n","\n","You can verify which GPU is active by running the cell below."]},{"cell_type":"code","metadata":{"id":"sUmTooTW3osf","colab_type":"code","outputId":"7d16c00a-9fb5-4fea-8c86-b52c95630d74","executionInfo":{"status":"ok","timestamp":1558282169927,"user_tz":-120,"elapsed":12720,"user":{"displayName":"Jonathan Ronen","photoUrl":"https://lh5.googleusercontent.com/-Vwc9Hx3WD8U/AAAAAAAAAAI/AAAAAAAAB3o/xUrrhmcrggA/s64/photo.jpg","userId":"16626941326606151664"}},"colab":{"base_uri":"https://localhost:8080/","height":328}},"source":["!nvidia-smi"],"execution_count":2,"outputs":[{"output_type":"stream","text":["Sun May 19 16:09:27 2019       \n","+-----------------------------------------------------------------------------+\n","| NVIDIA-SMI 418.67       Driver Version: 410.79       CUDA Version: 10.0     |\n","|-------------------------------+----------------------+----------------------+\n","| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n","| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n","|===============================+======================+======================|\n","|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n","| N/A   60C    P8    17W /  70W |      0MiB / 15079MiB |      0%      Default |\n","+-------------------------------+----------------------+----------------------+\n","                                                                               \n","+-----------------------------------------------------------------------------+\n","| Processes:                                                       GPU Memory |\n","|  GPU       PID   Type   Process name                             Usage      |\n","|=============================================================================|\n","|  No running processes found                                                 |\n","+-----------------------------------------------------------------------------+\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"0wXB05bPDYxS","colab_type":"text"},"source":["## Downloading GPT-2\n","\n","If you're retraining a model on new text, you need to download the GPT-2 model first. \n","\n","There are two sizes of GPT-2:\n","\n","* `117M` (default): the \"small\" model, 500MB on disk.\n","* `345M`: the \"medium\" model, 1.5GB on disk.\n","\n","Larger models have more knowledge, but take longer to finetune and longer to generate text. You can specify which base model to use by changing `model_name` in the cells below.\n","\n","The next cell downloads it from Google Cloud Storage and saves it in the Colaboratory VM at `/models/<model_name>`.\n","\n","This model isn't permanently saved in the Colaboratory VM; you'll have to redownload it if you want to retrain it at a later time."]},{"cell_type":"code","metadata":{"id":"P8wSlgXoDPCR","colab_type":"code","outputId":"33e3e945-281e-4855-807e-170738143669","executionInfo":{"status":"ok","timestamp":1558263044090,"user_tz":-120,"elapsed":22124,"user":{"displayName":"Jonathan Ronen","photoUrl":"https://lh5.googleusercontent.com/-Vwc9Hx3WD8U/AAAAAAAAAAI/AAAAAAAAB3o/xUrrhmcrggA/s64/photo.jpg","userId":"16626941326606151664"}},"colab":{"base_uri":"https://localhost:8080/","height":145}},"source":["gpt2.download_gpt2(model_name=\"345M\")"],"execution_count":6,"outputs":[{"output_type":"stream","text":["Fetching checkpoint: 1.00kit [00:00, 482kit/s]                                                      \n","Fetching encoder.json: 1.04Mit [00:00, 53.8Mit/s]                                                   \n","Fetching hparams.json: 1.00kit [00:00, 322kit/s]                                                    \n","Fetching model.ckpt.data-00000-of-00001: 1.42Git [00:20, 68.1Mit/s]                                 \n","Fetching model.ckpt.index: 11.0kit [00:00, 5.45Mit/s]                                               \n","Fetching model.ckpt.meta: 927kit [00:00, 27.3Mit/s]                                                 \n","Fetching vocab.bpe: 457kit [00:00, 36.4Mit/s]                                                       \n"],"name":"stderr"}]},{"cell_type":"markdown","metadata":{"id":"N8KXuKWzQSsN","colab_type":"text"},"source":["## Mounting Google Drive\n","\n","The best way to get input text to-be-trained into the Colaboratory VM, and to get the trained model *out* of Colaboratory, is to route it through Google Drive *first*.\n","\n","Running this cell (which will only work in Colaboratory) will mount your personal Google Drive in the VM, which later cells can use to get data in/out. (it will ask for an auth code; that auth is not saved anywhere)"]},{"cell_type":"code","metadata":{"id":"puq4iC6vUAHc","colab_type":"code","outputId":"d4004c9f-8f23-4717-9b94-ff9f0e03956b","executionInfo":{"status":"ok","timestamp":1558262998669,"user_tz":-120,"elapsed":29851,"user":{"displayName":"Jonathan Ronen","photoUrl":"https://lh5.googleusercontent.com/-Vwc9Hx3WD8U/AAAAAAAAAAI/AAAAAAAAB3o/xUrrhmcrggA/s64/photo.jpg","userId":"16626941326606151664"}},"colab":{"base_uri":"https://localhost:8080/","height":128}},"source":["gpt2.mount_gdrive()"],"execution_count":3,"outputs":[{"output_type":"stream","text":["Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n","\n","Enter your authorization code:\n","··········\n","Mounted at /content/drive\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"6OFnPCLADfll","colab_type":"code","colab":{}},"source":["file_name = '/content/drive/My Drive/projects/seinfeldvision/whole_corpus.txt'"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"HeeSKtNWUedE","colab_type":"text"},"source":["If your text file is larger than 10MB, it is recommended to upload that file to Google Drive first, then copy that file from Google Drive to the Colaboratory VM."]},{"cell_type":"markdown","metadata":{"id":"LdpZQXknFNY3","colab_type":"text"},"source":["## Finetune GPT-2\n","\n","The next cell will start the actual finetuning of GPT-2. It creates a persistent TensorFlow session which stores the training config, then runs the training for the specified number of `steps`. (to have the finetuning run indefinitely, set `steps = -1`)\n","\n","The model checkpoints will be saved in `/checkpoint/run1` by default. The checkpoints are saved every 500 steps (can be changed) and when the cell is stopped.\n","\n","The training might time out after 4ish hours; make sure you end training and save the results so you don't lose them!\n","\n","**IMPORTANT NOTE:** If you want to rerun this cell, **restart the VM first** (Runtime -> Restart Runtime). You will need to rerun imports but not recopy files.\n","\n","Other optional-but-helpful parameters for `gpt2.finetune`:\n","\n","\n","*  **`restore_from`**: Set to `fresh` to start training from the base GPT-2, or set to `latest` to restart training from an existing checkpoint.\n","* **`sample_every`**: Number of steps to print example output\n","* **`print_every`**: Number of steps to print training progress.\n","* **`learning_rate`**:  Learning rate for the training. (default `1e-4`, can lower to `1e-5` if you have <1MB input data)\n","*  **`run_name`**: subfolder within `checkpoint` to save the model. This is useful if you want to work with multiple models (will also need to specify  `run_name` when loading the model)"]},{"cell_type":"code","metadata":{"id":"aeXshJM-Cuaf","colab_type":"code","outputId":"f93c98d4-af7a-47cd-a7fe-49b0ab730b7c","executionInfo":{"status":"ok","timestamp":1557699170551,"user_tz":-120,"elapsed":2459489,"user":{"displayName":"Jonathan Ronen","photoUrl":"https://lh5.googleusercontent.com/-Vwc9Hx3WD8U/AAAAAAAAAAI/AAAAAAAAB3o/xUrrhmcrggA/s64/photo.jpg","userId":"16626941326606151664"}},"colab":{"base_uri":"https://localhost:8080/","height":1518}},"source":["sess = gpt2.start_tf_sess()\n","\n","gpt2.finetune(sess,\n","              dataset=file_name,\n","              model_name='345M',\n","              steps=10000,\n","              restore_from='latest',\n","              print_every=100,\n","              sample_every=1000,\n","              save_every=5000,\n","              learning_rate=1e-5,\n","              run_name='first_345m_run'\n","              )"],"execution_count":0,"outputs":[{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Colocations handled automatically by placer.\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:51: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use tf.cast instead.\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/gpt_2_simple/src/sample.py:53: multinomial (from tensorflow.python.ops.random_ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use tf.random.categorical instead.\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use tf.cast instead.\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Deprecated in favor of operator or tf.math.divide.\n","Loading checkpoint checkpoint/first_345m_run/model-10000\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use standard file APIs to check for files with this prefix.\n","INFO:tensorflow:Restoring parameters from checkpoint/first_345m_run/model-10000\n"],"name":"stdout"},{"output_type":"stream","text":["\r  0%|          | 0/1 [00:00<?, ?it/s]"],"name":"stderr"},{"output_type":"stream","text":["Loading dataset...\n"],"name":"stdout"},{"output_type":"stream","text":["\r100%|██████████| 1/1 [00:06<00:00,  6.54s/it]\n"],"name":"stderr"},{"output_type":"stream","text":["dataset has 1152731 tokens\n","Training...\n","Saving checkpoint/first_345m_run/model-10000\n","======== SAMPLE 1 ========\n"," the case, I'll call him back. Elaine: Do you want to come on down? Morty: (pinching Elaine's arm) We gotta fix it. It's a beautiful vehicle, Elaine! Elaine: Why are we taking it to the mechanic? Morty: We gotta fix 'em. Elaine: Yeah, but we're taking the whole thing apart. Morty: No, no. We gotta fix it. Because I have to. Elaine: All right, I just wanna get my car. I want it back. She leaves. MRS. Sokol: (holding up a small bill) It's ten dollars. Elaine: What? Mister Price: I bought this for my uncle. He's handicapped - he doesn't drive. He's too lazy. Elaine: Oh, ok. How much? (looks at her bill) Oh, alright. [Restaurant] Jerry and Karen at a table. Karen is putting away her food. Jerry: So what do you think? Karen: I always thought he was handsome. Jerry: Oh. Well, then what do you think? Karen: Yes, he's handsome, but? Jerry: Pretty ugly. The eyebrows, the eyes. Karen: That's it. Jerry: Hey, I've got to get going. I got to see Maggie. Karen: Okay. Jerry: Maggie's just down the hall. I'll just come right over. Karen: Okay. Jerry: (interrupting) Hey, I've got to meet Gail. Gail: Jerry? What are you talking about? Jerry: Well, I was talking with Gail. Gail: Jerry, what are you talking about? Jerry: It's not important, okay? She's not there. Gail: What's the difference? Jerry: I've talked with Gail. Karen: Gail, who is this? She's not Gail. Gail: Jerry, you're on! Jerry: You're right, she's not Gail. Gail: Jerry, I'm gonna beat you to it! You're gonna give me up for civil service? Jerry: Why wouldn't I take the test? You know it's a great place to be. Gail: Jerry. Jerry: No, don't go. Karen: What test? Jerry: You know, the one on the confidence thing? The one about the confidence? Gail: You've got a brain. Jerry: No, I'm not. Gail: So? Jerry: And you know, if I took it away from me, I would not be who I am. Karen: Yeah, me neither. Gail: So what are you doing now? Jerry: Well, I dunno, you can tell me anything. Gail: All right. What is it? Jerry: Uh, well, I think I wanna try and make this car payment. Gail, look, there's a man across the street doing sixty. Elaine is getting a car from the garages. Gail: Oh, okay. That's it. Mmm, I like this. Gail: Well, I like it! You just admire it. Jerry: Oh, yeah. Gail: Do you? You think I envy it? Jerry: Oh, do! Gail: So, how much do you want for the car? Jerry: I love the car. I could buy a new one tomorrow. Gail: What about the interest? Jerry: Oh, I don't mind if you do it. Gail: Do you? Gail is looking into Jerry's wallet. [garbage can] Karen: What are you getting? Mmm, can't find anything. Jerry: This is beautiful. That's the best part about getting older. You look at it, and it all makes perfect sense. Karen: This is so nice. It's so nice to see it make sense. Jerry: Uh-huh. That is so kind of you. Karen: What? Jerry: Yes, I think I might want to keep this. Karen: No, I don't want to keep it. Jerry: No, I don't want to keep it. Karen: I can't take it anymore! Jerry: Oh, come on, what are you gonna do? Karen: I'll just borrow it again. Do it! Do it now! I know this is unfair, but it is true. Jerry: I'm afraid I have some terrible insecurity. Karen: Oh, come on. Do it! Elaine is at the mall. Elaine: Hello, Rick. I thought you had this great collection. Can I borrow your guitar? MAN #1: You can have my guitar. ELAINE: No, no, I really wanted to say hi to you. Can I just borrow yours? MAN #2: Sure. Elaine: Well, I'm glad to do that. MAN #2: Okay, hold on. (Elaine walks over) Elaine: Hello. Morty, this is\n","\n","[10100 | 213.68] loss=0.73 avg=0.73\n","[10200 | 381.84] loss=1.27 avg=1.00\n","[10300 | 550.82] loss=1.33 avg=1.11\n","[10400 | 719.76] loss=1.71 avg=1.27\n","[10500 | 887.96] loss=2.08 avg=1.43\n","[10600 | 1056.36] loss=2.20 avg=1.56\n","[10700 | 1224.88] loss=1.97 avg=1.62\n","[10800 | 1393.51] loss=2.17 avg=1.69\n","[10900 | 1562.02] loss=1.08 avg=1.62\n","[11000 | 1730.58] loss=1.51 avg=1.61\n","======== SAMPLE 1 ========\n"," wanna know what you need in a friend. Jerry: How could you not give me an answer? That's your 'go to' answer! George: Well, I... Kramer: You're thinking of getting her an apartment? Jerry: I, am thinking. Kramer: Well, that's nice. I'll give her a call if I hear from you again. Kramer: Oh, ha. Ha. It's... it's a privilege to serve with you. Jerry: I'm sure it's nothing. [The Boss walks into the office - she's trying on a jacket and Kramer is already waiting for her.] [We cut to Elaine - she's on the phone talking to Boss] [Monk's] Jerry: So what are you up to now? Elaine: I'm painting. I'm... I'm painting my grandfather's graystone. Jerry: That's pretty cool. You paint in the church basement? Elaine: Why don't you try painting in my office? [Elaine tries on jacket] It still has the big holes, right? Jerry: Eh, it fades a little. Elaine: [laughs nervously] Where did you get it? Jerry: Painting. My uncle has it. Elaine: Uncle? Jerry: His name was Norman. He was a good friend of mine. Elaine: Oh, yeah. I broke up with her. [A woman walks by and Kramer and Jerry make eye contact.] Woman: Listen, I was at this charity auction and they're having a raffle. Jerry: Yeah, it's for donated furniture. I'll send you a check for that pronto. Elaine: Well, I'll make sure it gets picked out. Woman: Sure. [Jerry opens a gift] Elaine: [laughs nervously] This thing is velvet. Jerry: Oh, man. [George walks in with a box of chocolates.] Hey. How'd it turn out between the eyes? George: They put a tube in? Jerry: The eyeball socket. George: [gives Jerry a thumbs-down] Yeah, you don't need me. Jerry: So, how do you like the job? George: Oh, man, it's fantastic. Jerry: [doing the Jerry of the year impression] My Jerry. You know, the one whose work you don't notice is inferior to, and whose personal relationships are all that remain after he finishes. George: No, no, no, no. Your Jerry. Jerry: Here you go. How do they do it? George: They're doing it. We're... having a little mishap at their expense. Jerry: [in awe] Just like you? [thoughtful] Well... [George turns bright red] [Elaine walks in dressed in a black and white dress, with a slit down the front revealing her breast. George is looking at Elaine's breasts, with a face of deep fear.] Elaine: Oh! Hi! Jerry: [to nurse] Hey! It's me! Elaine: Hi, nurse. Jerry: Nice to see you. Elaine: Oh, you too, nurse. How you doin'? Jerry: Hey, a little better. Elaine: Soggy? Jerry: [examining Elaine's breasts] What? Elaine: Soggy? What kind of semen is that? Jerry: Oh, not soggy - bukake. Elaine: What - what the hell is that? Jerry: It's a kimono. Elaine: Then what are you? A samurai? What, are you some kind of badass, Jerry? Jerry: [trying to look shocked] Well, nice to meet you. Elaine: I don't know, what are you? A rapist? A murderer? Jerry: No, I'm just a regular guy passing on some good news to his friend. Elaine: Jerry, are you listening to this? Jerry: Oh, hey, this is the same guy who's calling me all these problems lately. Well, if you wanna hear his problems, well, listen, I, uh... [exits] Elaine: Hey! Kramer? Jerry: Hey! Kramer: Hey, what's with the kimono? Jerry: It's only me and Elaine doing the talking. Kramer: Oh, yeah, so? Kramer: Well, you remember a while back we made fun of you about that white swim trunks. Do ya realize now that was a huge cheap shot? Jerry: A cheap shot? I didn't say that. Kramer: Well, I did say it, and I take full responsibility for it. Jerry: I wouldn't wear a swim trunks to class, Jerry. Kramer: Oh, yeah? Well, now you'll be. Jerry: Yeah. Kramer: What about you? Did you take any offense to it? [Jerry and Elaine both laugh out loud] Elaine: Do you know, Elaine, that if I had said anything to anybody, I may have ended up in jail? Jerry: Yes, yes. Elaine: How's\n","\n","[11100 | 1921.22] loss=1.54 avg=1.60\n","[11200 | 2089.89] loss=2.03 avg=1.64\n","[11300 | 2258.65] loss=1.58 avg=1.64\n","[11400 | 2427.27] loss=1.50 avg=1.63\n","[11500 | 2595.87] loss=1.98 avg=1.65\n","[11600 | 2764.45] loss=1.26 avg=1.62\n","[11700 | 2933.11] loss=1.61 avg=1.62\n","[11800 | 3101.63] loss=1.38 avg=1.61\n","[11900 | 3270.08] loss=1.34 avg=1.59\n","[12000 | 3438.48] loss=0.29 avg=1.52\n","======== SAMPLE 1 ========\n","'s like that. (Manny, his supervisor enters the store, waves, and then leaves) The woman I work for, she works in the shoe section. She's a very sweet woman, you don't ever... ...tell her what you're doing. (Scene cuts to Kramer and Jerry in another conference room at the show. Their faces are to the side, showing they'reneither in a good mood nor at a loss for words. Kramer is scribbling something down while Jerry is still trying to make a point) Kramer: I can't stand by while these people are murdered. I can't bear it if two people are killed in fast succession. Jerry: (Explaining) Alright, alright. I'll tell ya what, let this thing drop... (Kramer, still at it, leaves for his office. We see Jerry and Kramer exiting Jerry's apartment) Kramer: I don't know if I'm ready to say it...(imitates Jerry) I'm not ready to say it...(again), I'm not ready to say it...(again)...but I feel it. ...It's...it's there. If one guy slips at the knee, the whole thing collapses. But...but...it's there. (Scene cuts to closer inspection of a torn shirt. Kramer, wearing the shirt, walks over) Kramer: I tear shirts! I've never even seen one that small! It's a miracle! (More laughter from the crowd and Jerry. Then our heroes see someone shooting into the crowd from the front. There's an unearthly feeling in the air. Everyone except Jerry, who is staring off into space. Jerry looks directly at Kramer who is still writing on his notebook) Jerry: Kramer! What are you doing down there?! Kramer: Jerry this is insane! Why are all these people shooting? Jerry: They must have a good laugh at our expense. (Jerry, clearly the more sane of the two, is about to answer Kramer's question, turns around and sees George in a meeting discussing Jerry's movie recommendations) Jerry: What are you alluding to? Kramer: No one's ever done a 'Good Will Hunting' scene before. Jerry: No, not a peep. George: (in awe) Kramer! Kramer! Jerry: Is that right? (Kramer walks over to the open door) Kramer: Alright everybody, I'm back. I found 'em all, (he waves at George as he enters) I'm off the wagon! George: (grabs Kramer and heads for the door as Jerry speaks up) Hey, George, where's the bathroom? Jerry: (turns back around to Kramer) What about it? George: The bathroom? The bathroom? It's all over the bulletin board. (Kramer quickly follows George into the store) Kramer: (walking over to one of the store's bathroom sinks) Hey, look at this one. See that's full? (flushes as he sees the elbow he's holding steady at) Can you do that in here? (flushes while reaching the sink and empties the needle from his arm into the drain) (Scene cuts to outside Jerry's apartment, George and Kramer are still hammering out the details of the movie, we just saw them from Kramer's angle. Kramer is now holding the end of the shower head in contact with George's temple, we can see his face through the shower curtain, probably thinking) Kramer: Alright. Five hundred dollars. George: (surprised) What?! Kramer: You believe me? (starts pouring a glass of wine) Alright, So, the movie's set in the nineteen-thirt... George: Nineteen-thirt... Nineteen-thirt. Thirty-five dollars. (makes swallowing noises) Kramer: (impatient) Five hundred dollars, fifty-five fifty dollars. George: Alright, yep. Alright. Fifty-five dollars. (starts pouring again, louder and louder) Alright, now, the problem with this city is, they have no water. Wherever you go, wherever you live, wherever you work, you never know. (we see Jerry's face through the curtain as Kramer finally succumbs to his own wine) Kramer: (still recovering) Ahh! Ahh... Jerry: The rain? Kramer: Yeah, yeah, yeah. It rained today in this store. And it rained earlier today than yesterday. That's because yesterday was my period. Jerry: (adamant) You are not exaggerating. (starts finishing his wine) What? Kramer: Yeah, today was my period. I've never been piquered. Jerry: You mean, all the time you've been pumping yourself full of Dilaudid? Kramer: Every moment I've stayed up, drinking is time wasted. (A worried Jerry looks up to see George standing a few feet off the floor. He takes a second or two to collect himself then continues his wine drinking. He's wearing\n","\n","[12100 | 3629.01] loss=1.08 avg=1.50\n","[12200 | 3796.82] loss=1.55 avg=1.50\n","[12300 | 3965.09] loss=0.18 avg=1.44\n","[12400 | 4133.31] loss=1.52 avg=1.44\n","[12500 | 4301.62] loss=1.22 avg=1.43\n","[12600 | 4470.01] loss=1.28 avg=1.42\n","[12700 | 4638.30] loss=1.36 avg=1.42\n","[12800 | 4806.68] loss=1.25 avg=1.41\n","[12900 | 4975.03] loss=1.80 avg=1.43\n","[13000 | 5143.47] loss=1.60 avg=1.44\n","======== SAMPLE 1 ========\n","  Elaine and Kramer at Elaine's Place. Elaine is wearing work clothes while Kramer is looking very fashionable. Kramer: Eh, uh, excuse me. Elaine: Well, look at my outfit. What can I tell you? Kramer: Oh c'mon I got cuticles in Blazer and Chinos. Elaine: No, none of that. What's up? (George enters) George: I got a date with Susan. (Kramer stops in his tracks) Kramer: What? The Susan you mentioned? George: Yeah, I'm thinking about Susan. She's one of those women that seems to come about because of you asking her out. Kramer: Oh uh, it's a beautiful woman, Jerry. Beautiful soul, Jerry. Elaine: Yeah, it's just that today when I found out Susan is engaged...George...I...I cannot tell you how much I appreciate that. George: It's not that you don't appreciate it, Elaine. Elaine: I don't think she'd have much problem with it. George: Well uh, I'll tell you Jerry, I have a feeling about Susan's status. I think she's thinking about it. If...if you say something, you could see her in the spring of '93. Now Kramer, let me tell you something About That: Susan's parents might be very upset, and I can't see what use it is to them to confuse this with a test. Kramer: Yeah, I know what you mean. George: Maybe I could ask her to the wedding. Kramer: Well, that's nice. Elaine: (laughs) Well, what's more polite, \"Would you like to wed Susan?\" \"Her Parents: Yes, We Would Very Much Like To Be Conquered Again.\"\" Or, \"Would You Like To Be Conquered Again By Your Parents: By Your Cousin, Or Your Uncle Bucket Of Fudge?\" Jerry: What is this, like a yodel? George: No, more like a waltz. You start the Fifth, then People: \"Well, maybe this waltz should do.\" George: What could be more politically incorrect? Jerry: Oh, no, no no. I can't hear Jerry. That's why I got you that Klaxon. George: (shouting back) Get the TV, get the TV! Elaine: Oh, they're right downstairs. Oh, listen, there's this woman that works at Pendant Publishing. She said that they'd be glad to send you over to Los Angeles to work on their graphic novels. George: Oh, I can't believe this. I love graphic novels. I have a, a, a love for graphic novels. She said if I wanted to read one, I could do it. Elaine: Oh, I got it. That's very sweet. George: Thank you. (they exit) Elaine: (to Jerry) Here, have a bite. Jerry: Ah, the cookie. Elaine: Here, have a bite. Jerry: Ah, the cookie. Kramer: Woah, woah, woah, woah. Listen, I gotta go. Jerry: Listen, you just keep an eye on George because he can really screw you over. Kramer: Yeah? What, what, what are you talking about? Jerry: No, what am I talking about? Kramer: All right, what did I do to you? Jerry: I don't know. Kramer: Oh, oh, it was an accident, it was an accident. You look terrible. Jerry: I don't feel anything. Kramer: What, what happened? Jerry: I feel like I was daubed on with a wet cotton ball. (Scene ends.) [Setting: Night club] Jerry: I hate going in with girls. I really hate it. I had this whole thing where I would go in with a girl - you know, I'd say, \"Hey, would you mind if I put dirt all over this carpet?\" And she'd say, \"Sure.\" Then the next week, \"Hey, would you mind if I put dirt all over your legs?\" Then the next week, \"Hey, if you put dirt on my feet, you'd mind if I put it all over your feet?\" And the third week, \"If it was a car, would you mind if I mud on my shoes?\" And the fourth week, \"Uh, shoes, I do mud.\" Then the fifth week \"Uh, mud.\" You know, it's like, why does it get better and better the farther you go in your life? Why don't they just, just give you heroin? The better you get, the better you get, the more you're gonna be offered. You know what I'm saying? Why do women keep putting up with this crap? Why do men have to be bred to do this shit? It's barbaric. Boy, I really wish I had more nerve. George: All right. Let's stick with the story.\n","\n","[13100 | 5334.46] loss=1.04 avg=1.42\n","[13200 | 5502.92] loss=1.22 avg=1.41\n","[13300 | 5671.31] loss=1.58 avg=1.42\n","[13400 | 5839.32] loss=1.10 avg=1.41\n","[13500 | 6007.25] loss=1.26 avg=1.40\n","[13600 | 6176.28] loss=1.63 avg=1.41\n","[13700 | 6345.01] loss=0.79 avg=1.39\n","[13800 | 6513.49] loss=1.22 avg=1.39\n","[13900 | 6682.40] loss=1.73 avg=1.40\n","[14000 | 6851.13] loss=0.77 avg=1.38\n","======== SAMPLE 1 ========\n"," Stock photo.   Kramer: Well, I should be heading home. Jerry, you   and George are gonna escort me to my apartment.   Jerry: Where?   New scene - George and Estelle in George's apartment. Estelle is in Jerry's bed.   George: Where are we eating?   JERRY (shudders): Oh, I'm really sorry about last night. It was really bad.   George: What did we have on Maestro Molina's falafel?   Estelle: We had anchovies and she didn't even know it.   George: That was awful.   Estelle: Well, if you don't mind.   George: Not that I wouldn't have loooved it.   Jerry: Well, that's a start.   George: How about that? Looks pretty good.   Estelle: Yeah. Good one, George.   George: You know, you're getting a little nutty.   Jerry: You're right. I'm getting a little nutty.   Estelle: I don't know why I ever liked George in the first place.   George: You know what - we should probably get a little something to go,   before we go downstairs.   Jerry: Because I could never find my pens.   New scene - Jerry and Kramer in the sauna at Moscone. Kramer is   making a hot noise.   Kramer: Oh, man!   Jerry: What the hell is that noise?   Kramer: It's the click of a rubber mallet against a wooden countertop.   Jerry: Wait a minute, what is that noise? Are you using your toilet?   Kramer: Oh, that's my urine dispenser. It dispenses with a bowl.   Jerry: A bowl!? That is not a bowl.   Kramer: Well, I got a whole roll of toilet paper in there too, you know.   Jerry: Then what's the difference?   Kramer: Look, I'll just get the paper, okay?   Kramer goes into the bedroom to get the paper dispenser and turns to say   something to the sliding glass door. But as he opens it, he finds the sliding   glass door locked.   Jerry: What's going on?   Kramer: That's a door I don't use. It's locked.   Jerry: Why do you need a sliding glass door?   Kramer: Look, I'm going to the airport now and I want to get that roll of   toilet paper out of the house as soon as possible.   (Commercial)   Commercial.   [George and Jerry in Jerry's apartment, talking by phone. Jerry is   starting the phone booth]   George: You sure you don't wanna do it over the wire?   Jerry: No good?   George: No kidding?   Jerry: Yeah.   George: (frustrated) There. The bucks and yen's making the speedball,   that old flame who never came back from twos.   George: (to Jerry) It's the same thing with Elaine.   Jerry: (to George) Oh my god, I can't believe what I'm hearing.   The phone rings. Jerry picks up.   Jerry: Hello?   On the other end.   George: (horrified): It's me!   Jerry: Come on, come on, come on.   George: You know what? (explaining) I'm running outta paper.   Jerry: No more paper!   George: Why not?   Jerry: Because the sliding glass door is much harder to open in an emergency.   You gotta get as wide of a opening as you can.   (as George attempts the door)   George: It's a sliding glass door, it's not gonna be a latch-key.   (Jerry looks down at the paper on the end of the phone)   Jerry: (shouting) What about the \"67 Club\"?   George: The \"67 Club\"? Are you serious? You're kidding!   Jerry: I'm really gonna need that.   George: The \"67 Club\"? You're giving me away for free?!   Jerry: I don't think I want any \"67 Club\".   George: Yeah, the whole package - all the money, the women, the thrill.   Jerry: Alright, we're in to something here.   George: It's a sliding glass door. It's gonna   stay that way.   Jerry: What about \"The Gilmore List\"? (snaps fingers)   The \"Gilmore Ned\"?   George: (recoil) W\n","\n","[14100 | 7043.53] loss=1.33 avg=1.38\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"IXSuTNERaw6K","colab_type":"text"},"source":["After the model is trained, you can copy the checkpoint folder to your own Google Drive.\n","\n","If you want to download it to your personal computer, it's strongly recommended you copy it there first, then download from Google Drive. (NB: if you are downloading the model to your personal computer, download the large model checkpoint file *seperately*, download the other files, and reconstruct the `/checkpoint/run1` folder hierarchy locally)."]},{"cell_type":"code","metadata":{"id":"Tvxf13FTuAMG","colab_type":"code","colab":{}},"source":["import os"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"VHdTL8NDbAh3","colab_type":"code","colab":{}},"source":["# gpt2.copy_checkpoint_to_gdrive()\n","gpt2.copy_checkpoint_to_gdrive(checkpoint_folder=os.path.join('checkpoint', 'first_345m_run'))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"qQJgV_b4bmzd","colab_type":"text"},"source":["You're done! Feel free to go to the **Generate Text From The Trained Model** section to generate text based on your retrained model."]},{"cell_type":"markdown","metadata":{"id":"pel-uBULXO2L","colab_type":"text"},"source":["## Load a Trained Model Checkpoint\n","\n","Running the next cell will copy the `checkpoint` folder from your Google Drive into the Colaboratory VM."]},{"cell_type":"code","metadata":{"id":"DCcx5u7sbPTD","colab_type":"code","colab":{}},"source":["# gpt2.copy_checkpoint_from_gdrive()"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"RTa6zf3e_9gV","colab_type":"text"},"source":["The next cell will allow you to load the retrained model checkpoint + metadata necessary to generate text.\n","\n","**IMPORTANT NOTE:** If you want to rerun this cell, **restart the VM first** (Runtime -> Restart Runtime). You will need to rerun imports but not recopy files."]},{"cell_type":"code","metadata":{"id":"-fxL77nvAMAX","colab_type":"code","outputId":"89b5593d-a39d-4377-8c81-ceab5811e4c3","executionInfo":{"status":"ok","timestamp":1558219044792,"user_tz":-120,"elapsed":16953,"user":{"displayName":"Jonathan Ronen","photoUrl":"https://lh5.googleusercontent.com/-Vwc9Hx3WD8U/AAAAAAAAAAI/AAAAAAAAB3o/xUrrhmcrggA/s64/photo.jpg","userId":"16626941326606151664"}},"colab":{"base_uri":"https://localhost:8080/","height":183}},"source":["sess = gpt2.start_tf_sess()\n","gpt2.load_gpt2(sess, run_name='first_345m_run')"],"execution_count":5,"outputs":[{"output_type":"stream","text":["WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Colocations handled automatically by placer.\n","Loading checkpoint checkpoint/first_345m_run/model-1000\n","WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n","Instructions for updating:\n","Use standard file APIs to check for files with this prefix.\n","INFO:tensorflow:Restoring parameters from checkpoint/first_345m_run/model-1000\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ClJwpF_ACONp","colab_type":"text"},"source":["## Generate Text From The Trained Model\n","\n","After you've trained the model or loaded a retrained model from checkpoint, you can now generate text. `generate` generates a single text from the loaded model."]},{"cell_type":"code","metadata":{"id":"4RNY6RBI9LmL","colab_type":"code","colab":{}},"source":["samples = gpt2.generate(sess, nsamples=10, batch_size=10, return_as_list=True,\n","                       prefix=\"(Ella put the brown cheese in an ice bath)\",\n","                        include_prefix=True, length=200, temperature=0.7,\n","                       run_name='first_345m_run')"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"JBr9G3hrJ8G8","colab_type":"code","outputId":"2fbbfbd8-abe8-4a42-e19b-44e29d4acf5a","executionInfo":{"status":"ok","timestamp":1558281914980,"user_tz":-120,"elapsed":15254,"user":{"displayName":"Jonathan Ronen","photoUrl":"https://lh5.googleusercontent.com/-Vwc9Hx3WD8U/AAAAAAAAAAI/AAAAAAAAB3o/xUrrhmcrggA/s64/photo.jpg","userId":"16626941326606151664"}},"colab":{"base_uri":"https://localhost:8080/","height":2231}},"source":["[e.split('. ') for e in samples]"],"execution_count":13,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[[\"(Ella put the brown cheese in an ice bath) [Setting: Ikea ] (Jerry and Susan are in a booth) Jerry: (Looking at his watch) So, how long is it gonna be until we can go? Susan: Oh, I don't know, maybe an hour\",\n","  'Jerry: (Smiling) Sounds like a pretty good night to us',\n","  \"(They both laugh) [Setting: Coffee Shop ] (George and Susan are eating at the same table) Susan: (Taking a bite of her apple) I'm sorry\",\n","  \"I know it's not good for you, but for me, it just doesn't cut it\",\n","  \"I mean, we've both known that the other suffered from an eating disorder, but this is the first time that we've come out and admitted it\",\n","  'And I love it',\n","  'I love the way you look at me, and you know that I suffer from no ill will towards you',\n","  \"And I promise you, that no matter what happens in the future, you're going to be the first person\"],\n"," [\"(Ella put the brown cheese in an ice bath) Oh, you're really going to town with this, aren't you? Jerry: Yeah\",\n","  \"(Elaine is doing it, she's laughing it up) Elaine: Oh, man! Told you! (she touches Jerry's ear) Jerry: Elaine, Elaine, don't do that\",\n","  'Elaine: (coughing) My head hurts',\n","  '(Jerry rubs his head with his sleeve) Elaine: What? What is it? Jerry: Oh, Elaine',\n","  'You have a cold',\n","  \"Elaine: That's terrible\",\n","  \"Jerry: (emph) It's not terrible! Elaine: It's terrible to do that\",\n","  \"Jerry: (pained) I can't\",\n","  \"I can't\",\n","  \"Elaine: Do you have any key chains? Jerry: (pained) What? How did I get into this? Elaine: Don't you see what's happening here? Jerry: This is a woman I know\",\n","  \"I should be able to talk to her like we're friends\",\n","  'Elaine: ('],\n"," ['(Ella put the brown cheese in an ice bath) George: (staring at the buttery brown stuff) Oh, my god',\n","  'Alton: (doing the voice) Oh, uh, Mr',\n","  'Costanza',\n","  \"Costanza: Uh, George, I thought I'd just buy you some coffee, but..\",\n","  'George: (doing the voice) No, no, no, no, no',\n","  'I want to take you out',\n","  \"(George shakes hands with Alton) George: (doing the voice) I'm very, uh, I'm not gonna have anything with the brown stuff\",\n","  'Alton: Mr',\n","  'Costanza, may I? Costanza: Oh, yes',\n","  \"I've noticed that you're keeping a very low profile\",\n","  \"George: (doing the voice) Ah, well, I'm not\",\n","  'Costanza: Very good',\n","  '(George looks around and sees the entire party sitting around him, eating at the same time) (George leaves and tries to go back, but a look of shock shows through his clothes',\n","  'He runs'],\n"," [\"(Ella put the brown cheese in an ice bath) I'm sure you're very concerned about my welfare\",\n","  \"(he nods) Well, I'm very concerned about yours\",\n","  'Elaine: Well, what is my chance of ever becoming a dentist? (he frowns) (pause) Look, you know, you should take advantage of this',\n","  \"You'll get a chance to meet someone, and maybe even be married\",\n","  \"Jerry: Marriage? [Setting: Jerry's apartment, night] (Elaine and Jerry are sitting on the couch) Elaine: I'm sorry, Jerry, I know this is hard for you\",\n","  \"But, I've realized one thing\",\n","  \"I don't deserve this love\",\n","  'Jerry: No, Elaine',\n","  'Love is not something that comes easily',\n","  \"It's a demanding thing\",\n","  'It demands concentration, it demands skill',\n","  \"But, it's also very, very superficial\",\n","  \"Elaine: Well, I'm not complaining\",\n","  \"(she yawns) Jerry: (Mocking her) It's a good thing you're not married, because you'd\"],\n"," [\"(Ella put the brown cheese in an ice bath) So when you're ready, I want you to meet my parents\",\n","  'Jerry: Oh',\n","  \"Elaine: Oh, my parents, yes! Elaine: (sits on the couch) Dad, this place is gorgeous! (George enters) Good morning! George: Morning! Jerry: George! Jerry: So, you're dating? George: I got, like, a little crush on her\",\n","  'Jerry: Really? George: Yeah',\n","  \"Jerry: So, you're taking her out to dinner\",\n","  'Can you do that? George: Uh, well, I got to get some sleep',\n","  \"I got to, uh, run, run, run, the half-marathon, run the half-marathon, run the half-marathon, and then I'll see you\",\n","  \"Jerry: I'm sorry\",\n","  'George: Oh, come on, come on',\n","  \"I'm in bed, and you can't even go to the dinner\",\n","  \"I don't even want to see you! Jerry: Well,\"],\n"," ['(Ella put the brown cheese in an ice bath) Jerry: What was that? Elaine: That was okay',\n","  \"(Leaves) George: I'm back\",\n","  \"(Picks up the phone) Yeah, that's the place\",\n","  'Yeah, they got any more new material',\n","  'Yeah, call me',\n","  'Yeah',\n","  'Yeah',\n","  \"(Hangs up) Jerry: I couldn't, I had to stay here\",\n","  \"I'm sorry\",\n","  \"It's just a -- a little thing\",\n","  '(Hangs up) What was that? George: I almost got fired for this',\n","  \"(Hangs up) Jerry: What? I didn't do anything\",\n","  'I deserved it',\n","  'George: What, you think I deserved it? George: Well, you know what I think? I think you deserved it',\n","  'I mean, I thought I deserved it',\n","  \"Jerry: George, I didn't do anything\",\n","  \"George: Well, let's not forget that\",\n","  '(Gets up and walks to the phone) Jerry: What? (Hangs up) George: Alright, alright'],\n"," [\"(Ella put the brown cheese in an ice bath)   Kramer: How much longer?   [The brown cheese is being strained - trying to get the liquid out]   Jerry: What are you doing?   Jerry: I don't know\",\n","  \"  Kramer: Why?   Jerry: I don't know\",\n","  \"  Kramer: What are you doing?   Jerry: I don't know\",\n","  \"  Kramer: Why?   Jerry: I don't know\",\n","  \"  Jerry: How much longer are you gonna do this?   Kramer: I'm gonna do it until I'm blue in the face\",\n","  \"  Jerry: Blue in the face? How blue is blue in the face?   Kramer: I don't know\",\n","  '  Jerry: Blue in the face?   Kramer: Blue in the face?   [Elaine enters]   Jerry: Elaine?   Elaine: Yeah',\n","  '  Jerry: Remember what we talked about last night?   Elaine:'],\n"," ['(Ella put the brown cheese in an ice bath) Jerry: So what do you say? Elaine: Oh, I know',\n","  'I said yes',\n","  \"I'm going to Detroit\",\n","  \"(She turns around and walks away) Jerry: Really? Elaine: Yeah, I'm going to Detroit\",\n","  \"I'm going to the airport\",\n","  \"I'm going to pick up her car\",\n","  'Jerry: Okay, what do you want? Elaine: Well, I think she likes me',\n","  \"Jerry: Well, I don't think so\",\n","  \"Elaine: Well, you're not going to like her\",\n","  \"Jerry: You're not going to like her? Elaine: I don't know\",\n","  \"I mean, maybe she doesn't like me, but..\",\n","  'Jerry: Not like her',\n","  \"Elaine: I don't know, you don't like her? Jerry: No, you don't like her\",\n","  \"You don't like her\",\n","  \"Elaine: How can I not like her? Jerry: Well, if you don't like her, you wouldn't date her\",\n","  'Elaine: Oh, come on'],\n"," ['(Ella put the brown cheese in an ice bath) Thank you',\n","  \"Jerry: (takes the brown cheese out of the ice bath) What? You're not giving it back? Elaine: No, no, no\",\n","  'I bought it',\n","  \"It's my baby\",\n","  'Jerry: You bought the brown cheese for my baby? Elaine: Yes',\n","  \"Jerry: So, what are you complaining about? Elaine: I didn't get a chance to use it\",\n","  'Jerry: You spent three hours in the ice bath for? Elaine: Well, I was in a lot of pain',\n","  'Jerry: A lot of pain? Elaine: Yes',\n","  'Jerry: You were in the ice bath for three hours? Elaine: Yes',\n","  \"Jerry: And you didn't complain? Elaine: I didn't have the guts to say anything\",\n","  '(Ella walks away) Jerry: Did you complain? Elaine: Yes, I complained',\n","  'It was a very unpleasant experience, in that I was constantly being tested by my patient for my personal hygiene',\n","  'I was constantly asking, \"Have you'],\n"," [\"(Ella put the brown cheese in an ice bath)   Jerry: (yelling) That's it!   (Kramer turns to see what's going on\",\n","  'Elaine is in front of him)   Kramer: Hey!   Jerry: What happened?   Kramer: Well, I thought you were gonna get a pair of pants at that Swarovski party',\n","  \"  Jerry: It was like I was back in the 80's!   (Elaine takes out the brown cheese from the ice bath and puts it on Jerry's head)   Elaine: I just wanted you to know that I appreciate everything you're doing for me\",\n","  \"  Kramer: Really?   (Kramer gives a big smile and a thumbs up)   (Elaine's apartment\",\n","  'Jerry and Elaine are talking',\n","  \"Elaine's on the phone in New York with her boyfriend, Brian)   Elaine: (mouths) Well, you know, (mumbles) I really think about you guys all the time.\"]]"]},"metadata":{"tags":[]},"execution_count":13}]},{"cell_type":"markdown","metadata":{"id":"oF4-PqF0Fl7R","colab_type":"text"},"source":["If you're creating an API based on your model and need to pass the generated text elsewhere, you can do `text = gpt2.generate(sess, return_as_list=True)[0]`\n","\n","You can also pass in a `prefix` to the generate function to force the text to start with a given character sequence and generate text from there (good if you add an indicator when the text starts).\n","\n","You can also generate multiple texts at a time by specifing `nsamples`. Unique to GPT-2, you can pass a `batch_size` to generate multiple samples in parallel, giving a massive speedup (in Colaboratory, set a maximum of 20 for `batch_size`).\n","\n","Other optional-but-helpful parameters for `gpt2.generate` and friends:\n","\n","*  **`length`**: Number of tokens to generate (default 1023, the maximum)\n","* **`temperature`**: The higher the temperature, the crazier the text (default 0.7, recommended to keep between 0.7 and 1.0)\n","* **`top_k`**: Limits the generated guesses to the top *k* guesses (default 0 which disables the behavior; if the generated output is super crazy, you may want to set `top_k=40`)\n","* **`truncate`**: Truncates the input text until a given sequence, excluding that sequence (e.g. if `truncate='<|endoftext|>'`, the returned text will include everything before the first `<|endoftext|>`). It may be useful to combine this with a smaller `length` if the input texts are short.\n","*  **`include_prefix`**: If using `truncate` and `include_prefix=False`, the specified `prefix` will not be included in the returned text."]},{"cell_type":"code","metadata":{"id":"8DKMc0fiej4N","colab_type":"code","outputId":"ec1fc622-c3fe-43e5-9312-f26cec6e52fa","executionInfo":{"status":"ok","timestamp":1555618327694,"user_tz":420,"elapsed":9197,"user":{"displayName":"Max Woolf","photoUrl":"https://lh6.googleusercontent.com/-SAn9dAVrqaw/AAAAAAAAAAI/AAAAAAAAA0o/9b0Ok-MDXIU/s64/photo.jpg","userId":"10954469476206133987"}},"colab":{"base_uri":"https://localhost:8080/","height":3679}},"source":["gpt2.generate(sess,\n","              length=250,\n","              temperature=0.7,\n","              prefix=\"LORD\",\n","              nsamples=5,\n","              batch_size=5\n","              )"],"execution_count":0,"outputs":[{"output_type":"stream","text":["LORD FITZWATER:\n","Saw'st thou not, sir?\n","\n","QUEEN ELIZABETH:\n","Madam, didst thou not mean, as some do,\n","To cram others with beggary?\n","\n","QUEEN ELIZABETH:\n","Ha, ha! what should you say? why didst thou say no?\n","\n","QUEEN ELIZABETH:\n","Hath this woman in this world hungrier than man?\n","Think'st thou that all these woes can be avoided?\n","\n","QUEEN ELIZABETH:\n","As many lives she hath prevented as man.\n","\n","DUCHESS OF YORK:\n","Why, hark! the Tower is brought down.\n","\n","QUEEN ELIZABETH:\n","The King of Naples comes apace.\n","\n","DUCHESS OF YORK:\n","It shall be so, I hope.\n","\n","QUEEN ELIZABETH:\n","The Tower comes down.\n","\n","DUCHESS OF YORK:\n","The Naples comes apace.\n","\n","QUEEN ELIZABETH:\n","It shall be so, I hope.\n","\n","DUCHESS OF YORK:\n","\n","====================\n","\n","LORDY:\n","Every man that's gone hath died\n","And that's the end of all.\n","\n","GLOUCESTER:\n","Your lordship must be revenged, Hermione.\n","\n","LADY ANNE:\n","And she shall be the man that I dreamt of.\n","\n","GLOUCESTER:\n","I cannot sleep: never a waking man shall\n","Go to any sleepder than to sleep.\n","\n","LADY ANNE:\n","It shall be such a dream, I'll bear witness to it,\n","That I may believe what I say.\n","\n","GLOUCESTER:\n","All tongues to hell: never man to man,\n","Let light hear me?\n","\n","LADY ANNE:\n","No, I'll not bear witness to it.\n","\n","GLOUCESTER:\n","I said you had heard it, but could not put\n","You in words.\n","\n","LADY ANNE:\n","It is a strange tale, the tale doth it:\n","Hermione's great care is felling on wild Bolingbroke,\n","And Bolingbroke is the king.\n","\n","GLOUCESTER:\n","I could not put mine hand\n","====================\n","\n","LORD FITZWATER:\n","On pain of death, will he live to see this day\n","His country requite itself again.\n","\n","DUKE OF YORK:\n","What country?\n","\n","HENRY BOLINGBROKE:\n","Nor none but himself,\n","That is, most certainly, from this day on\n","My proudest unknown man shall be your king.\n","\n","DUKE OF AUMERLE:\n","Of all men I am the one who shall lose.\n","\n","HENRY BOLINGBROKE:\n","Take thou the crown, and crown it well\n","With all the proudest that have touch'd it.\n","\n","DUKE OF YORK:\n","I'll have the crown too, and set it on me.\n","\n","HENRY BOLINGBROKE:\n","Now York shall stand for his wish.\n","\n","DUKE OF YORK:\n","Stand you like a leader, like a great king\n","And will you yield all obedience to him?\n","\n","HENRY BOLINGBROKE:\n","I'll be king, I vow thee.\n","\n","DUKE OF YORK:\n","And shall stand for that aim so well\n","In all\n","====================\n","\n","LORD FITZWATER:\n","Sir, your company is witness to that.\n","\n","Servant:\n","The duke shall have a chamber-maid,\n","And not an officer of the people, to take\n","The absolute duke of his city.\n","\n","GLOUCESTER:\n","That's as many rogues as there are.\n","\n","Servant:\n","Nine!\n","\n","Servant:\n","Nine rogues!\n","\n","Servant:\n","Nine!\n","\n","GLOUCESTER:\n","A foul witch! Undone, go with her:\n","This jade shall be queen of the dead.\n","\n","Servant:\n","I'll not be queen.\n","\n","Servant:\n","By any other name\n","Will destroy Richard's crown,\n","And London enter in a new sense.\n","\n","Lord:\n","What is the king's name?\n","\n","Lord:\n","Richard's name; 'tis the name of the duke slain\n","In your famous battle.\n","\n","Servant:\n","Nine!\n","\n","Lord:\n","Nine!\n","\n","Servant:\n","Nine!\n","\n","GLOUCESTER:\n","The king!\n","\n","Servant:\n","Nine!\n","\n","Lord:\n","\n","====================\n","\n","LORD STANLEY:\n","Away with her!\n","\n","CATESBY:\n","What counsel pluck, you, or me, or any of your party?\n","\n","WARWICK:\n","Fear me not: I'll tell her she hath made a wrong.\n","\n","CATESBY:\n","I shall feel it, sir, when I see it.\n","\n","WARWICK:\n","Now, how shall we avoid her?\n","\n","KING HENRY VI:\n","By living by, I'll hate myself and my tongue,\n","And never bid farewell my body.\n","\n","CATESBY:\n","'Tis a disgrace to live by,\n","And I will hate myself as well as any\n","Whom fortune hath held dear dear.\n","\n","WARWICK:\n","What is the matter, my lord?\n","\n","NORTHUMBERLAND:\n","Get you hence, gentle queen, by way of reprieve:\n","The Earl of Hereford, with all the rest of Spain,\n","Staying in London till the war be done,\n","And Goverrus with the rest of England,\n","To prevent this loss of life.\n","\n","CATESBY:\n","This is the business.\n","\n","NORTHUM\n","====================\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"zjjEN2Tafhl2","colab_type":"text"},"source":["For bulk generation, you can generate a large amount of text to a file and sort out the samples locally on your computer. The next cell will generate a generated text file with a unique timestamp and then download it.\n","\n","You can rerun the cell as many times as you want for even more generated texts!"]},{"cell_type":"code","metadata":{"id":"Fa6p6arifSL0","colab_type":"code","colab":{}},"source":["gen_file = 'gpt2_gentext_{:%Y%m%d_%H%M%S}.txt'.format(datetime.utcnow())\n","\n","gpt2.generate_to_file(sess,\n","                      destination_path=gen_file,\n","                      length=500,\n","                      temperature=0.7,\n","                      nsamples=100,\n","                      batch_size=20\n","                      )\n","\n","files.download(gen_file)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ig-KVgkCDCKD","colab_type":"text"},"source":["# Etcetera\n","\n","If the notebook has errors (e.g. GPU Sync Fail or out-of-memory/OOM), force-kill the Colaboratory virtual machine and restart it with the command below:"]},{"cell_type":"code","metadata":{"id":"rIHiVP53FnsX","colab_type":"code","colab":{}},"source":["!kill -9 -1"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"wmTXWNUygS5E","colab_type":"text"},"source":["# LICENSE\n","\n","MIT License\n","\n","Copyright (c) 2019 Max Woolf\n","\n","Permission is hereby granted, free of charge, to any person obtaining a copy\n","of this software and associated documentation files (the \"Software\"), to deal\n","in the Software without restriction, including without limitation the rights\n","to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n","copies of the Software, and to permit persons to whom the Software is\n","furnished to do so, subject to the following conditions:\n","\n","The above copyright notice and this permission notice shall be included in all\n","copies or substantial portions of the Software.\n","\n","THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n","IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n","FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n","AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n","LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n","OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n","SOFTWARE."]}]}