ngs-70-hmp-diginorm.ipynb

{
 "metadata": {
  "name": "ngs-70-hmp-diginorm"
 },
 "nbformat": 3,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "source": [
      "## Running digital normalization and artifact removal an an HMP mock data set",
      "",
      "This notebook runs 'digital normalization' (see http://ged.msu.edu/papers/2012-diginorm/) and Illumina artifact removal on an HMP Illumina mock data set.  The dataset and description originated from http://www.ncbi.nlm.nih.gov/bioproject/48475.  ",
      "",
      "Prerequisites for this tutorial",
      "",
      "1. Start a new NGS 2012 EC2 instance",
      "http://ged.msu.edu/angus/tutorials-2012/start-up-an-ec2-instance.html",
      "",
      "2. Install khmer and screed",
      "In your ipython notebook homepage:  run ngs-00-update-notebooks and ngs-03-install-khmer",
      "",
      "3. Make a volume of a snapshot (snap-08efea77) containing the HMP data and mount it on your instance as /hmp-mock-tutorial",
      "",
      "",
      "We are going to start with the HMP gzipped fastq Illumina sequencing reads and:",
      "",
      "1. Normalize to coverage = 10",
      "2. Trim high-abundance (likely Illumina artifacts)",
      "",
      "This produces files that can be used as input into the partitioning algorithm (see the next notebook!)",
      "",
      ""
     ]
    },
    {
     "cell_type": "code",
     "input": [
      "cd /hmp-mock-tutorial/raw-data"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/hmp-mock-tutorial/raw-data",
        ""
       ]
      }
     ],
     "prompt_number": 1
    },
    {
     "cell_type": "code",
     "input": [
      "!ls"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "SRR172902.fastq.gz  SRR172903.fastq.gz",
        ""
       ]
      }
     ],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "source": [
      "We are going to be working with the SRR172903 dataset (staggered mixture).  You can play with a combination of the two or the other dataset (even mixture) later if you like."
     ]
    },
    {
     "cell_type": "markdown",
     "source": [
      "### Pass 1: normalize to C=10.",
      "",
      "The only parameter to change here is the memory, which is fixed at 4gb (multiply the -N and -x parameters).",
      "",
      "This should take 10-15 minutes."
     ]
    },
    {
     "cell_type": "code",
     "input": [
      "cd /hmp-mock-tutorial"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/hmp-mock-tutorial",
        ""
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "input": [
      "mkdir tutorial-files"
     ],
     "language": "python",
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "input": [
      "cd /hmp-mock-tutorial/tutorial-files"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/hmp-mock-tutorial/tutorial-files"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "",
        ""
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "source": [
      "Now, let's run the digital normalization for a word length of -k 20, we are going to remove redundant reads which are contribute to a coverage greater than -C 10.  We'll be using a hashtable size of -x 1e9 and four of them -N 4.  We'll save this hashtable as mock-pass1.kh.  The actual size of your hashtable is dependent on the characteristics of your dataset - the important thing is that the false positive rate in the hashtable (caluclated at the end) is less than 15%.  The output will be saved in pass1.report.  The input of reads is the last parameter of the command (there can be multiple read files here).  The output shows you the number of reads kept over the total of reads processed.  "
     ]
    },
    {
     "cell_type": "code",
     "input": [
      "!python /usr/local/src/khmer/scripts/normalize-by-median.py -k 20 -C 10 -N 4 -x 1e9 -s /hmp-mock-tutorial/tutorial-files/mock-pass1.kh -R /hmp-mock-tutorial/tutorial-files/pass1.report /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "",
        "PARAMETERS:",
        " - kmer size =    20 \t\t(-k)",
        " - n hashes =     4 \t\t(-N)",
        " - min hashsize = 1e+09 \t(-x)",
        "",
        "Estimated memory usage is 4e+09 bytes (n_hashes x min_hashsize)",
        "--------",
        "making hashtable",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 98471 of 100000 , or 98 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 194971 of 200000 , or 97 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 289555 of 300000 , or 96 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 383133 of 400000 , or 95 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 476015 of 500000 , or 95 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 568701 of 600000 , or 94 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 661095 of 700000 , or 94 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 753224 of 800000 , or 94 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 845404 of 900000 , or 93 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 937527 of 1000000 , or 93 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 1030022 of 1100000 , or 93 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 1121084 of 1200000 , or 93 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 1211308 of 1300000 , or 93 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 1301535 of 1400000 , or 92 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 1391544 of 1500000 , or 92 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 1480364 of 1600000 , or 92 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 1568191 of 1700000 , or 92 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 1655168 of 1800000 , or 91 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 1743478 of 1900000 , or 91 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 1828559 of 2000000 , or 91 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 1912000 of 2100000 , or 91 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 1997606 of 2200000 , or 90 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 2080707 of 2300000 , or 90 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 2164877 of 2400000 , or 90 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 2247921 of 2500000 , or 89 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 2324387 of 2600000 , or 89 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 2398885 of 2700000 , or 88 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 2471525 of 2800000 , or 88 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 2543473 of 2900000 , or 87 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 2621185 of 3000000 , or 87 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 2691512 of 3100000 , or 86 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 2771750 of 3200000 , or 86 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 2841597 of 3300000 , or 86 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 2912094 of 3400000 , or 85 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 2976001 of 3500000 , or 85 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3038330 of 3600000 , or 84 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3099250 of 3700000 , or 83 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3156056 of 3800000 , or 83 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3212197 of 3900000 , or 82 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3267587 of 4000000 , or 81 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3320406 of 4100000 , or 80 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3371773 of 4200000 , or 80 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3422626 of 4300000 , or 79 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3471702 of 4400000 , or 78 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3520891 of 4500000 , or 78 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3580270 of 4600000 , or 77 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3632578 of 4700000 , or 77 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3681283 of 4800000 , or 76 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3737769 of 4900000 , or 76 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3796621 of 5000000 , or 75 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3841452 of 5100000 , or 75 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3883343 of 5200000 , or 74 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3926327 of 5300000 , or 74 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 3967557 of 5400000 , or 73 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4007282 of 5500000 , or 72 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4046391 of 5600000 , or 72 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4084792 of 5700000 , or 71 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4122792 of 5800000 , or 71 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4159952 of 5900000 , or 70 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4196225 of 6000000 , or 69 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4231920 of 6100000 , or 69 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4267195 of 6200000 , or 68 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4301540 of 6300000 , or 68 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4335437 of 6400000 , or 67 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4369518 of 6500000 , or 67 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4402650 of 6600000 , or 66 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4434925 of 6700000 , or 66 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4467070 of 6800000 , or 65 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4499248 of 6900000 , or 65 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4531198 of 7000000 , or 64 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4562298 of 7100000 , or 64 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4593518 of 7200000 , or 63 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4624607 of 7300000 , or 63 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4655495 of 7400000 , or 62 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4686324 of 7500000 , or 62 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4717418 of 7600000 , or 62 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4747864 of 7700000 , or 61 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4780874 of 7800000 , or 61 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... kept 4813815 of 7900000 , or 60 %",
        "... in file /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "DONE with /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz ; kept 4823693 of 7932819 or 60 %",
        "output in SRR172903.fastq.gz.keep",
        "Saving hashfile through /hmp-mock-tutorial/raw-data/SRR172903.fastq.gz",
        "...saving to /hmp-mock-tutorial/tutorial-files/mock-pass1.kh",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "fp rate estimated to be 0.000",
        ""
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "source": [
      "Check out the fp rate.  We're good to go...  What output files were produced?"
     ]
    },
    {
     "cell_type": "code",
     "input": [
      "!ls"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "mock-pass1.kh  pass1.report  SRR172903.fastq.gz.keep",
        ""
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "input": [
      "!head pass1.report"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "100000 98471 0.98471",
        "200000 194971 0.974855",
        "300000 289555 0.965183333333",
        "400000 383133 0.9578325",
        "500000 476015 0.95203",
        "600000 568701 0.947835",
        "700000 661095 0.944421428571",
        "800000 753224 0.94153",
        "900000 845404 0.939337777778",
        "1000000 937527 0.937527",
        ""
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "source": [
      "You can make a graph of the rate at which diginorm eliminates reads.  What does this tell you about the diversity (or redundancy of your dataset?)"
     ]
    },
    {
     "cell_type": "markdown",
     "source": [
      "### Pass 2: Trim off high-abundance k-mers",
      "",
      "These are likely to be the results of Illumina crap.  Partitioning (what we do next) is much harder if you don't do this, and the assemblies may be less good.  We'll be trimming sequences where we find k-mers which are present in our dataset at a coverage greater than 50.  Note, that we should not do this without doing digital normalization first, why?",
      "",
      "This should take ~10-15 minutes."
     ]
    },
    {
     "cell_type": "code",
     "input": [
      "!cd /hmp-mock-tutorial/tutorial-files/",
      ""
     ],
     "language": "python",
     "outputs": [],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "input": [
      "!python /usr/local/src/khmer/sandbox/filter-below-abund.py /hmp-mock-tutorial/tutorial-files/mock-pass1.kh /hmp-mock-tutorial/tutorial-files/*.keep"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "file with ht: /hmp-mock-tutorial/tutorial-files/mock-pass1.kh",
        "-- settings:",
        "N THREADS 8",
        "--",
        "making hashtable",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "filtering /hmp-mock-tutorial/tutorial-files/SRR172903.fastq.gz.keep",
        "starting threads",
        "starting writer",
        "loading...",
        "... filtering 0",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 500000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "processed 500000 / wrote 497528 / removed 2472",
        "processed 37530000 bp / wrote 36322889 bp / removed 1207111 bp",
        "discarded 3.2%",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 600000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 700000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 800000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 900000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 1000000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "processed 1000000 / wrote 995045 / removed 4955",
        "processed 75007500 bp / wrote 72670517 bp / removed 2336983 bp",
        "discarded 3.1%",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 1100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 1200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 1300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 1400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 1500000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "processed 1500000 / wrote 1467162 / removed 32838",
        "processed 112552500 bp / wrote 107221701 bp / removed 5330799 bp",
        "discarded 4.7%",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 1600000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 1700000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 1800000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 1900000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 2000000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "processed 2000000 / wrote 1929021 / removed 70979",
        "processed 150045000 bp / wrote 141062463 bp / removed 8982537 bp",
        "discarded 6.0%",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 2100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 2200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 2300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 2400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 2500000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "processed 2500000 / wrote 2403245 / removed 96755",
        "processed 187530000 bp / wrote 175725490 bp / removed 11804510 bp",
        "discarded 6.3%",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 2600000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 2700000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 2800000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 2900000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 3000000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "processed 3000000 / wrote 2845946 / removed 154054",
        "processed 225052500 bp / wrote 208177480 bp / removed 16875020 bp",
        "discarded 7.5%",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 3100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 3200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 3300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 3400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 3500000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "processed 3500000 / wrote 3343199 / removed 156801",
        "processed 262545000 bp / wrote 244592062 bp / removed 17952938 bp",
        "discarded 6.8%",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 3600000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 3700000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 3800000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 3900000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 4000000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "processed 4000000 / wrote 3802478 / removed 197522",
        "processed 300090000 bp / wrote 278301488 bp / removed 21788512 bp",
        "discarded 7.3%",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 4100000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 4200000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 4300000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 4400000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 4500000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "processed 4500000 / wrote 4298953 / removed 201047",
        "processed 337680000 bp / wrote 314656018 bp / removed 23023982 bp",
        "discarded 6.8%",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 4600000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 4700000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "... filtering 4800000",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "done loading in sequences",
        ""
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "DONE writing.",
        "processed 4815993 / wrote 4613117 / removed 202876",
        "processed 361424475 bp / wrote 337650426 bp / removed 23774049 bp",
        "discarded 6.6%",
        ""
       ]
      }
     ],
     "prompt_number": 10
    },
    {
     "cell_type": "markdown",
     "source": [
      "This needs to be fixed later but for some reason our output filtered file is being saved somewhere else...we'll fix this later but for now let's just go move it to where I want it."
     ]
    },
    {
     "cell_type": "code",
     "input": [
      "cd /hmp-mock-tutorial/tutorial-files/"
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/hmp-mock-tutorial/tutorial-files",
        ""
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "input": [
      "ls",
      ""
     ],
     "language": "python",
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "mock-pass1.kh  SRR172903.fastq.gz.keep",
        "pass1.report   SRR172903.fastq.gz.keep.below",
        ""
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "markdown",
     "source": [
      "Okay, now we have a set of reads which are can be assembled -- we've normalized the coverage and removed artifacts.  Sometimes, you may have a dataset that is too large to assemble on your computational resources.  If this is the case, you can partition your reads by connectivity.  That is in the next notebook, hmp-partitioning."
     ]
    },
    {
     "cell_type": "code",
     "input": [
      ""
     ],
     "language": "python",
     "outputs": []
    }
   ]
  }
 ]
}