sampling_algorithm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sampling algorithm\n",
    "---\n",
    "Implementation of Vertex's sampling algorithm described in [_Web-scale information extraction with vertex_](https://ieeexplore.ieee.org/abstract/document/5767842)\n",
    "\n",
    "## Outline\n",
    "---\n",
    "1. [Retrieve XPaths from a page](#1.-retrieve-xpaths-from-a-page)\n",
    "2. [Compute XPath weight]()\n",
    "3. [Sampling algorithm]()\n",
    "\n",
    "## 1. Retrieve XPaths from a page\n",
    "As reported in the paper:\n",
    "\n",
    "> One simple way to achieve this is to treat each page as a set of XPaths contained in it, and then greedily select pages that cover the most number of uncovered XPaths. \n",
    "\n",
    "However, the paper does not specify which Xpaths are extracted from a page. Therefore we decided to extract XPaths which retrieves textual leaf nodes.\n",
    "\n",
    "To do so we used the [lxml]() library to select all leaf textual nodes in a page. Then, using the same library we obtained the respective XPath of each leaf node previously selected. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from lxml import html\n",
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ``get_all_xpath``\n",
    "Given a page HTML source code returns a dict { _xpath_ : _value_ }, where _xpath_ is an xpath and _value_ is the string retrieved from the xpath on _src_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_all_xpath(html_src):\n",
    "        \n",
    "    # select nodes whose children include text nodes\n",
    "    XPATH_SELECTOR = \"//*[child::text()]\" \n",
    "        \n",
    "    root = html.fromstring(html_src)\n",
    "    \n",
    "    tree = root.getroottree()\n",
    "    \n",
    "    # leaf_nodes is not properly a list of all leaf nodes. \n",
    "    # It contains nodes which are parent of text elements in the DOM\n",
    "    leaf_nodes = root.xpath(XPATH_SELECTOR)\n",
    "    \n",
    "    xpath_value_dict = {}\n",
    "    \n",
    "    # extract xpath from previously selected nodes and filter out \"noisy\" nodes\n",
    "    for leaf in leaf_nodes:\n",
    "        \n",
    "        xpath = tree.getpath(leaf) + \"/text()\"\n",
    "        \n",
    "        # Filtering out xpaths which extract javascript code or css stylesheet\n",
    "        if  \"/script\" not in xpath and \"/noscript\" not in xpath and \"/style\" not in xpath:\n",
    "                        \n",
    "            selected_values = root.xpath(xpath)\n",
    "            selected_string = ''.join(selected_values).strip()\n",
    "            \n",
    "            # Filtering out xpaths which extract empty strings\n",
    "            if selected_string:\n",
    "                xpath_value_dict.update({xpath: selected_string})\n",
    "    \n",
    "    return xpath_value_dict    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Compute XPaths weights\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### get_data_structures\n",
    "Return necessary data structures for computing xpaths weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data_structures(url_to_html_map):\n",
    "        \n",
    "    url_to_xpaths = {}\n",
    "    xpath_to_value_list = defaultdict(list)\n",
    "    \n",
    "    for url in list(url_to_html_map):\n",
    "        \n",
    "        page = url_to_html_map[url]\n",
    "        xpath_to_single_value = get_all_xpath(page)\n",
    "        xpath_list = list(xpath_to_single_value)\n",
    "        url_to_xpaths[url] = xpath_list\n",
    "                \n",
    "        for xpath in xpath_to_single_value:\n",
    "            value = xpath_to_single_value[xpath]\n",
    "            xpath_to_value_list[xpath].append(value)\n",
    "    \n",
    "    return (url_to_xpaths, xpath_to_value_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute frequency\n",
    "Given a list of values extracted from a xpath _Xi_ returns the frequency of _Xi_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_frequency(values_list):\n",
    "    return len(values_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute informativeness\n",
    "Given cluster size and a list of values extracted from a xpath _Xi_ returns the informativeness of _Xi_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_informativeness(M, values_list):\n",
    "\n",
    "    values_set = set(values_list)\n",
    "    Ti = len(values_set)\n",
    "    \n",
    "    sum_F_Xi = compute_frequency(values_list)\n",
    "\n",
    "    return 1 - sum_F_Xi/(M*Ti)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### xpath weight\n",
    "Given a list of values extracted from a xpath _Xi_ returns the weight of _Xi_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def xpath_weight(cluster_size, list_of_values):\n",
    "    return compute_frequency(list_of_values)*compute_informativeness(cluster_size, list_of_values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### xpath_to_weight\n",
    "Arguments:\n",
    "- **xpath_to_values_map**: dictionary where keys are xpath and values are values retrieved from the xpath\n",
    "- **cluster_size**\n",
    "\n",
    "Returns a dictionary where keys are xpaths and values are their weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def xpath_to_weight(xpath_to_values_map, cluster_size):\n",
    "    \n",
    "    result = {}\n",
    "    for xpath in xpath_to_values_map:\n",
    "        list_of_values = xpath_to_values_map[xpath]\n",
    "        weight = xpath_weight(cluster_size, list_of_values)\n",
    "        result.update({xpath: weight})\n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### page_weight\n",
    "Arguments:\n",
    "- **list of xpath**: list of xpath of a given page\n",
    "- **xpath_to_weight_map**: dictionary where keys are xpath and values are their weights\n",
    "- **cluster_size**\n",
    "- **intersection** (optional): if None nothing happens. Otherwise only xpath in **list of xpath** $\\cap$ **intersection** will be considered in computing weight\n",
    "\n",
    "Returns page's weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def page_weight(list_of_xpath, xpath_to_weight_map, cluster_size, intersection = None):\n",
    "        \n",
    "    weight = 0\n",
    "    \n",
    "    if intersection is None:\n",
    "        intersection = list_of_xpath\n",
    "        \n",
    "    for xpath in list_of_xpath:\n",
    "        if xpath in intersection:\n",
    "            weight_of_xpath = xpath_to_weight_map[xpath]\n",
    "            weight += weight_of_xpath\n",
    "    \n",
    "    return weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Max weight page\n",
    "Arguments:\n",
    "- **url_to_xpaths_map**: dictionary where keys are urls and values are xpaths extracted from urls\n",
    "- **xpath_to_weight_map**: dictionary where keys are xpath and values are their weights\n",
    "- **cluster_size**\n",
    "- **intersection** (optional): if None nothing happens. Otherwise only xpath in **list of xpath** $\\cap$ **intersection** will be considered in computing weight\n",
    "\n",
    "Output: \n",
    "- the URL of the page with the highest weight value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def max_weight_page(url_to_xpaths_map, xpath_to_weight_map, cluster_size, intersection = None):\n",
    "        \n",
    "    max_weight = 0\n",
    "    max_weight_page = None\n",
    "    \n",
    "    for url in url_to_xpaths_map:\n",
    "        \n",
    "        xpaths = url_to_xpaths_map[url]\n",
    "        weight = page_weight(xpaths, xpath_to_weight_map, cluster_size, intersection)\n",
    "        \n",
    "        if weight > max_weight:\n",
    "            max_weight = weight\n",
    "            max_weight_page = url\n",
    "            \n",
    "    print(\"INFO\\tMax weight url is  {}\".format(max_weight_page))\n",
    "    print(\"INFO\\tMax weight is {}\".format(max_weight))\n",
    "    \n",
    "    return max_weight_page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### coverage\n",
    "Returns cluster's page coverage. TODO: add more explanations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def coverage(X, sample_pages_urls, cluster_pages_urls, url_to_xpaths_map, xpath_to_weight_map):\n",
    "    covered = 0\n",
    "    cluster_size = len(cluster_pages_urls)\n",
    "    for url in cluster_pages_urls:\n",
    "        if url not in sample_pages_urls:\n",
    "            xpaths = url_to_xpaths_map[url]\n",
    "            weight = page_weight(xpaths, xpath_to_weight_map, cluster_size, X)\n",
    "            if weight == 0:\n",
    "                covered = covered + 1\n",
    "    \n",
    "    return (covered + len(sample_pages_urls))/cluster_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#another metric to evaluate sample coverage\n",
    "def coverage2(samplePagesUrl,urlToXpathsMap,XpathNumber):\n",
    "    sampleXpathList=[]\n",
    "    for url in samplePagesUrl:\n",
    "        xpaths=urlToXpathsMap[url]\n",
    "        sampleXpathList=sampleXpathList+xpaths\n",
    "    sampleXpathSet=set(sampleXpathList)\n",
    "    return (len(sampleXpathSet)/XpathNumber)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Sampling algorithm\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from copy import copy\n",
    "\n",
    "def sampling(url_to_html_map, k = 20):\n",
    "        \n",
    "    cluster_size = len(url_to_html_map)\n",
    "    \n",
    "    cluster_pages_urls = list(url_to_html_map)\n",
    "    \n",
    "    url_to_xpaths_map, xpath_to_values_map = get_data_structures(url_to_html_map)\n",
    "    url_to_xpaths_map_copy=copy(url_to_xpaths_map)\n",
    "    \n",
    "    xpath_to_weight_map = xpath_to_weight(xpath_to_values_map, cluster_size)\n",
    "    xPathsSize=len(xpath_to_weight_map)\n",
    "    X = list(xpath_to_values_map) #insert dictionary keys into a list\n",
    "    \n",
    "    result = []\n",
    "    \n",
    "    iteration_no = 1\n",
    "    \n",
    "    while X and len(result) < k:\n",
    "        print(\"-------------------\")\n",
    "        print(\"INFO\\tIteration {}\".format(iteration_no))\n",
    "        \n",
    "        max_weight_url = max_weight_page(url_to_xpaths_map, xpath_to_weight_map, cluster_size, X)\n",
    "        result.append(max_weight_url)\n",
    "        X = [xpath for xpath in X if xpath not in url_to_xpaths_map[max_weight_url]]\n",
    "        url_to_xpaths_map.pop(max_weight_url)\n",
    "        \n",
    "        coverage_value = coverage(X, result, cluster_pages_urls, url_to_xpaths_map_copy, xpath_to_weight_map)\n",
    "        coverage2_value= coverage2(result, url_to_xpaths_map_copy, xPathsSize)\n",
    "        print(\"INFO\\tWeight based Coverage is {}\".format(coverage_value))\n",
    "        print(\"INFO\\tXpath based Coverage is {}\".format(coverage2_value))\n",
    "        iteration_no = iteration_no +1\n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "# Importing libraries\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('datasets/books_dataset.csv', nrows = 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>src</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http://www.leggereditore.it/libro.php?id=1707</td>\n",
       "      <td>b'&lt;!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http://www.leggereditore.it/libro.php?id=11236</td>\n",
       "      <td>b'&lt;!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http://www.leggereditore.it/libro.php?id=47</td>\n",
       "      <td>b'&lt;!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http://www.leggereditore.it/libro.php?id=1062</td>\n",
       "      <td>b'&lt;!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http://www.leggereditore.it/libro.php?id=780</td>\n",
       "      <td>b'&lt;!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              url  \\\n",
       "0   http://www.leggereditore.it/libro.php?id=1707   \n",
       "1  http://www.leggereditore.it/libro.php?id=11236   \n",
       "2     http://www.leggereditore.it/libro.php?id=47   \n",
       "3   http://www.leggereditore.it/libro.php?id=1062   \n",
       "4    http://www.leggereditore.it/libro.php?id=780   \n",
       "\n",
       "                                                 src  \n",
       "0  b'<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0...  \n",
       "1  b'<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0...  \n",
       "2  b'<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0...  \n",
       "3  b'<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0...  \n",
       "4  b'<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>url</th>\n",
       "      <th>src</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>100</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>100</td>\n",
       "      <td>93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>http://www.leggereditore.it/libro.php?id=11270</td>\n",
       "      <td>b'&lt;!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   url  \\\n",
       "count                                              100   \n",
       "unique                                             100   \n",
       "top     http://www.leggereditore.it/libro.php?id=11270   \n",
       "freq                                                 1   \n",
       "\n",
       "                                                      src  \n",
       "count                                                 100  \n",
       "unique                                                 93  \n",
       "top     b'<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0...  \n",
       "freq                                                    8  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_dict(df):\n",
    "    result = {}\n",
    "    for index, row in df.iterrows():\n",
    "        key = row['url']\n",
    "        value = row['src']\n",
    "        result.update({key: value})\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------\n",
      "INFO\tIteration 1\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=11113\n",
      "INFO\tMax weight is 3476.709901652295\n",
      "INFO\tWeight based Coverage is 0.01\n",
      "INFO\tXpath based Coverage is 0.34375\n",
      "-------------------\n",
      "INFO\tIteration 2\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=970\n",
      "INFO\tMax weight is 336.59158730158725\n",
      "INFO\tWeight based Coverage is 0.16\n",
      "INFO\tXpath based Coverage is 0.40625\n",
      "-------------------\n",
      "INFO\tIteration 3\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=1707\n",
      "INFO\tMax weight is 264.96000000000026\n",
      "INFO\tWeight based Coverage is 0.24\n",
      "INFO\tXpath based Coverage is 0.51875\n",
      "-------------------\n",
      "INFO\tIteration 4\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=11344\n",
      "INFO\tMax weight is 144.54644444444438\n",
      "INFO\tWeight based Coverage is 0.36\n",
      "INFO\tXpath based Coverage is 0.584375\n",
      "-------------------\n",
      "INFO\tIteration 5\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=1061\n",
      "INFO\tMax weight is 64.99499999999996\n",
      "INFO\tWeight based Coverage is 0.4\n",
      "INFO\tXpath based Coverage is 0.65625\n",
      "-------------------\n",
      "INFO\tIteration 6\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=678\n",
      "INFO\tMax weight is 49.666666666666664\n",
      "INFO\tWeight based Coverage is 0.46\n",
      "INFO\tXpath based Coverage is 0.6875\n",
      "-------------------\n",
      "INFO\tIteration 7\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=11270\n",
      "INFO\tMax weight is 44.55999999999999\n",
      "INFO\tWeight based Coverage is 0.49\n",
      "INFO\tXpath based Coverage is 0.75\n",
      "-------------------\n",
      "INFO\tIteration 8\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=59\n",
      "INFO\tMax weight is 25.727999999999994\n",
      "INFO\tWeight based Coverage is 0.53\n",
      "INFO\tXpath based Coverage is 0.778125\n",
      "-------------------\n",
      "INFO\tIteration 9\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=69\n",
      "INFO\tMax weight is 22.76999999999999\n",
      "INFO\tWeight based Coverage is 0.54\n",
      "INFO\tXpath based Coverage is 0.81875\n",
      "-------------------\n",
      "INFO\tIteration 10\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=11415\n",
      "INFO\tMax weight is 18.798\n",
      "INFO\tWeight based Coverage is 0.59\n",
      "INFO\tXpath based Coverage is 0.85\n",
      "-------------------\n",
      "INFO\tIteration 11\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=11306\n",
      "INFO\tMax weight is 10.59\n",
      "INFO\tWeight based Coverage is 0.68\n",
      "INFO\tXpath based Coverage is 0.859375\n",
      "-------------------\n",
      "INFO\tIteration 12\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=11182\n",
      "INFO\tMax weight is 9.9\n",
      "INFO\tWeight based Coverage is 0.69\n",
      "INFO\tXpath based Coverage is 0.890625\n",
      "-------------------\n",
      "INFO\tIteration 13\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=165\n",
      "INFO\tMax weight is 7.87\n",
      "INFO\tWeight based Coverage is 0.74\n",
      "INFO\tXpath based Coverage is 0.896875\n",
      "-------------------\n",
      "INFO\tIteration 14\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=805\n",
      "INFO\tMax weight is 6.91\n",
      "INFO\tWeight based Coverage is 0.76\n",
      "INFO\tXpath based Coverage is 0.915625\n",
      "-------------------\n",
      "INFO\tIteration 15\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=11412\n",
      "INFO\tMax weight is 5.94\n",
      "INFO\tWeight based Coverage is 0.77\n",
      "INFO\tXpath based Coverage is 0.934375\n",
      "-------------------\n",
      "INFO\tIteration 16\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=762\n",
      "INFO\tMax weight is 3.96\n",
      "INFO\tWeight based Coverage is 0.8\n",
      "INFO\tXpath based Coverage is 0.940625\n",
      "-------------------\n",
      "INFO\tIteration 17\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=857\n",
      "INFO\tMax weight is 3.94\n",
      "INFO\tWeight based Coverage is 0.83\n",
      "INFO\tXpath based Coverage is 0.946875\n",
      "-------------------\n",
      "INFO\tIteration 18\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=1044\n",
      "INFO\tMax weight is 2.9699999999999998\n",
      "INFO\tWeight based Coverage is 0.84\n",
      "INFO\tXpath based Coverage is 0.95625\n",
      "-------------------\n",
      "INFO\tIteration 19\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=47\n",
      "INFO\tMax weight is 2.955\n",
      "INFO\tWeight based Coverage is 0.87\n",
      "INFO\tXpath based Coverage is 0.959375\n",
      "-------------------\n",
      "INFO\tIteration 20\n",
      "INFO\tMax weight url is  http://www.leggereditore.it/libro.php?id=703\n",
      "INFO\tMax weight is 2.955\n",
      "INFO\tWeight based Coverage is 0.9\n",
      "INFO\tXpath based Coverage is 0.9625\n"
     ]
    }
   ],
   "source": [
    "cluster = create_dict(df)\n",
    "sample_pages = sampling(cluster)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}