3a_retrieval1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data retrieval I\n",
    "\n",
    "In this notebook, we will work with the following:\n",
    "\n",
    "- Web scraping process.\n",
    "- Read one page.\n",
    "- Find the content we want.\n",
    "- Automate many pages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import pandas as pd\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web scraping\n",
    "\n",
    "One helpful way of gathering text data is web scraping.\n",
    "We usually do this in three steps:\n",
    "\n",
    "1. Retrieve the pages with information we want.\n",
    "1. Extract the data from the pages.\n",
    "1. Clean and save the resulting data.\n",
    "\n",
    "Let's walk through an example of getting press releases from the [Microsoft website](https://news.microsoft.com/category/press-releases/).\n",
    "\n",
    "I often prefer to work out of order as follows:\n",
    "\n",
    "1. Figure out how to extract data from one page that has the data.\n",
    "1. Then, figure out how to automate getting the pages of interest.\n",
    "1. Run those pages through the procedure in step 1.\n",
    "1. Clean and save.\n",
    "\n",
    "This has the benefit of solving what is usually the hardest problem first."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Important note\n",
    "\n",
    "As you'll see, the difficulty ramps up a lot here.\n",
    "Web scraping is easily a full day topic on its own.\n",
    "Hence, I have two main goals for you:\n",
    "\n",
    "1. Get a sense of the logic and the process in solving the problem. This is a good start if you want to learn it yourself.\n",
    "1. Understand what is feasible and achievable. This helps whether you do it yourself or farm it out (and there's a ready talent pool for this)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read one page\n",
    "\n",
    "This is the hardest part.\n",
    "\n",
    "Note that we add a user agent header that is sent as part of the request.\n",
    "The reason is that a lot of web servers block user agents that are web scraping tools."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "_AGENT= 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0'\n",
    "\n",
    "pr_url_1 = 'https://news.microsoft.com/2018/10/04/redline-communications-and-microsoft-announce-partnership-to-lower-the-cost-of-tv-white-space-solutions/'\n",
    "pr_req_1 = requests.get(pr_url_1, headers={'User-Agent': _AGENT})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We want this to be 200, which is the code for OK.\n",
    "pr_req_1.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The .text attribute of the request object is the HTML of the page.\n",
    "pr_soup_1 = BeautifulSoup(pr_req_1.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<meta content=\"2018-10-04T13:00:35+00:00\" property=\"article:published_time\"/>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The meta tags have some data we'd like to get.\n",
    "# For example, this is the published time.\n",
    "pr_soup_1.find('meta', property='article:published_time')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'article:published_time'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can get the property attribute of this meta tag, which has the name of the data item.\n",
    "pr_soup_1.find('meta', property='article:published_time')['property']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2018-10-04T13:00:35+00:00'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The content attribute has the data item itself.\n",
    "pr_soup_1.find('meta', property='article:published_time')['content']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# List of meta tags to get.\n",
    "# Note: when in doubt, get everything you might possibly use.\n",
    "#       It's easier to drop stuff than to re-scrape everything.\n",
    "\n",
    "_METAS = [\n",
    "    'article:published_time',\n",
    "    'article:modified_time',\n",
    "    'og:title',\n",
    "    'og:description',\n",
    "    'og:updated_time',\n",
    "    'og:url',\n",
    "    'article:section'\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This loop populates a dict with each of the meta attributes above and its content.\n",
    "pr_data_1 = {}\n",
    "for meta in _METAS:\n",
    "    prop = pr_soup_1.find('meta', property=meta)['property']\n",
    "    content = pr_soup_1.find('meta', property=meta)['content']\n",
    "    pr_data_1.update({prop: content})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'article:published_time': '2018-10-04T13:00:35+00:00',\n",
       " 'article:modified_time': '2018-10-04T14:43:59+00:00',\n",
       " 'og:title': 'Redline Communications and Microsoft announce partnership to lower the cost of TV White Space solutions - Stories',\n",
       " 'og:description': 'The partnership will help make broadband more affordable and accessible for unserved communities in rural areas of the U.S. and globally REDMOND, Wash. — Oct. 4, 2018 — On Thursday, Redline Communications (TSX:RDL) and Microsoft Corp. announced a new partnership that will help address the rural broadband gap using TV White Space technology. Redline, a […ellipsis]',\n",
       " 'og:updated_time': '2018-10-04T14:43:59+00:00',\n",
       " 'og:url': 'https://news.microsoft.com/2018/10/04/redline-communications-and-microsoft-announce-partnership-to-lower-the-cost-of-tv-white-space-solutions/',\n",
       " 'article:section': 'Press Releases'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pr_data_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The partnership will help make broadband more affordable and accessible for unserved communities in rural areas of the U.S. and globally'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pr_soup_1.find('div', {'class': 'entry-content m-blog-content'}).find('h3').text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "pr_data_1['h3'] = pr_soup_1.find('div', \n",
    "                                 {'class': 'entry-content m-blog-content'}\n",
    "                                ).find('h3').text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'article:published_time': '2018-10-04T13:00:35+00:00',\n",
       " 'article:modified_time': '2018-10-04T14:43:59+00:00',\n",
       " 'og:title': 'Redline Communications and Microsoft announce partnership to lower the cost of TV White Space solutions - Stories',\n",
       " 'og:description': 'The partnership will help make broadband more affordable and accessible for unserved communities in rural areas of the U.S. and globally REDMOND, Wash. — Oct. 4, 2018 — On Thursday, Redline Communications (TSX:RDL) and Microsoft Corp. announced a new partnership that will help address the rural broadband gap using TV White Space technology. Redline, a […ellipsis]',\n",
       " 'og:updated_time': '2018-10-04T14:43:59+00:00',\n",
       " 'og:url': 'https://news.microsoft.com/2018/10/04/redline-communications-and-microsoft-announce-partnership-to-lower-the-cost-of-tv-white-space-solutions/',\n",
       " 'article:section': 'Press Releases',\n",
       " 'h3': 'The partnership will help make broadband more affordable and accessible for unserved communities in rural areas of the U.S. and globally'}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pr_data_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p><strong>REDMOND, Wash. </strong><strong>—</strong> <strong>Oct. 4, 2018</strong> <strong>—</strong> On Thursday, <a href=\"https://rdlcom.com/\">Redline Communications</a> (TSX:RDL) and <a href=\"https://www.microsoft.com/en-us/\">Microsoft Corp.</a> announced a new partnership that will help address the rural broadband gap using TV White Space technology. Redline, a leader in private wireless networks, will provide its Virtual Fiber™ radio technology in the TV White Space band to Microsoft Airband Initiative partners. Together, Redline and Microsoft’s partnership will help make broadband internet more affordable and accessible to unserved and underserved customers in rural areas in the United States and globally.</p>,\n",
       " <p>New cloud services and other technologies make broadband connectivity a necessity to start and grow a small business and to take advantage of advances in agriculture, telemedicine and education. It is a vital part of 21st century infrastructure. Yet, more than 19.4 million Americans living in rural areas lack access to broadband and miss out on the opportunities that connectivity provides. According to a study from <a href=\"https://blogs.microsoft.com/uploads/2017/07/Rural-Broadband-Strategy-Microsoft-Whitepaper-FINAL-7-10-17.pdf\">Boston Consulting Group</a>, a connectivity model that uses a combination of technologies, including TV White Space, can reduce the cost of extending broadband coverage in rural communities. TV White Space is an important part of the solution, creating broadband connections in UHF bands and enabling communication in challenging rural terrains and highly vegetated areas, all while protecting broadcasters and other licensees from harmful interference.</p>,\n",
       " <p>This partnership is part of Microsoft’s <a href=\"https://news.microsoft.com/rural-broadband/\">Airband Initiative</a>, which aims to close the broadband gap by extending access to 2 million unserved people in rural America by 2022, using a mix of technology solutions, including TV White Space.</p>,\n",
       " <p>“Our work with Redline will increase the availability of competitively priced TV White Space technology, enabling internet service providers (ISPs) to provide access to customers at an affordable price point,” said Paul Garnett, senior director of the Microsoft Airband Initiative. “This availability and utilization of TV White Space is absolutely critical to closing the broadband gap. This partnership will bring rapid evolution to the technology, making a real impact on real lives.”</p>,\n",
       " <p>“With its Virtual Fiber™ technology, Redline has been leading the TV White Space market and has been active in digital divide projects for almost a decade,” said Rob Williams, CEO at Redline. “In discussions with Microsoft, we realized that we shared the same vision for approaching the rural broadband gap, and we each possessed critical components to the solution. This partnership with Microsoft will help us address the digital divide more effectively in the U.S. and around the world.”</p>,\n",
       " <p>In addition to the partnerships with companies like Redline, Microsoft’s <a href=\"https://news.microsoft.com/rural-broadband/\">Airband Initiative</a> invests in partnerships with ISPs and other telecommunications companies, introduces innovative solutions for rural connectivity, and provides digital skills training for people in newly connected communities.</p>,\n",
       " <p><strong>About Redline Communications</strong></p>,\n",
       " <p>Redline Communications (TSX:RDL) designs and manufactures powerful wide-area wireless networks for mission-critical applications in challenging locations. Redline networks are used by Oil &amp; Gas companies onshore and offshore, Mining companies on surface and underground operations, by municipalities to remotely monitor infrastructure, and by specialized telecom service providers to deliver premium services. Thousands of businesses worldwide rely on Redline to engineer, plan and deliver ruggedized, secure and reliable networks for their IoT, voice, data and video communications needs. For more information visit <a href=\"http://www.rdlcom.com/\">www.rdlcom.com</a>.</p>,\n",
       " <p><strong>About Microsoft</strong></p>,\n",
       " <p>Microsoft (Nasdaq “MSFT” @microsoft) enables digital transformation for the era of an intelligent cloud and an intelligent edge. Its mission is to empower every person and every organization on the planet to achieve more.</p>,\n",
       " <p><strong>For more information, press only:<br/>\n",
       " </strong>Microsoft Media Relations, WE Communications for Microsoft, (425) 638-7777,</p>,\n",
       " <p>rrt@we-worldwide.com</p>,\n",
       " <p>Louis Lambert, Senior VP Business Development and Marketing for Redline,</p>,\n",
       " <p>+1 (905) 479-8344, <a href=\"mailto:llambert@rdlcom.com\">llambert@rdlcom.com</a></p>,\n",
       " <p>Cory Pala, Investor Relations for Redline, +1 (416) 657-2400, <a href=\"mailto:cpala@rdlcom.com\">cpala@rdlcom.com</a></p>,\n",
       " <p><em>Note to editors</em><em>:</em> For more information, news and perspectives from Microsoft, please visit the Microsoft News Center at <a href=\"https://news.microsoft.com/\">http://news.microsoft.com</a>. Web links, telephone numbers and titles were correct at time of publication, but may have changed. For additional assistance, journalists and analysts may contact Microsoft’s Rapid Response Team or other appropriate contacts listed at <a href=\"https://news.microsoft.com/microsoft-public-relations-contacts/\">https://news.microsoft.com/microsoft-public-relations-contacts</a>.</p>,\n",
       " <p> </p>]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pr_soup_1.find('div', \n",
    "               {'class': 'entry-content m-blog-content'}\n",
    "              ).find_all('p')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a little gnarly.\n",
    "pr_data_1['body'] = '\\n\\n'.join(\n",
    "                        [i.text for i in pr_soup_1.find(\n",
    "                            'div', \n",
    "                            {'class': 'entry-content m-blog-content'}\n",
    "                            ).find_all('p')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'article:published_time': '2018-10-04T13:00:35+00:00',\n",
       " 'article:modified_time': '2018-10-04T14:43:59+00:00',\n",
       " 'og:title': 'Redline Communications and Microsoft announce partnership to lower the cost of TV White Space solutions - Stories',\n",
       " 'og:description': 'The partnership will help make broadband more affordable and accessible for unserved communities in rural areas of the U.S. and globally REDMOND, Wash. — Oct. 4, 2018 — On Thursday, Redline Communications (TSX:RDL) and Microsoft Corp. announced a new partnership that will help address the rural broadband gap using TV White Space technology. Redline, a […ellipsis]',\n",
       " 'og:updated_time': '2018-10-04T14:43:59+00:00',\n",
       " 'og:url': 'https://news.microsoft.com/2018/10/04/redline-communications-and-microsoft-announce-partnership-to-lower-the-cost-of-tv-white-space-solutions/',\n",
       " 'article:section': 'Press Releases',\n",
       " 'h3': 'The partnership will help make broadband more affordable and accessible for unserved communities in rural areas of the U.S. and globally',\n",
       " 'body': 'REDMOND, Wash. — Oct. 4, 2018 — On Thursday, Redline Communications (TSX:RDL) and Microsoft Corp. announced a new partnership that will help address the rural broadband gap using TV White Space technology. Redline, a leader in private wireless networks, will provide its Virtual Fiber™ radio technology in the TV White Space band to Microsoft Airband Initiative partners. Together, Redline and Microsoft’s partnership will help make broadband internet more affordable and accessible to unserved and underserved customers in rural areas in the United States and globally.\\n\\nNew cloud services and other technologies make broadband connectivity a necessity to start and grow a small business and to take advantage of advances in agriculture, telemedicine and education. It is a vital part of 21st century infrastructure. Yet, more than 19.4 million Americans living in rural areas lack access to broadband and miss out on the opportunities that connectivity provides. According to a study from Boston Consulting Group, a connectivity model that uses a combination of technologies, including TV White Space, can reduce the cost of extending broadband coverage in rural communities. TV White Space is an important part of the solution, creating broadband connections in UHF bands and enabling communication in challenging rural terrains and highly vegetated areas, all while protecting broadcasters and other licensees from harmful interference.\\n\\nThis partnership is part of Microsoft’s Airband Initiative, which aims to close the broadband gap by extending access to 2 million unserved people in rural America by 2022, using a mix of technology solutions, including TV White Space.\\n\\n“Our work with Redline will increase the availability of competitively priced TV White Space technology, enabling internet service providers (ISPs) to provide access to customers at an affordable price point,” said Paul Garnett, senior director of the Microsoft Airband Initiative. “This availability and utilization of TV White Space is absolutely critical to closing the broadband gap. This partnership will bring rapid evolution to the technology, making a real impact on real lives.”\\n\\n“With its Virtual Fiber™ technology, Redline has been leading the TV White Space market and has been active in digital divide projects for almost a decade,” said Rob Williams, CEO at Redline.\\xa0“In discussions with Microsoft, we realized that we shared the same vision for approaching the rural broadband gap, and we each possessed critical components to the solution. This partnership with Microsoft will help us address the digital divide more effectively in the U.S. and around the world.”\\n\\nIn addition to the partnerships with companies like Redline, Microsoft’s\\xa0Airband Initiative\\xa0invests in partnerships with ISPs and other telecommunications companies, introduces innovative solutions for rural connectivity, and provides digital skills training for people in newly connected communities.\\n\\nAbout Redline Communications\\n\\nRedline Communications (TSX:RDL) designs and manufactures powerful wide-area wireless networks for mission-critical applications in challenging locations. Redline networks are used by Oil & Gas companies onshore and offshore, Mining companies on surface and underground operations, by municipalities to remotely monitor infrastructure, and by specialized telecom service providers to deliver premium services. Thousands of businesses worldwide rely on Redline to engineer, plan and deliver ruggedized, secure and reliable networks for their IoT, voice, data and video communications needs. For more information visit\\xa0www.rdlcom.com.\\n\\nAbout Microsoft\\n\\nMicrosoft (Nasdaq “MSFT” @microsoft) enables digital transformation for the era of an intelligent cloud and an intelligent edge. Its mission is to empower every person and every organization on the planet to achieve more.\\n\\nFor more information, press only:\\nMicrosoft Media Relations, WE Communications for Microsoft, (425) 638-7777,\\n\\nrrt@we-worldwide.com\\n\\nLouis Lambert, Senior VP\\u202fBusiness Development\\u202fand Marketing for Redline,\\n\\n+1 (905) 479-8344, llambert@rdlcom.com\\n\\nCory Pala, Investor Relations for Redline, +1 (416) 657-2400, cpala@rdlcom.com\\n\\nNote to editors: For more information, news and perspectives from Microsoft, please visit the Microsoft News Center at http://news.microsoft.com. Web links, telephone numbers and titles were correct at time of publication, but may have changed. For additional assistance, journalists and analysts may contact Microsoft’s Rapid Response Team or other appropriate contacts listed at https://news.microsoft.com/microsoft-public-relations-contacts.\\n\\n\\xa0'}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pr_data_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automate our one page work.\n",
    "\n",
    "This is fairly easy. We have the code for it already.\n",
    "We just need to wrap it in a function.\n",
    "\n",
    "**Note:** I'm using an `if` statement to check whether these properties exist, and guarding against the case where they don't.\n",
    "I did this iteratively while building this content, because I noticed (from errors) that many press releases do not have modification dates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data_from_soup(soup):\n",
    "    data = {}\n",
    "    for meta in _METAS:\n",
    "        if soup.find('meta', property=meta) is not None:\n",
    "            prop = soup.find('meta', property=meta)['property']\n",
    "        if soup.find('meta', property=meta) is not None:\n",
    "            content = soup.find('meta', property=meta)['content']\n",
    "        if prop is not None and content is not None:\n",
    "            data.update({prop: content})\n",
    "    try:\n",
    "        data['h3'] = soup.find('div', \n",
    "                               {'class': 'entry-content m-blog-content'}\n",
    "                              ).find('h3').string\n",
    "    except AttributeError:\n",
    "        data['h3'] = ''\n",
    "    \n",
    "    data['body'] = '\\n\\n'.join(\n",
    "                        [i.text for i in soup.find(\n",
    "                            'div', \n",
    "                            {'class': 'entry-content m-blog-content'}\n",
    "                            ).find_all('p')])\n",
    "    \n",
    "    return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'article:published_time': '2018-10-04T13:00:35+00:00',\n",
       " 'article:modified_time': '2018-10-04T14:43:59+00:00',\n",
       " 'og:title': 'Redline Communications and Microsoft announce partnership to lower the cost of TV White Space solutions - Stories',\n",
       " 'og:description': 'The partnership will help make broadband more affordable and accessible for unserved communities in rural areas of the U.S. and globally REDMOND, Wash. — Oct. 4, 2018 — On Thursday, Redline Communications (TSX:RDL) and Microsoft Corp. announced a new partnership that will help address the rural broadband gap using TV White Space technology. Redline, a […ellipsis]',\n",
       " 'og:updated_time': '2018-10-04T14:43:59+00:00',\n",
       " 'og:url': 'https://news.microsoft.com/2018/10/04/redline-communications-and-microsoft-announce-partnership-to-lower-the-cost-of-tv-white-space-solutions/',\n",
       " 'article:section': 'Press Releases',\n",
       " 'h3': 'The partnership will help make broadband more affordable and accessible for unserved communities in rural areas of the U.S. and globally',\n",
       " 'body': 'REDMOND, Wash. — Oct. 4, 2018 — On Thursday, Redline Communications (TSX:RDL) and Microsoft Corp. announced a new partnership that will help address the rural broadband gap using TV White Space technology. Redline, a leader in private wireless networks, will provide its Virtual Fiber™ radio technology in the TV White Space band to Microsoft Airband Initiative partners. Together, Redline and Microsoft’s partnership will help make broadband internet more affordable and accessible to unserved and underserved customers in rural areas in the United States and globally.\\n\\nNew cloud services and other technologies make broadband connectivity a necessity to start and grow a small business and to take advantage of advances in agriculture, telemedicine and education. It is a vital part of 21st century infrastructure. Yet, more than 19.4 million Americans living in rural areas lack access to broadband and miss out on the opportunities that connectivity provides. According to a study from Boston Consulting Group, a connectivity model that uses a combination of technologies, including TV White Space, can reduce the cost of extending broadband coverage in rural communities. TV White Space is an important part of the solution, creating broadband connections in UHF bands and enabling communication in challenging rural terrains and highly vegetated areas, all while protecting broadcasters and other licensees from harmful interference.\\n\\nThis partnership is part of Microsoft’s Airband Initiative, which aims to close the broadband gap by extending access to 2 million unserved people in rural America by 2022, using a mix of technology solutions, including TV White Space.\\n\\n“Our work with Redline will increase the availability of competitively priced TV White Space technology, enabling internet service providers (ISPs) to provide access to customers at an affordable price point,” said Paul Garnett, senior director of the Microsoft Airband Initiative. “This availability and utilization of TV White Space is absolutely critical to closing the broadband gap. This partnership will bring rapid evolution to the technology, making a real impact on real lives.”\\n\\n“With its Virtual Fiber™ technology, Redline has been leading the TV White Space market and has been active in digital divide projects for almost a decade,” said Rob Williams, CEO at Redline.\\xa0“In discussions with Microsoft, we realized that we shared the same vision for approaching the rural broadband gap, and we each possessed critical components to the solution. This partnership with Microsoft will help us address the digital divide more effectively in the U.S. and around the world.”\\n\\nIn addition to the partnerships with companies like Redline, Microsoft’s\\xa0Airband Initiative\\xa0invests in partnerships with ISPs and other telecommunications companies, introduces innovative solutions for rural connectivity, and provides digital skills training for people in newly connected communities.\\n\\nAbout Redline Communications\\n\\nRedline Communications (TSX:RDL) designs and manufactures powerful wide-area wireless networks for mission-critical applications in challenging locations. Redline networks are used by Oil & Gas companies onshore and offshore, Mining companies on surface and underground operations, by municipalities to remotely monitor infrastructure, and by specialized telecom service providers to deliver premium services. Thousands of businesses worldwide rely on Redline to engineer, plan and deliver ruggedized, secure and reliable networks for their IoT, voice, data and video communications needs. For more information visit\\xa0www.rdlcom.com.\\n\\nAbout Microsoft\\n\\nMicrosoft (Nasdaq “MSFT” @microsoft) enables digital transformation for the era of an intelligent cloud and an intelligent edge. Its mission is to empower every person and every organization on the planet to achieve more.\\n\\nFor more information, press only:\\nMicrosoft Media Relations, WE Communications for Microsoft, (425) 638-7777,\\n\\nrrt@we-worldwide.com\\n\\nLouis Lambert, Senior VP\\u202fBusiness Development\\u202fand Marketing for Redline,\\n\\n+1 (905) 479-8344, llambert@rdlcom.com\\n\\nCory Pala, Investor Relations for Redline, +1 (416) 657-2400, cpala@rdlcom.com\\n\\nNote to editors: For more information, news and perspectives from Microsoft, please visit the Microsoft News Center at http://news.microsoft.com. Web links, telephone numbers and titles were correct at time of publication, but may have changed. For additional assistance, journalists and analysts may contact Microsoft’s Rapid Response Team or other appropriate contacts listed at https://news.microsoft.com/microsoft-public-relations-contacts.\\n\\n\\xa0'}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Notice how easy this is once we make a function.\n",
    "get_data_from_soup(pr_soup_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read many pages\n",
    "\n",
    "Now we need to get the URLs for all of the pages we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "many_pr_url_1 = 'https://news.microsoft.com/category/press-releases/'\n",
    "many_pr_page_1 = requests.get(many_pr_url_1, headers={'User-Agent': _AGENT}).text\n",
    "many_pr_soup_1 = BeautifulSoup(many_pr_page_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<a class=\"f-post-link c-heading-6 m-chevron\" href=\"https://news.microsoft.com/2019/09/24/microsoft-and-engie-announce-innovative-renewable-initiatives/\" ms.title=\"Microsoft and ENGIE announce innovative renewable initiatives\" rel=\"bookmark\">\n",
       " \t\tMicrosoft and ENGIE announce innovative renewable initiatives\t</a>,\n",
       " <a class=\"f-post-link c-heading-6 m-chevron\" href=\"https://news.microsoft.com/2019/09/23/ste-michelle-wine-estates-leverages-microsoft-dynamics-365-to-transform-customer-experience/\" ms.title=\"Ste. Michelle Wine Estates leverages Microsoft Dynamics 365 to transform customer experience\" rel=\"bookmark\">\n",
       " \t\tSte. Michelle Wine Estates leverages Microsoft Dynamics 365 to transform customer experience\t</a>,\n",
       " <a class=\"f-post-link c-heading-6 m-chevron\" href=\"https://news.microsoft.com/2019/09/19/microsoft-announces-board-changes-and-proposes-election-of-new-board-member/\" ms.title=\"Microsoft announces board changes and proposes election of new board member\" rel=\"bookmark\">\n",
       " \t\tMicrosoft announces board changes and proposes election of new board member\t</a>,\n",
       " <a class=\"f-post-link c-heading-6 m-chevron\" href=\"https://news.microsoft.com/2019/09/18/microsoft-announces-quarterly-dividend-increase-and-new-share-repurchase-program/\" ms.title=\"Microsoft announces quarterly dividend increase and new share repurchase program\" rel=\"bookmark\">\n",
       " \t\tMicrosoft announces quarterly dividend increase and new share repurchase program\t</a>,\n",
       " <a class=\"f-post-link c-heading-6 m-chevron\" href=\"https://news.microsoft.com/2019/09/18/nextlink-internet-and-microsoft-closing-broadband-gap-in-central-us/\" ms.title=\"Nextlink Internet and Microsoft closing broadband gap in central US\" rel=\"bookmark\">\n",
       " \t\tNextlink Internet and Microsoft closing broadband gap in central US\t</a>,\n",
       " <a class=\"f-post-link c-heading-6 m-chevron\" href=\"https://news.microsoft.com/2019/09/17/schlumberger-chevron-and-microsoft-announce-collaboration-to-accelerate-digital-transformation/\" ms.title=\"Schlumberger, Chevron and Microsoft announce collaboration to accelerate digital transformation\" rel=\"bookmark\">\n",
       " \t\tSchlumberger, Chevron and Microsoft announce collaboration to accelerate digital transformation\t</a>,\n",
       " <a class=\"f-post-link c-heading-6 m-chevron\" href=\"https://news.microsoft.com/2019/09/13/microsoft-and-the-walt-disney-studios-to-develop-scene-to-screen-content-workflows/\" ms.title=\"Microsoft and The Walt Disney Studios to develop ‘scene-to-screen’ content workflows\" rel=\"bookmark\">\n",
       " \t\tMicrosoft and The Walt Disney Studios to develop ‘scene-to-screen’ content workflows\t</a>,\n",
       " <a class=\"f-post-link c-heading-6 m-chevron\" href=\"https://news.microsoft.com/2019/08/12/jio-and-microsoft-announce-alliance-to-accelerate-digital-transformation-in-india/\" ms.title=\"Jio and Microsoft announce alliance to accelerate digital transformation in India\" rel=\"bookmark\">\n",
       " \t\tJio and Microsoft announce alliance to accelerate digital transformation in India\t</a>,\n",
       " <a class=\"f-post-link c-heading-6 m-chevron\" href=\"https://news.microsoft.com/2019/07/30/microsoft-announces-iot-signals-research-report-on-state-of-iot-adoption/\" ms.title=\"Microsoft announces IoT Signals research report on state of IoT adoption\" rel=\"bookmark\">\n",
       " \t\tMicrosoft announces IoT Signals research report on state of IoT adoption\t</a>,\n",
       " <a class=\"f-post-link c-heading-6 m-chevron\" href=\"https://news.microsoft.com/2019/07/22/openai-forms-exclusive-computing-partnership-with-microsoft-to-build-new-azure-ai-supercomputing-technologies/\" ms.title=\"OpenAI forms exclusive computing partnership with Microsoft to build new Azure AI supercomputing technologies\" rel=\"bookmark\">\n",
       " \t\tOpenAI forms exclusive computing partnership with Microsoft to build new Azure AI supercomputing technologies\t</a>,\n",
       " <a class=\"f-post-link c-heading-6 m-chevron\" href=\"https://news.microsoft.com/2019/07/18/microsoft-earnings-press-release-available-on-investor-relations-website-3/\" ms.title=\"Microsoft earnings press release available on Investor Relations website\" rel=\"bookmark\">\n",
       " \t\tMicrosoft earnings press release available on Investor Relations website\t</a>,\n",
       " <a class=\"f-post-link c-heading-6 m-chevron\" href=\"https://news.microsoft.com/2019/07/18/microsoft-cloud-powers-record-fourth-quarter-results/\" ms.title=\"Microsoft Cloud powers record fourth quarter results\" rel=\"bookmark\">\n",
       " \t\tMicrosoft Cloud powers record fourth quarter results\t</a>,\n",
       " <a aria-label=\"Go to page 2 of results\" href=\"https://news.microsoft.com/category/press-releases/page/2/\" ms.title=\"2\">2</a>,\n",
       " <a aria-label=\"Go to page 3 of results\" href=\"https://news.microsoft.com/category/press-releases/page/3/\" ms.title=\"3\">3</a>,\n",
       " <a aria-label=\"Go to page 841 of results\" href=\"https://news.microsoft.com/category/press-releases/page/841/\" ms.title=\"841\">841</a>,\n",
       " <a aria-label=\"Go to next page\" class=\"c-glyph\" href=\"/category/press-releases/?paged=2\" ms.title=\"Next Page\">Next</a>]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Almost, but note the ones at the bottom.\n",
    "many_pr_soup_1.find('section', id='primary').find_all('a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://news.microsoft.com/2019/09/24/microsoft-and-engie-announce-innovative-renewable-initiatives/',\n",
       " 'https://news.microsoft.com/2019/09/23/ste-michelle-wine-estates-leverages-microsoft-dynamics-365-to-transform-customer-experience/',\n",
       " 'https://news.microsoft.com/2019/09/19/microsoft-announces-board-changes-and-proposes-election-of-new-board-member/',\n",
       " 'https://news.microsoft.com/2019/09/18/microsoft-announces-quarterly-dividend-increase-and-new-share-repurchase-program/',\n",
       " 'https://news.microsoft.com/2019/09/18/nextlink-internet-and-microsoft-closing-broadband-gap-in-central-us/',\n",
       " 'https://news.microsoft.com/2019/09/17/schlumberger-chevron-and-microsoft-announce-collaboration-to-accelerate-digital-transformation/',\n",
       " 'https://news.microsoft.com/2019/09/13/microsoft-and-the-walt-disney-studios-to-develop-scene-to-screen-content-workflows/',\n",
       " 'https://news.microsoft.com/2019/08/12/jio-and-microsoft-announce-alliance-to-accelerate-digital-transformation-in-india/',\n",
       " 'https://news.microsoft.com/2019/07/30/microsoft-announces-iot-signals-research-report-on-state-of-iot-adoption/',\n",
       " 'https://news.microsoft.com/2019/07/22/openai-forms-exclusive-computing-partnership-with-microsoft-to-build-new-azure-ai-supercomputing-technologies/',\n",
       " 'https://news.microsoft.com/2019/07/18/microsoft-earnings-press-release-available-on-investor-relations-website-3/',\n",
       " 'https://news.microsoft.com/2019/07/18/microsoft-cloud-powers-record-fourth-quarter-results/']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Almost, but note the ones at the bottom.\n",
    "articles = many_pr_soup_1.find('section', id='primary').find_all('article')\n",
    "links = [i.find('a')['href'] for i in articles]\n",
    "links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "many_pr_links_1 = links.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automate getting links and data from each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We need to turn links into soup objects a lot, so let's make a function.\n",
    "def link_to_soup(link):\n",
    "    page = requests.get(link, headers={'User-Agent': _AGENT}).text\n",
    "    soup = BeautifulSoup(page)\n",
    "    return soup\n",
    "    \n",
    "def get_links_from_link_page(link_page):\n",
    "    soup = link_to_soup(link_page)\n",
    "    articles = soup.find('section', id='primary').find_all('article')\n",
    "    links = [i.find('a')['href'] for i in articles]\n",
    "    return links\n",
    "\n",
    "def get_data_from_links(links):\n",
    "    data_list = []\n",
    "    for link in links:\n",
    "        soup = link_to_soup(link)\n",
    "        data_list.append(get_data_from_soup(soup))\n",
    "        \n",
    "    return data_list\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article:published_time</th>\n",
       "      <th>og:title</th>\n",
       "      <th>og:description</th>\n",
       "      <th>og:url</th>\n",
       "      <th>article:section</th>\n",
       "      <th>h3</th>\n",
       "      <th>body</th>\n",
       "      <th>article:modified_time</th>\n",
       "      <th>og:updated_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2019-09-24T15:00:30+00:00</td>\n",
       "      <td>Microsoft and ENGIE announce innovative renewa...</td>\n",
       "      <td>CHICAGO – Sept. 24, 2019 – Microsoft Corp. and...</td>\n",
       "      <td>https://news.microsoft.com/2019/09/24/microsof...</td>\n",
       "      <td>Press Releases</td>\n",
       "      <td></td>\n",
       "      <td>CHICAGO – Sept. 24, 2019 – Microsoft Corp. and...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2019-09-23T12:00:05+00:00</td>\n",
       "      <td>Ste. Michelle Wine Estates leverages Microsoft...</td>\n",
       "      <td>WOODINVILLE, Wash., and REDMOND, Wash. — Sept....</td>\n",
       "      <td>https://news.microsoft.com/2019/09/23/ste-mich...</td>\n",
       "      <td>Press Releases</td>\n",
       "      <td></td>\n",
       "      <td>WOODINVILLE, Wash., and REDMOND, Wash. — Sept....</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2019-09-19T20:05:23+00:00</td>\n",
       "      <td>Microsoft announces board changes and proposes...</td>\n",
       "      <td>REDMOND, Wash. — Sept. 19, 2019 — Microsoft Co...</td>\n",
       "      <td>https://news.microsoft.com/2019/09/19/microsof...</td>\n",
       "      <td>Press Releases</td>\n",
       "      <td></td>\n",
       "      <td>REDMOND, Wash. — Sept. 19, 2019 — Microsoft Co...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2019-09-18T20:05:36+00:00</td>\n",
       "      <td>Microsoft announces quarterly dividend increas...</td>\n",
       "      <td>Annual shareholders meeting set for Dec. 4, 20...</td>\n",
       "      <td>https://news.microsoft.com/2019/09/18/microsof...</td>\n",
       "      <td>Press Releases</td>\n",
       "      <td>Annual shareholders meeting set for Dec. 4, 2019</td>\n",
       "      <td>REDMOND, Wash. — Sept. 18, 2019 — Microsoft Co...</td>\n",
       "      <td>2019-09-18T20:08:02+00:00</td>\n",
       "      <td>2019-09-18T20:08:02+00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2019-09-18T14:00:21+00:00</td>\n",
       "      <td>Nextlink Internet and Microsoft closing broadb...</td>\n",
       "      <td>The agreement could bring broadband access to ...</td>\n",
       "      <td>https://news.microsoft.com/2019/09/18/nextlink...</td>\n",
       "      <td>Press Releases</td>\n",
       "      <td>The agreement could bring broadband access to ...</td>\n",
       "      <td>REDMOND, Wash. — Sept. 18, 2019 — On Wednesday...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      article:published_time  \\\n",
       "0  2019-09-24T15:00:30+00:00   \n",
       "1  2019-09-23T12:00:05+00:00   \n",
       "2  2019-09-19T20:05:23+00:00   \n",
       "3  2019-09-18T20:05:36+00:00   \n",
       "4  2019-09-18T14:00:21+00:00   \n",
       "\n",
       "                                            og:title  \\\n",
       "0  Microsoft and ENGIE announce innovative renewa...   \n",
       "1  Ste. Michelle Wine Estates leverages Microsoft...   \n",
       "2  Microsoft announces board changes and proposes...   \n",
       "3  Microsoft announces quarterly dividend increas...   \n",
       "4  Nextlink Internet and Microsoft closing broadb...   \n",
       "\n",
       "                                      og:description  \\\n",
       "0  CHICAGO – Sept. 24, 2019 – Microsoft Corp. and...   \n",
       "1  WOODINVILLE, Wash., and REDMOND, Wash. — Sept....   \n",
       "2  REDMOND, Wash. — Sept. 19, 2019 — Microsoft Co...   \n",
       "3  Annual shareholders meeting set for Dec. 4, 20...   \n",
       "4  The agreement could bring broadband access to ...   \n",
       "\n",
       "                                              og:url article:section  \\\n",
       "0  https://news.microsoft.com/2019/09/24/microsof...  Press Releases   \n",
       "1  https://news.microsoft.com/2019/09/23/ste-mich...  Press Releases   \n",
       "2  https://news.microsoft.com/2019/09/19/microsof...  Press Releases   \n",
       "3  https://news.microsoft.com/2019/09/18/microsof...  Press Releases   \n",
       "4  https://news.microsoft.com/2019/09/18/nextlink...  Press Releases   \n",
       "\n",
       "                                                  h3  \\\n",
       "0                                                      \n",
       "1                                                      \n",
       "2                                                      \n",
       "3   Annual shareholders meeting set for Dec. 4, 2019   \n",
       "4  The agreement could bring broadband access to ...   \n",
       "\n",
       "                                                body  \\\n",
       "0  CHICAGO – Sept. 24, 2019 – Microsoft Corp. and...   \n",
       "1  WOODINVILLE, Wash., and REDMOND, Wash. — Sept....   \n",
       "2  REDMOND, Wash. — Sept. 19, 2019 — Microsoft Co...   \n",
       "3  REDMOND, Wash. — Sept. 18, 2019 — Microsoft Co...   \n",
       "4  REDMOND, Wash. — Sept. 18, 2019 — On Wednesday...   \n",
       "\n",
       "       article:modified_time            og:updated_time  \n",
       "0                        NaN                        NaN  \n",
       "1                        NaN                        NaN  \n",
       "2                        NaN                        NaN  \n",
       "3  2019-09-18T20:08:02+00:00  2019-09-18T20:08:02+00:00  \n",
       "4                        NaN                        NaN  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "msft_prs = pd.DataFrame(get_data_from_links(many_pr_links_1))\n",
    "msft_prs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Further automation\n",
    "\n",
    "**Note**: for running time reasons, we're not going to make a multi-links-page version, but note that there's a next page link at the bottom of those pages that can be extracted to build that:\n",
    "\n",
    "```html\n",
    "<a href=\"/category/press-releases/page/2/?paged=3\" \n",
    "   class=\"c-glyph x-hidden-focus\" \n",
    "   aria-label=\"Go to next page\" ms.title=\"Next Page\">\n",
    "```\n",
    "\n",
    "However, we could also notice that the link pages have a number in the URL that is incremented by one for each page.\n",
    "We would have to look at a page to get the end number, but we could also simply use a loop to construct a URL for each of those numbers.\n",
    "\n",
    "`https://news.microsoft.com/category/press-releases/page/2/`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}