CBF_Foursquare_NY_2017.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a href=\"http://www.galvanize.com/event/cognitive-builder-faire-san-francisco-4-21-4-23/\"><img src=\"https://ibm.box.com/shared/static/yx6xj0mobw6lagt7g60wgplo7mae8ufk.png\" width=1000></a>\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center> Learning FourSquare API with Python\n",
    "## <center> Room: Lab Seating / Studio 54\n",
    "## <center><a href=\"https://www.linkedin.com/in/aklson/\">Alex Aklson</a>, Ph.D. (Data Scientist, IBM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <center> How to use notebooks:\n",
    "<img src=https://ibm.box.com/shared/static/4g2hs88inr5mwx2midhg9j66emuwit9y.gif width=800 align=center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now try converting the following _code_ cell into a _markdown_ cell and _run_ it "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "<a id=\"Change this into a code cell\"></a>\n",
    "# Great\n",
    "## Wow\n",
    "### Amazing\n",
    "#### Super\n",
    "\n",
    "1. Eat\n",
    "2. _Sleep_\n",
    "3. **Code**\n",
    "> Build machine learning models  \n",
    "\n",
    "`Write more code`\n",
    "\n",
    "- double click this cell to edit  \n",
    "\n",
    "Create fancy $functions$\n",
    "\n",
    "<font color=\"red\">Data Science.</font>  \n",
    "\n",
    "<img src=https://api.apsportal.ibm.com/v2/nav/graphics/dsx-logo-lg.svg width=100 align=left> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "<h2> Table of Contents</h2>  \n",
    "<font size = 3>\n",
    "1. <a href=\"#item1\">Foursquare API Search Function</a>    \n",
    "2. <a href=\"#item2\">Explore a Given Venue</a>   \n",
    "3. <a href=\"#item3\">Explore a User</a>  \n",
    "4. <a href=\"#item4\">Foursquare API Explore Function</a>  \n",
    "5. <a href=\"#item5\">Get Trending Venues</a>    \n",
    "6. <a href=\"#item6\">Explore real world data - San Francisco Crime Rate</a>  \n",
    "7. <a href=\"#item7\">Use Foursquare API to do cool analysis</a>  \n",
    "</font>\n",
    "<br>\n",
    "<p></p>\n",
    "\n",
    "Estimated Time Needed: <strong>60 min</strong>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Import necessary Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from geopy.geocoders import Nominatim # module to convert an address into latitude and longitude values\n",
    "import requests # library to handle requests\n",
    "import pandas as pd # library for data analsysis\n",
    "import numpy as np # library to handle data in a vectorized manner\n",
    "import random # library for random number generation\n",
    " \n",
    "# tranforming json file into a pandas dataframe library\n",
    "from pandas.io.json import json_normalize\n",
    "\n",
    "!pip install folium\n",
    "import folium # plotting library\n",
    "\n",
    "print \"Folium installed\"\n",
    "print \"Libraries imported.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Foursquare Credentials and Version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Make sure that you have created a Foursquare developer account and have your credentials handy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "CLIENT_ID = \"your-client_id-here\"\n",
    "CLIENT_SECRET = \"your-client_secret-here\"\n",
    "VERSION = \"20170511\"\n",
    "LIMIT = 30\n",
    "print \"Your credentails:\"\n",
    "print \"CLIENT_ID: \" + CLIENT_ID\n",
    "print \"CLIENT_SECRET:\" + CLIENT_SECRET"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define an address to explore and convert it to its latitude and longitude equivalence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "address = \"315 Hudson Street New York, NY\"\n",
    "\n",
    "geolocator = Nominatim()\n",
    "location = geolocator.geocode(address)\n",
    "latitude = location.latitude\n",
    "longitude = location.longitude\n",
    "print latitude, longitude"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"item1\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Search for a specific venue category\n",
    "> `https://api.foursquare.com/v2/venues/`**search**`?client_id=`**CLIENT_ID**`&client_secret=`**CLIENT_SECRET**`&ll=`**LATITUDE**`,`**LONGITUDE**`&v=`**VERSION**`&query=`**QUERY**`&radius=`**RADIUS**`&limit=`**LIMIT**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define a search query for Italian restaurants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "search_query = \"Italian Restaurant\"\n",
    "print search_query + \" ¯\\_(ツ)_/¯ .... OK!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define the corresponding URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "radius = 1000\n",
    "url=\"https://api.foursquare.com/v2/venues/search?client_id={}&client_secret={}&ll={},{}&v={}&query={}&radius={}&limit={}\".format(CLIENT_ID, CLIENT_SECRET, latitude, longitude, VERSION, search_query, radius, LIMIT)\n",
    "url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Send the GET Request and examine the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "results = requests.get(url).json()\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get relevant part of JSON and transform it into a *pandas* dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# assign relevant part of JSON to venues\n",
    "venues = results[\"response\"][\"venues\"]\n",
    "\n",
    "# tranform venues into a dataframe\n",
    "dataframe = json_normalize(venues)\n",
    "dataframe.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define information of interest and filter dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# keep only columns that include venue name, url, and anything that is associated with location\n",
    "filtered_columns = ['name', 'url', 'categories', 'verified'] + [col for col in dataframe.columns if col.startswith('location.')] + ['id']\n",
    "dataframe_filtered = dataframe.ix[:, filtered_columns]\n",
    "\n",
    "# function that extracts the category of the venue\n",
    "def get_category_type(row):\n",
    "    try:\n",
    "        categories_list = row[\"categories\"]\n",
    "    except:\n",
    "        categories_list = row[\"venue.categories\"]\n",
    "        \n",
    "    if len(categories_list) == 0:\n",
    "        return None\n",
    "    else:\n",
    "        return categories_list[0][\"name\"].encode('ascii',errors='ignore')\n",
    "\n",
    "# filter the category for each row\n",
    "dataframe_filtered['categories'] = dataframe_filtered.apply(get_category_type, axis=1)\n",
    "\n",
    "# clean column names by keeping only last term\n",
    "dataframe_filtered.columns = [column.split(\".\")[-1] for column in dataframe_filtered.columns]\n",
    "dataframe_filtered.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's retrieve Foursquare's categories and corresponding IDs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# define URL for categories\n",
    "url = \"https://api.foursquare.com/v2/venues/categories?client_id={}&client_secret={}&v={}\".format(CLIENT_ID, CLIENT_SECRET, VERSION)\n",
    "\n",
    "# send call request and get categories\n",
    "results = requests.get(url).json()\n",
    "categories = results[\"response\"][\"categories\"]\n",
    "\n",
    "# loop through categories and print them\n",
    "total = 0\n",
    "for i in range(len(categories)):\n",
    "    main_category = categories[i][\"name\"]\n",
    "    print \"Main Category: \" + main_category, categories[i][\"id\"]\n",
    "    count = 0\n",
    "    for category in categories[i][\"categories\"]:\n",
    "        print category[\"name\"], category[\"id\"]\n",
    "        count += 1\n",
    "        total += 1\n",
    "    print count\n",
    "print total"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Redefine URL to include category ID for better matching"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "category_id = \"4bf58dd8d48988d110941735\" # ID for Italian restaurants \n",
    "url=\"https://api.foursquare.com/v2/venues/search?client_id={}&client_secret={}&ll={},{}&v={}&radius={}&query={}&categoryId={}&limit={}\".format(CLIENT_ID, CLIENT_SECRET, latitude, longitude, VERSION, radius, search_query, category_id, LIMIT)\n",
    "url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# send GET request\n",
    "results = requests.get(url).json()\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Send GET request and place results in a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# send GET request\n",
    "results = requests.get(url).json()\n",
    "\n",
    "# assign relevant part of JSON to venues\n",
    "venues = results[\"response\"][\"venues\"]\n",
    "\n",
    "# tranform venues into a dataframe\n",
    "dataframe = json_normalize(venues)\n",
    "\n",
    "# filter columns\n",
    "dataframe_filtered = dataframe.ix[:, filtered_columns]\n",
    "\n",
    "# filter the category for each row\n",
    "dataframe_filtered['categories'] = dataframe_filtered.apply(get_category_type, axis=1)\n",
    "\n",
    "# clean column names by keeping only last term\n",
    "dataframe_filtered.columns = [column.split(\".\")[-1] for column in dataframe_filtered.columns]\n",
    "dataframe_filtered.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's visualize the Italian restaurants that are nearby"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "venues_map = folium.Map(location=[latitude, longitude], zoom_start=15) # generate map centred around Galvanize\n",
    "\n",
    "\n",
    "# create a feature group for Galvanize and add it to the map\n",
    "galvanize = folium.map.FeatureGroup()\n",
    "galvanize.add_child(\n",
    "    folium.features.CircleMarker(\n",
    "        [latitude, longitude],\n",
    "        radius = 10,\n",
    "        color = \"red\",\n",
    "        fill_color = \"red\",\n",
    "        fill_opacity = 0.6\n",
    "    )\n",
    ")\n",
    "venues_map.add_child(galvanize)\n",
    "\n",
    "\n",
    "# create a feature group for the Italian restaurants around Galvanize and add it to the map\n",
    "restaurants = folium.map.FeatureGroup()\n",
    "for lat, lng, in zip(dataframe_filtered.lat, dataframe_filtered.lng):\n",
    "    restaurants.add_child(\n",
    "        folium.features.CircleMarker(\n",
    "            [lat, lng],\n",
    "            radius=5,\n",
    "            color=\"blue\",\n",
    "            fill_color=\"blue\",\n",
    "            fill_opacity=0.6)\n",
    "        )\n",
    "venues_map.add_child(restaurants)\n",
    "\n",
    "\n",
    "# add labels to each point on the map\n",
    "latitudes_list = list(dataframe_filtered.lat) + [latitude]\n",
    "longitudes_list = list(dataframe_filtered.lng) + [longitude]\n",
    "labels = list(dataframe_filtered.name) + [\"Galvanize\"]\n",
    "\n",
    "for lat, lng, label in zip(latitudes_list, longitudes_list, labels):\n",
    "    folium.Marker([lat, lng], popup=label).add_to(venues_map)\n",
    "\n",
    "# display map\n",
    "venues_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"item2\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Explore a Given Venue\n",
    "> `https://api.foursquare.com/v2/venues/`**VENUE_ID**`?client_id=`**CLIENT_ID**`&client_secret=`**CLIENT_SECRET**`&v=`**VERSION**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A. Let's explore the first verified restaurant -- _Galli Restaurant_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "venue_id = \"5018507fe4b03a729d0b40f9\" # ID of Galli Restaurant\n",
    "url=\"https://api.foursquare.com/v2/venues/{}?client_id={}&client_secret={}&v={}\".format(venue_id, CLIENT_ID, CLIENT_SECRET, VERSION)\n",
    "url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Send GET request for result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "result = requests.get(url).json()\n",
    "result[\"response\"][\"venue\"].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### B. Get the venue's overall rating"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "result[\"response\"][\"venue\"][\"rating\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### C. Get the number of tips"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "result[\"response\"][\"venue\"][\"tips\"][\"count\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### D. Get the venue's tips\n",
    "> `https://api.foursquare.com/v2/venues/`**VENUE_ID**`/tips?client_id=`**CLIENT_ID**`&client_secret=`**CLIENT_SECRET**`&v=`**VERSION**`&limit=`**LIMIT**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create URL and send GET request. Make sure to set limit to get all tips"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "## Galli Restaurant tips\n",
    "limit = 150 # set limit to be greater than the total number of tips\n",
    "url=\"https://api.foursquare.com/v2/venues/{}/tips?client_id={}&client_secret={}&v={}&limit={}\".format(venue_id, CLIENT_ID, CLIENT_SECRET, VERSION, limit)\n",
    "\n",
    "results = requests.get(url).json()\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get tips and list of associated features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tips = results[\"response\"][\"tips\"][\"items\"]\n",
    "\n",
    "tip = results[\"response\"][\"tips\"][\"items\"][0]\n",
    "tip.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Format column width and display all tips"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "pd.set_option('display.max_colwidth', -1)\n",
    "\n",
    "tips_df = json_normalize(tips) # json normalize tips\n",
    "\n",
    "# columns to keep\n",
    "filtered_columns = [\"text\", \"agreeCount\", \"disagreeCount\", \"id\", \"user.firstName\", \"user.lastName\", \"user.gender\", \"user.id\"]\n",
    "tips_filtered = tips_df.ix[:, filtered_columns]\n",
    "\n",
    "# display tips\n",
    "tips_filtered"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"item3\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 3. Search a Foursquare User\n",
    "> `https://api.foursquare.com/v2/users/`**USER_ID**`?client_id=`**CLIENT_ID**`&client_secret=`**CLIENT_SECRET**`&v=`**VERSION**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define URL, send GET request and display features associated with user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user_id=\"15149512\" # user ID with most agree counts and complete profile\n",
    "\n",
    "url=\"https://api.foursquare.com/v2/users/{}?client_id={}&client_secret={}&v={}\".format(user_id, CLIENT_ID, CLIENT_SECRET, VERSION) # define URL\n",
    "\n",
    "# send GET request\n",
    "results = requests.get(url).json()\n",
    "user_data = results[\"response\"][\"user\"]\n",
    "\n",
    "# display features associated with user\n",
    "user_data.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print \"First Name: \" + user_data[\"firstName\"]\n",
    "print \"Last Name: \" + user_data[\"lastName\"]\n",
    "print \"Home City: \" + user_data[\"homeCity\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### How many tips has this user submitted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "user_data[\"tips\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get User's tips"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# define tips URL\n",
    "url=\"https://api.foursquare.com/v2/users/{}/tips?client_id={}&client_secret={}&v={}&limit={}\".format(user_id, CLIENT_ID, CLIENT_SECRET, VERSION, limit)\n",
    "\n",
    "# send GET request and get user's tips\n",
    "results = requests.get(url).json()\n",
    "tips = results[\"response\"][\"tips\"][\"items\"]\n",
    "\n",
    "# format column width\n",
    "pd.set_option('display.max_colwidth', -1)\n",
    "\n",
    "tips_df = json_normalize(tips)\n",
    "\n",
    "# filter columns\n",
    "filtered_columns = [\"text\", \"agreeCount\", \"disagreeCount\", \"id\"]\n",
    "tips_filtered = tips_df.ix[:, filtered_columns]\n",
    "\n",
    "# display user's tips\n",
    "tips_filtered"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's get the venue for the tip with a disagree count of 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tip_id = \"56fc15bb498e44ba0b86da1a\" # tip id\n",
    "\n",
    "# define URL\n",
    "url = \"http://api.foursquare.com/v2/tips/{}?client_id={}&client_secret={}&v={}\".format(tip_id, CLIENT_ID, CLIENT_SECRET, VERSION)\n",
    "\n",
    "# send GET Request and examine results\n",
    "result = requests.get(url).json()\n",
    "print result[\"response\"][\"tip\"][\"venue\"][\"name\"]\n",
    "print result[\"response\"][\"tip\"][\"venue\"][\"location\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get User's friends"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "user_friends = json_normalize(user_data[\"friends\"][\"groups\"][0][\"items\"])\n",
    "user_friends"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"item4\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Explore a location\n",
    "> `https://api.foursquare.com/v2/venues/`**explore**`?client_id=`**CLIENT_ID**`&client_secret=`**CLIENT_SECRET**`&ll=`**LATITUDE**`,`**LONGITUDE**`&v=`**VERSION**`&limit=`**LIMIT**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get latitude and longitude values of Galli Restaurant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "latitude = 40.721628\n",
    "longitude = -74.001227"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "url=\"https://api.foursquare.com/v2/venues/explore?client_id={}&client_secret={}&ll={},{}&v={}&radius={}&limit={}\".format(CLIENT_ID, CLIENT_SECRET, latitude, longitude, VERSION, radius, LIMIT)\n",
    "url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Send GET request and examine results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "results = requests.get(url).json()\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get relevant part of JSON"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "items = results[\"response\"][\"groups\"][0][\"items\"]\n",
    "items[0][\"venue\"].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Process JSON and convert it to a clean dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "dataframe = json_normalize(items) # flatten JSON\n",
    "\n",
    "# filter columns\n",
    "filtered_columns = ['venue.name', 'venue.url', 'venue.categories'] + [\"venue.rating\"] + [col for col in dataframe.columns if col.startswith('venue.location.')] + [\"venue.id\"]\n",
    "dataframe_filtered = dataframe.ix[:, filtered_columns]\n",
    "\n",
    "# filter the category for each row\n",
    "dataframe_filtered['venue.categories'] = dataframe_filtered.apply(get_category_type, axis=1)\n",
    "\n",
    "# clean columns\n",
    "dataframe_filtered.columns = [col.split(\".\")[-1] for col in dataframe_filtered.columns]\n",
    "\n",
    "dataframe_filtered.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### Let's visualize these items on the map around our location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "venues_map = folium.Map(location=[latitude, longitude], zoom_start=15) # generate map centred around Galli Restaurant\n",
    "\n",
    "\n",
    "# create a feature group for Galli Restaurant and add it to the map\n",
    "galli = folium.map.FeatureGroup()\n",
    "galli.add_child(\n",
    "    folium.features.CircleMarker(\n",
    "        [latitude, longitude],\n",
    "        radius = 10,\n",
    "        color = \"red\",\n",
    "        fill_color = \"red\",\n",
    "        fill_opacity = 0.6\n",
    "    )\n",
    ")\n",
    "venues_map.add_child(galli)\n",
    "\n",
    "\n",
    "# create a feature group for popular venues around Galli Restaurant and add it to the map\n",
    "popular_venues = folium.map.FeatureGroup()\n",
    "for lat, lng, in zip(dataframe_filtered.lat, dataframe_filtered.lng):\n",
    "    popular_venues.add_child(\n",
    "        folium.features.CircleMarker(\n",
    "            [lat, lng],\n",
    "            radius=5,\n",
    "            color=\"blue\",\n",
    "            fill_color=\"blue\",\n",
    "            fill_opacity=0.6)\n",
    "        )\n",
    "venues_map.add_child(popular_venues)\n",
    "\n",
    "\n",
    "# add labels to each point on the map\n",
    "latitudes_list = list(dataframe_filtered.lat) + [latitude]\n",
    "longitudes_list = list(dataframe_filtered.lng) + [longitude]\n",
    "labels = list(dataframe_filtered.name) + [\"Galli Restaurant\"]\n",
    "venue_categories = list(dataframe_filtered.categories) + [\"\"]\n",
    "\n",
    "for lat, lng, label, category in zip(latitudes_list, longitudes_list, labels, venue_categories):\n",
    "    popup_text = label + \", \" + category\n",
    "    folium.Marker([lat, lng], popup=popup_text).add_to(venues_map)\n",
    "\n",
    "# display map\n",
    "venues_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<a id=\"item5\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 5. Explore Trending Venues\n",
    "> `https://api.foursquare.com/v2/venues/trending?client_id=`**CLIENT_ID**`&client_secret=`**CLIENT_SECRET**`&ll=`**LATITUDE**`,`**LONGITUDE**`&v=`**VERSION**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's get trending venues around Galli Restaurant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define URL\n",
    "url=\"https://api.foursquare.com/v2/venues/trending?client_id={}&client_secret={}&ll={},{}&v={}\".format(CLIENT_ID, CLIENT_SECRET, latitude, longitude, VERSION)\n",
    "\n",
    "# send GET request and get trending venues\n",
    "results = requests.get(url).json()\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check if any venues are trending at this time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if len(results[\"response\"][\"venues\"]) == 0:\n",
    "    print \"No trending venues are available at the moment!\"\n",
    "    \n",
    "else:\n",
    "    trending_venues = results[\"response\"][\"venues\"]\n",
    "    trending_venues_df = json_normalize(trending_venues)\n",
    "\n",
    "    # filter columns\n",
    "    columns_filtered = [\"name\", \"url\", \"categories\"] + [col for col in trending_venues_df.columns if col.startswith(\"stats\")] + [\"location.distance\", \"location.address\", \"location.city\", \"location.postalCode\", \"location.state\", \"location.country\", \"location.lat\", \"location.lng\"]\n",
    "    trending_venues_df = trending_venues_df.ix[:, columns_filtered]\n",
    "\n",
    "    # filter the category for each row\n",
    "    trending_venues_df['categories'] = trending_venues_df.apply(get_category_type, axis=1)\n",
    "\n",
    "    # display trending venues\n",
    "    trending_venues_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize trending venues"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "if len(results[\"response\"][\"venues\"]) == 0:\n",
    "    print \"Can't generate visual as no trending venues are available at the moment!\"\n",
    "\n",
    "else:\n",
    "    venues_map = folium.Map(location=[latitude, longitude], zoom_start=15) # generate map centred around Galli Restaurant\n",
    "\n",
    "\n",
    "    # create a feature group for Galli Restaurant and add it to the map\n",
    "    galli = folium.map.FeatureGroup()\n",
    "    galli.add_child(\n",
    "        folium.features.CircleMarker(\n",
    "            [latitude, longitude],\n",
    "            radius = 10,\n",
    "            color = \"red\",\n",
    "            fill_color = \"red\",\n",
    "            fill_opacity = 0.6\n",
    "        )\n",
    "    )\n",
    "    venues_map.add_child(galli)\n",
    "\n",
    "\n",
    "    # create a feature group for popular venues around Galli Restaurant and add it to the map\n",
    "    trending_venues = folium.map.FeatureGroup()\n",
    "    for lat, lng, in zip(trending_venues_df[\"location.lat\"], trending_venues_df[\"location.lng\"]):\n",
    "        popular_venues.add_child(\n",
    "            folium.features.CircleMarker(\n",
    "                [lat, lng],\n",
    "                radius=5,\n",
    "                color=\"blue\",\n",
    "                fill_color=\"blue\",\n",
    "                fill_opacity=0.6\n",
    "            )\n",
    "        )\n",
    "    venues_map.add_child(trending_venues)\n",
    "\n",
    "\n",
    "    # add labels to each point on the map\n",
    "    latitudes_list = list(trending_venues_df[\"location.lat\"]) + [latitude]\n",
    "    longitudes_list = list(trending_venues_df[\"location.lng\"]) + [longitude]\n",
    "    labels = list(trending_venues_df.name) + [\"Galli Restaurant\"]\n",
    "    venue_categories = list(trending_venues_df.categories) + [\"\"]\n",
    "\n",
    "    for lat, lng, label, category in zip(latitudes_list, longitudes_list, labels, venue_categories):\n",
    "        popup_text = label + \", \" + category\n",
    "        folium.Marker([lat, lng], popup=popup_text).add_to(venues_map)\n",
    "\n",
    "    # display map\n",
    "    venues_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<a id=\"item6\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Crime in San Francisco Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://dev.socrata.com/foundry/data.sfgov.org/cuks-n6tp\">Crime incidents in San Francisco dataset compiled by police department</a>. The dataset contains data about incidents that occured from **01/01/2003** - **12/31/2016**. You can learn more about the dataset by clicking the link above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download dataset from IBM server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!wget -O crime_data_SF.csv https://ibm.box.com/shared/static/z2rjkrelj9fc87d1lrw6gkm9sb91zu5h.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read the data into a *pandas* dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "crime_data = pd.read_csv('crime_data_SF.csv')\n",
    "print \"CSV file read into a pandas dataframe\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quick Exploration of the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. First five crimes that occured in 2016"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "crime_data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Number of crimes and corresponding attributes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "crime_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Let's zoom into the month of June 2016"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "crime_data_jun = crime_data[(crime_data[\"Date\"] >= '2016-06-01') & (crime_data[\"Date\"] <= '2016-06-30')]\n",
    "crime_data_jun.reset_index(inplace=True, drop=True)\n",
    "crime_data_jun.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. List of attributes -- *any with missing values*?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "crime_data_jun.notnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. How many types of crimes-- *and* what are the top 10?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "jun_categories_count = crime_data_jun.Category.value_counts()\n",
    "print \"There at {} types of crimes. And the top 10 are:\".format(len(jun_categories_count))\n",
    "print jun_categories_count.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"item7\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Using Foursquare to further analyze Crime in San Francisco"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write a function that returns the venues categories around a given location"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# a function that returns the categories and the distances of the 10 closest venues to a crime scene\n",
    "def get_nearby_venues(row):\n",
    "    \n",
    "    latitude = row[\"Y\"]\n",
    "    longitude = row[\"X\"]\n",
    "    \n",
    "    # create the URL using the latitude and longitude of the crime scene\n",
    "    url = \"https://api.foursquare.com/v2/venues/explore?client_id={}&client_secret={}&ll={},{}&v={}\".format(CLIENT_ID, CLIENT_SECRET, latitude, longitude, VERSION)\n",
    "    \n",
    "    # send the GET request and get the venues\n",
    "    results = requests.get(url).json()\n",
    "    venues = results[\"response\"][\"groups\"][0][\"items\"]\n",
    "    \n",
    "    # convert the venues JSON into a dataframe for efficient processing\n",
    "    dataframe = json_normalize(venues)\n",
    "    \n",
    "    # extract the categories and sort the venues by ascending order of distance and reset indices\n",
    "    dataframe['venue.categories'] = dataframe.apply(get_category_type, axis=1)\n",
    "    dataframe.sort_values(['venue.location.distance'], ascending=[True], inplace=True)\n",
    "    dataframe.reset_index(inplace=True, drop=True)\n",
    "    \n",
    "    # return categories of the 10 closest venues\n",
    "    num_venues = 10\n",
    "    categories_nearby_venues = dataframe.loc[0:num_venues, \"venue.categories\"].str.cat(sep=', ')\n",
    "    distances_nearby_venues = (dataframe.loc[0:num_venues, \"venue.location.distance\"].apply(str)).str.cat(sep=', ')\n",
    "    return categories_nearby_venues, distances_nearby_venues\n",
    "\n",
    "def get_nearby_venues_categories(row):\n",
    "    return get_nearby_venues(row)[0]\n",
    "\n",
    "def get_nearby_venues_distances(row):\n",
    "    return get_nearby_venues(row)[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write another function that prints statistics for a given crime category"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def print_most_common_categories(crime):\n",
    "    # filter data for the given crime\n",
    "    crime_category_data = crime_data[crime_data[\"Category\"] == crime]\n",
    "    crime_category_data.reset_index(inplace=True, drop=True)\n",
    "\n",
    "    # randomly select 10 crimes of this category\n",
    "    random.seed(1234)\n",
    "    num_crimes = 10\n",
    "    sample_rows = np.arange(len(crime_category_data))\n",
    "    np.random.shuffle(sample_rows)\n",
    "    crime_category_data = crime_category_data.ix[sample_rows[0:num_crimes], :]\n",
    "    crime_category_data.reset_index(inplace=True, drop=True)\n",
    "    crime_category_data.head()\n",
    "    \n",
    "    # create columns of nearby venues categories and distances\n",
    "    crime_category_data[\"nearby_categories\"] = crime_category_data.apply(get_nearby_venues_categories, axis=1)\n",
    "    crime_category_data[\"nearby_distances\"] = crime_category_data.apply(get_nearby_venues_distances, axis=1)\n",
    "\n",
    "    # process columns to compute mean distance\n",
    "    most_common_categories = crime_category_data[\"nearby_categories\"].str.cat(sep=', ')\n",
    "    most_common_categories = most_common_categories.split(\", \")\n",
    "    most_common_categories_distances = crime_category_data[\"nearby_distances\"].str.cat(sep=', ')\n",
    "    most_common_categories_distances = map(int, most_common_categories_distances.split(\", \"))\n",
    "\n",
    "    # print average distance for each category\n",
    "    summary_dataframe = pd.DataFrame(columns = ['Category', 'Distance'])\n",
    "    summary_dataframe[\"Category\"] = pd.Series(most_common_categories)\n",
    "    summary_dataframe[\"Distance\"] = pd.Series(most_common_categories_distances)\n",
    "    print summary_dataframe.groupby([\"Category\"])[\"Distance\"].mean().sort_values()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Let's investigate LARCENY/THEFT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "crime = \"VEHICLE THEFT\"\n",
    "print_most_common_categories(crime)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### How close to the nearest bar are people who drive under the influence when they get pulled over by cops?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "DUI = crime_data_jun.loc[crime_data_jun[\"Category\"] == \"DRIVING UNDER THE INFLUENCE\"] # subset dataframe for driving under the influence crimes\n",
    "DUI.head() # display the first five instances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# get_closest_distance is a function that returns the distance between the crime scene and the closest bar\n",
    "def get_closest_distance(row):\n",
    "    \n",
    "    latitude = row[\"Y\"] # get latitude value of crime scene\n",
    "    longitude = row[\"X\"] # get longitude value of crime scence\n",
    "    \n",
    "    search_query = \"bar\" # define search query\n",
    "\n",
    "    # define URL and send GET request\n",
    "    url=\"https://api.foursquare.com/v2/venues/search?client_id={}&client_secret={}&ll={},{}&v={}&query={}&limit={}\".format(CLIENT_ID, CLIENT_SECRET, latitude, longitude, VERSION, search_query, LIMIT)\n",
    "    results = requests.get(url).json()\n",
    "\n",
    "    # Get venues from returned results\n",
    "    venues = results[\"response\"][\"venues\"]\n",
    "    bars_df = json_normalize(venues)\n",
    "\n",
    "    # filter columns\n",
    "    filtered_columns = ['name', 'url'] + [col for col in bars_df.columns if col.startswith('location')]\n",
    "    bars_df = bars_df.ix[:, filtered_columns]\n",
    "\n",
    "    # clean column names by keeping only last term\n",
    "    bars_df.columns = [column.split(\".\")[-1] for column in bars_df.columns]\n",
    "\n",
    "    # return distance\n",
    "    bars_df.sort_values(['distance'], ascending=True, inplace=True)\n",
    "    return bars_df[\"distance\"][0] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "DUI[\"closest_distance\"] = DUI.apply(get_closest_distance, axis=1) # add column of distance to closest bar for each crime incident"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "DUI.groupby([\"Category\"])[\"closest_distance\"].mean() # compute the mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <center> To create a new notebook: \n",
    "<img src=https://ibm.box.com/shared/static/widfmgekf12oksgw37uhcsfilah00h14.gif width=800 align=center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center>Thank you for attending!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2 with Spark 1.6",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}