Social Web 2016-2017 - Hands-on Session 5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*****************************************************\n",
    "# The Social Web\n",
    "\n",
    "## 2016-2017 Master Information Sciences\n",
    "\n",
    "* Instructors: Davide Ceolin, Cristina Bucur\n",
    "* Exercises for Hands-on session 5\n",
    "* 16 March 2017 11:00 - 12:45\n",
    "* WN-S201, WN-S205, WN-S209 \n",
    "*****************************************************"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Required Software: \n",
    "* Python 2.7 \n",
    "* Python packages: twitter\n",
    "\n",
    "In this session you are going to learn how to browse user profiles information. As in\n",
    "previous exercises, we are going to read the scripts from plain text files. To load\n",
    "scripts you type <code>python < nameofscript.py\\></code> on your command line.\n",
    "\n",
    "NOTE 1: you need to place all the provided scripts in your working folder.\n",
    "NOTE 2: All provided links end with dl=1, which instructs dropbox to offer a direct dowbload of the file. If you rather have the text displayed in the browser, change this to dl=0 before hitting return. The file will then be shown in a lightbox on the dropbox website, allowing you to copy the text.\n",
    "\n",
    "First, download the script for setting up the Twitter API permissions. You can download it here:\n",
    "https://www.dropbox.com/s/t4bgfbsc3cmcvqh/twitter_login.py?dl=1\n",
    "You need to fill in the empty strings with your credentials. You will need it for all the exercises.\n",
    "\n",
    "Second, download the script for making robust Twitter requests. You can download it here:\n",
    "https://www.dropbox.com/s/ayhjcqwsrdytck6/make_twitter_request.py?dl=1\n",
    "\n",
    "**Exercise 1:** Resolving user profile information (from example 9-17 in Mining the Social\n",
    "Web).\n",
    "\n",
    "Many APIs, such as GET friends/ids and GET followers/ids, return opaque ID values that\n",
    "need to be resolved to usernames or other profile information for meaningful analysis.\n",
    "Twitter provides a GET users/lookup API that can be used to resolve as many as 100 IDs or\n",
    "usernames at a time, and a simple pattern can be employed to iterate over larger batches.\n",
    "\n",
    "Download the script:\n",
    "https://www.dropbox.com/s/0tr2j79uouehjwc/get_user_profile.py?dl=1\n",
    "You need to run it like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from get_user_profile import get_user_profile\n",
    "\n",
    "# you can substitute the strings with others you like more\n",
    "get_user_profile(screen_names=[\"SocialWebMining\", \"ptwobrussell\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 2:** Getting all friends or followers for a user (from example 9-19 in Mining the\n",
    "Social Web).\n",
    "\n",
    "Download the script from\n",
    "https://www.dropbox.com/s/qdw41d7b83gzsbg/get_friends_followers_ids.py?dl=1\n",
    "You need to run it like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from get_friends_followers_ids import get_friends_followers_ids\n",
    "\n",
    "# you can substitute the strings with others you are more interested in\n",
    "screen_name=\"SocialWebMining\"\n",
    "friends_ids, followers_ids = get_friends_followers_ids(screen_name=screen_name,friends_limit=10,followers_limit=10)\n",
    "\n",
    "print friends_ids\n",
    "print followers_ids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3:** Analyzing a user’s friends and followers (from example 9-20 in Mining the\n",
    "Social Web).\n",
    "\n",
    "After harvesting all of a user’s friends and followers, you can conduct some primitive\n",
    "analyses using only the ID values themselves with the help of setwise operations such as\n",
    "intersection and difference, as shown in the following exercise. Given two sets, the\n",
    "intersection of the sets returns the items that they have in common, whereas the\n",
    "difference between the sets “subtracts” the items in one set from the other, leaving\n",
    "behind the difference. Recall that intersection is a commutative operation, while\n",
    "difference is not commutative. \n",
    "\n",
    "In the context of analyzing friends and followers, the\n",
    "intersection of two sets can be interpreted as “mutual friends” or people you are\n",
    "following who are also following you back, while the difference of two sets can be\n",
    "interpreted as followers who you aren’t following back or people you are following who\n",
    "aren’t following you back, depending on the order of the operands.\n",
    "\n",
    "Download the script from\n",
    "https://www.dropbox.com/s/so3m941u3ores1j/setwise_friends_followers_analysis.py?dl=1\n",
    "You need to run it like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from get_friends_followers_ids import get_friends_followers_ids \n",
    "from setwise_friends_followers_analysis import setwise_friends_followers_analysis\n",
    "\n",
    "screen_name = \"ptwobrussell\"\n",
    "friends_ids, followers_ids = get_friends_followers_ids(screen_name=screen_name)\n",
    "setwise_friends_followers_analysis(screen_name, friends_ids, followers_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 4:** Harvesting a user’s tweets (from example 9-21 in Mining the Social Web).\n",
    "Timelines are a fundamental concept in the Twitter developer ecosystem, and Twitter\n",
    "provides a convenient API endpoint for the purpose of harvesting tweets by user through\n",
    "the concept of a “user timeline.”\n",
    "\n",
    "Download the script from\n",
    "https://www.dropbox.com/s/vcndxk4rijk9usd/harvest_user_tweets.py?dl=1\n",
    "You need to run it like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from harvest_user_tweets import harvest_user_timeline\n",
    "\n",
    "tweets = harvest_user_timeline(screen_name=\"SocialWebMining\", max_results=200)\n",
    "print tweets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 5:** Analyzing tweet content (from example 9-23 in Mining the Social Web). \n",
    "You will be using simple statistics, such as lexical diversity and average number of words per\n",
    "tweet, to gain elementary insight into what is being talked about as a first step in\n",
    "sizing up the nature of the language being used. In order to collect tweets, you need to use two additional scripts.\n",
    "Download them from:\n",
    "https://www.dropbox.com/s/ci1pxy6u0p1yf2y/twitter_search.py?dl=1\n",
    "and:\n",
    "https://www.dropbox.com/s/b2j9bdp4e3rh50m/extract_tweet_entities.py?dl=1\n",
    "and put them in your working folder.\n",
    "\n",
    "Download the main script from \n",
    "https://www.dropbox.com/s/1qiw2i3uqkuqaee/analyze_tweet.py?dl=1\n",
    "You need to run it like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from analyze_tweet import analyze_tweet_content \n",
    "from twitter_search import twitter_search \n",
    "\n",
    "q = 'CrossFit' \n",
    "search_results = twitter_search(q) \n",
    "analyze_tweet_content(search_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*****************************************************\n",
    "###TASK 1: Analyze all of the tweets that you (or another user you are interested in) have ever retweeted. Are you at all surprised about what you have retweeted or how your (or his/her) interests have evolved over time?\n",
    "*****************************************************\n",
    "\n",
    "*****************************************************\n",
    "###TASK 2: Write a recipe to identify followers that you are not following back but perhaps should follow back based upon the content of their tweets. A few similarity measurements that may make suitable starting points were introduced in Section 3.3.3 on page 112 in the Mining the Social Web.\n",
    "*****************************************************"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}