Skip to content

Latest commit

 

History

History
47 lines (37 loc) · 2.69 KB

README.md

File metadata and controls

47 lines (37 loc) · 2.69 KB
Raw

Plan

  • First extractor extracts tweets as usual.
  • Tweets are cleaned and dumped into MongoDB.
  • Aggregation is done for the whole day.
  • Based on the aggregation, top 100 entities are found and the respective tweets are clubbed into one collection.
  • Before it is dumped into the collection, sentiment analysis is done on them.
  • Using each of the 100 collections as a separate document, LDA is performed. If 100 documents is too low, we can split the big documents into smaller ones.
  • The tweets are iterated individually to find the topic to which it belongs.
  • URLs are extracted for each topic which seem to be most relevant.
  • Webpages corresponding to the URLs are downloaded and parsed.
  • A portion of the main content can be displayed after extraction.
  • The graph is approximated as usual but the time span has to be discussed upon.
  • The graph, related tweets and summarizations of the URLs along with the hyperlinks is displayed for each topic on the portal.

Workflow

  • Control of engine starts with manager.py
  • manager.py makes us of multiprocess and subprocess to spawn extractor, preprocessor and postprocessor as separate processes
  • config.py in the utilities package stores tuning parameters such as 'alarm' times, file limit etc.
  • Refer to this .ppt for further information.

Dataset

  • Download dataset(s) from the Drive folder
    • The full_dataset.rar contains all 2 Million tweets
    • Optionally, you can download parts of this dataset from the Parts folder, each (dataset*.rar) containing 200,000 tweets
    • Each .json file contains 10,000 tweets

init

  • Clone the git repository
  • Run python_path.bat to add PYTHONPATH env variable. This needs to be done only once
  • Make necessary changes in the config.py file in *engine\utilities*
  • Run python init.py in Command Prompt to start engine
  • To stop, close all Command Prompt and Python windows

Portal

  • The portal folder is the django project for the web portal
  • Create a database called 'trends'
  • In the settings file, change password for mysql root, in case it is different
  • Run createsuperuser to create an admin
  • Create some top trends using the admin site. I have included a screenshot for UI after creating some sample topics(with ranks). It will redirect to the details page after clicking(see screenshots).
  • Homepage can be opened using the url: 127.0.0.1:8000 or localhost:8000
  • TopTrends model has a topic object and a rank object. Will be modified to include graphs n all when implementation is done.