A Python web scriping program which reads from a txt file what details to scrape for and what website to visit, then returning the results in a new txt file.
Developed by C. Hans Culton
Apache-2.0
- BeautifulSoup4
- Selenium
- Firefox
Download the project. Enter the URL into the txt file, URLToScrape.txt. Add elements you would like to scrape for to the same txt file. Run the python file, and a new file, results.txt, will be created in the same directory, populated with the results of the scrape.
Initially this project visits Stocktwits
https://stocktwits.com/discover/earnings-calendar/
I chose this website as it is heavily reliant off of AJAX requests and JavaScript, making it challenging to scrape information from. Furthermore, it has unique and changing content everyday, making for an interesting website to scrape. The URL to visit, however, can be changed in the URLToScrape.txt.