The Moodle Scraper is a Python script that allows you to automate the process of downloading files and posts from Moodle courses. This script is designed to work with the current Moodle verson used at the Indian Institute of Technology Bombay (IITB), and Firefox as browser.
Before using the Moodle Scraper, ensure you have the following dependencies installed:
- Python 3.x
- Required Python packages (install using
pip
):requests
BeautifulSoup
selenium
webdriver_manager
- Specific versions are listed in
requirements.txt
.
-
Clone this repository to your local machine.
-
Install the required packages by executing the following command in your terminal.
pip intsll -r requirements.txt
-
Modify the
config.json
file as described in Configuration. -
Run the script by executing the following command in your terminal:
python moodle.py
The script will log in to your Moodle account, download course materials, and scrape forum posts.
The config.json file contains configuration settings for the browser preferences. You should modify this file to match your specific setup.
Set up your browser.
-
The script uses Selenium and WebDriver to automate browser actions, such as logging in to Moodle.
-
It scrapes course information from your Moodle homepage and creates folders for each course in a specified directory (e.g., ./Sem5).
-
It downloads course materials (PDFs) that haven't been downloaded before and saves them in the corresponding course folders.
-
It scrapes forum posts and downloads any attachments associated with them.
Feel free to contribute to this project by opening issues or submitting pull requests. Your contributions are welcome