The web application can be accessed at http://webdependency.andrew.cmu.edu:8080/
The repo consists of the following components
The Python scripts can process measurements for target services (DNS, CDN, and CA).
The general measurement workflow is the following:
- Fetch a list of popular website from Google BigQuery
- Please follow https://cloud.google.com/bigquery/docs/reference/libraries to setup Google BigQuery client authentication
- Once credentials retrieved and stored locally, save the path to the credential file in the variable
export GOOGLE_APPLICATION_CREDENTIALS="<path_to_credentials_json>"
- Measurement: retrieve service dependencies for all websites in the list returned from step 1
- Classify: Based on the measurement results, classify if the service dependency is Private, Third-party, or unknown
- Group: Group the dependencies together based on their attributes so they won't be repeated
- Print: Print out results, including website/client, service, service type, and group name
- Analyze: Process analyze the results to obtain provider stats and client stats
- Plot: Generate a json file for dependency graph plotting on the web application
Note: The workflow is only set up with DNS for now
Request access to the following if you don't have permissions:
- Box folder is available here: https://cmu.app.box.com/folder/182556826473
- The application to upload/download file is https://cmu.app.box.com/developers/console/app/1897359
To run the measurement:
- Go to measurements directory
cd measurements
- Install dependencies:
pipenv install
- Start Python virtual environment:
pipenv shell
- Run scripts:
python main.py <country_code>
country_code
is the two character code for the country, e.g. the country code for the United States isus
The web application is a React frontend application. The web application may load files in this repo or call the Box File Server to fetch files from CMU Box. The application uses Material UI with some Material Dashboard Components.
- The source files for the home page is in
src/layouts/home/
. - The source files for region analysis, country analysis are in
src/layouts/dashboard
. - The source files for side nav bar are in
src/examples/Sidenav
The soruce files for top nav bar are insrc/examples/Navbars/DashboardNavbar
To run the application:
- Install dependencies:
npm install
- Run application:
npm start
- The application should be available at http://localhost:8080/
The Box file server is a backend service written in Python Flask. It is used to fetch files from CMU Box. Currently the route /country/<country>/service/<service>/month/<month>
is in use. It first look for requested file from directory files
, and fetch from Box if the file doesn't exist locally. The file server is deployed at http://webdependency.andrew.cmu.edu:5000/
Request access to the following if you don't have permissions:
- Box folder is available here: https://cmu.app.box.com/folder/182556826473
- The application to upload/download file is https://cmu.app.box.com/developers/console/app/1897359
Before running the server, you will need to fetch the client secret for the application:
- Go to https://cmu.app.box.com/developers/console/app/1897359/configuration
- Click "Fetch Client Secret" in "OAuth 2.0 Credentials" section
- Copy the client secret
- Create a new file in your home directory
~/.secrets/credentials.json
- Format:
{ "box_client": <client_secret> }
- If you create the file at another location, change the path for the
credentials_file
in theinit_client
function inbox_client.py
- Format:
To run the server:
- Go to box file server directory:
cd box_file_server
- Installl dependencies in the virtual environment:
pipenv install
- Activate Python virtual environment
pipenv shell
- Run the server:
flask run
- The server should be available at http://localhost:5000