Skip to content

Initialize a new project based on Spidergram, the structural analysis tool for large web sites.

License

Notifications You must be signed in to change notification settings

autogram-is/create-spidergram

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

36 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Build a custom web analysis tool with Spidergram

Spidergram is a toolkit for crawling and analyzing complex web properties. create-spidergram is a quick and easy way to set up a new Spidergram project of your own.

Usage

  1. Ensure you're running NodeJS 18 (node -v)
  2. Install ArangoDB via direct download or homebrew. Alternately, if you've got Docker installed, you can use Spidergram's included docker-compose.yml file to spin up an Arango container for testing and development.
  3. Create a new project directory, cd into it, and run npx create-spidergram. You'll be prompted for the project's name and your choice of project template.
  4. Run npm install
  5. Kick the tires with npm run crawl <url>, or dive right in to customizing the project.

The Templates

  • Boilerplate is an NPM project that fires up a Spidergram crawler, grabs the contents of one or more sites, and prints out a summary report of their URL structures.
  • Boilerplate (Typescript) is a Typescript version of Boilerplate, with no other functional differences.
  • Crawl with Report (Typescript) demonstrates basic data extraction and custom report generation in plaintext and Excel formats.
  • JSON config uses a static config file to control most Spidergram settings in conjunction with the globally-installed CLI. If you're interested in kicking the tires, just install this one, then npm install -g spidergram, brew install docker-compose, and docker-compose up. You're ready to Spidergram.
  • YAML config What if Spidergram, but YAML?

About

Initialize a new project based on Spidergram, the structural analysis tool for large web sites.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published