last.fm-spelunker has a few requirements:
- Python. Both Python 2.X and 3.X are supported, but Python 3.X is recommended.
- Some Python packages—these may be easily installed via the
requirements.txt.
Generates an exhaustive SQL database of the user's scrobbles and any related info (e.g. data for scrobbled songs or artists) through the last.fm API. Note: the artists functionality is currently disabled but may easily be re-enabled in scraper.py.
Usage: python scraper.py <last.fm username> ... <last.fm username>
~$ python scraper.py sbansal21
Retrieved scrobble history.... 444 of 444 for sbansal21.
If a user's scrobble history has already been scraped previously, scraper.py will automatically update that history with any new scrobbles. Additionally, scraper.py supports scraping multiple users' scrobble history at once:
Usage: python scraper.py <last.fm username> ... <last.fm username>
~$ python scraper.py sbansal21 iuyag
Updated scrobble history with 745 new scrobbles for sbansal21.
Retrieved scrobble history.... 47 of 47 for iuyag.
Programmatically cleans up a user's scrobble history in the database by renaming "variant" album names. This essentially means if there are multiple variants of the same album name (e.g. The Sun's Tirade and The Sun's Tirade (Explicit)), those variants will all be standardized to the album's original name (in this example, The Sun's Tirade). Optionally applies a SQL script.
Usage: python scrubber.py <last.fm username> [SQL script]
~$ python scrubber.py sbansal21
Renamed variant album names for sbansal21.
~$ python scrubber.py iuyag scrubscript.sql
Renamed variant album names.
Applied scrubscript.sql to the database.
Generates a series of graphs based on scrobble data. Currently supports pie charts and streamgraphs.
This requires first generating the database via the scraper tool, as outlined above. scrobble-pie is currently deprecated as a result of recent schema changes but will be made functional again soon.
- Enter the
scrobble-piedirectory and process some data:
~$ cd scrobble-pie
Usage: node db.js <percentage limit> <artist limit>
~$ node db.js 0.4 25
percentage limit puts an upper bound on how much of the chart can be taken up by the "Other" category and artist limit sets a max number of artists that can be displayed on the chart.
- Within the main directory, set up an HTTP server—this is pretty simple with Python. The commands differ for Python 2.X and 3.X. The version can be easily checked:
~$ cd ..
~$ python --version
For Python 2.X:
~$ python -m SimpleHTTPServer 8000
For Python 3.X:
~$ python -m http.server 8000
From here, go to localhost:8000/scrobble-pie/index.html on any browser to view the streamgraph.
- Run the
spelunkerscript, which packages up the scraper, scrubber, and actual streamgraph generation.
Usage: python spelunker.py <username> <minimum scrobbles>
~$ python spelunker.py sbansal21 75
minimum scrobbles just means the minimum number of scrobbles for an artist in any month to avoid cluttering the graph with random one-time scrobbles—25 can be a good number to start off with but larger minimums can reduce both noise and lag on the graph.
- Set up an HTTP server—this is pretty simple with Python. The commands differ for Python 2.X and 3.X. The version can be easily checked:
~$ python --version
For Python 2.X:
~$ python -m SimpleHTTPServer 8000
For Python 3.X:
~$ python -m http.server 8000
From here, go to localhost:8000/scrobble-streamgraph/index.html on any browser to view the streamgraph.
- pie chart in pure D3.js and bubble charts
- hierarchical edge bundling graphs for related artists
- bullet charts for percentile visualizations
- classic last.fm-style hierarchical bar charts
- historical pattern detection based on probability distribution outliers
- better opts and a full command-line spelunker client
- move to a full Node.js or Flask/Django application with an actual routing system instead of just viewing files
- evolve basic graph pages to full webpages with real CSS