Skip to content

Latest commit

 

History

History
19 lines (15 loc) · 1.06 KB

README.md

File metadata and controls

19 lines (15 loc) · 1.06 KB
Raw

jupyterlab-livy-spark

Jupyterlab with spark kernel (integrated via livy and sparkmagic)

Pre-requisites

Docker and docker-compose should be installed

Build steps

Clone the repo, and run the build.sh file. This will create an image jupyterlab-with-spark:latest and will start the container. [Note: Image can also be directly downloaded by 'docker pull senchandra/jupyterlab-with-spark'. If using this, change the image in docker-compose.yml from jupyterlab-with-spark to senchandra/jupyterlab-with-spark]

List of programs running

  1. Spark standalone cluster manager (Master and slave). Spark UI at port 8080 and spark master at spark://127.0.1.1:7077
  2. Livy (Running on port 8889)
  3. Jupyterlab (Running on port 8888)

Running the Spark kernel on Notebook

  1. Open jupyterlab at localhost:8888. It does not require password or token.
  2. Select either of the kernel pyspark, sparkr or sparkshell and create a new notebook.
  3. Type 'spark' in the cell and run the cell in Notebook. The spark session gets created and gets registered in livy as well as Spark Clustermanager.