Skip to content

touma-I/spark-kubernetes

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Deploying Spark on Kubernetes

Want to learn how to build this?

Check out the post.

Want to use this project?

Minikube Setup

Install and run Minikube:

  1. Install a Hypervisor (like VirtualBox or HyperKit) to manage virtual machines
  2. Install and Set Up kubectl to deploy and manage apps on Kubernetes
  3. Install Minikube

Start the cluster:

$ minikube start --memory 8192 --cpus 4
$ minikube dashboard

Build the Docker image:

$ eval $(minikube docker-env)
$ docker build -t spark-hadoop:2.2.1 -f ./docker/Dockerfile ./docker

Create the deployments and services:

$ kubectl create -f ./kubernetes/spark-master-deployment.yaml
$ kubectl create -f ./kubernetes/spark-master-service.yaml
$ kubectl create -f ./kubernetes/spark-worker-deployment.yaml
$ minikube addons enable ingress
$ kubectl apply -f ./kubernetes/minikube-ingress.yaml

Add an entry to /etc/hosts:

$ echo "$(minikube ip) spark-kubernetes" | sudo tee -a /etc/hosts

Test it out in the browser at http://spark-kubernetes/.

About

spark on kubernetes

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Shell 52.8%
  • Dockerfile 47.2%