Play Spark Module

This is a plugin for Play 2.4, enabling support for Apache Spark. The goal is to simply Spark Context management and job submission.

Some considerations:

• This project is under active development and the APIs might change in the future.
• It wasn't tested on a production environment (yet!)
• No remote spark job submissions tested (yet)

For contribution please check CONTRIBUTING.md

Requirements

• JRE 1.8

Build manually

To build from source run

sbt publish-local

To run all tests run:

sbt test

API

The API is divided into two categories/groups:

• Context API: for managing spark contexts and configuration
• Job API: for job submission

trait PlaySparkApi {

  // Context API
  def saveSparkContext(conf: SparkConf)(implicit timeout: Timeout): Future[Try[String]]
  def startContext(contextId: String)(implicit timeout: Timeout): Future[Try[Unit]]
  def getContextsConfig()(implicit timeout: Timeout): Future[List[SparkConf]]
  def deleteContext(contextId: String)(implicit timeout: Timeout): Future[Try[Unit]]
  def stopContext()(implicit timeout: Timeout): Future[Try[Unit]]

  // Job API
  def startJob(job: SparkJob, contextId: String)(implicit timeout: Timeout): Future[SparkJobResult]
  def startSyncJob(job: SparkJob, contextId: String)(implicit timeout: Timeout): SparkJobResult
}

Before calling any method from the Job API, you must create a spark context. This methods creates a spark context using a configuration you chose. Remember that you only can have one Spark Context running per JVM. You can do so by calling:

def saveSparkContext(conf: SparkConf)(implicit timeout: Timeout): Future[Try[String]]

For more detailed API information please check the Wiki.

Additional Documentation

To generate Scaladoc documentation (on folder doc/) just run:

  sbt doc