Need way to add resource jars at runtime to server.

[Brian-P-Burns]

Currently the server jar is an uber jar with dependencies such as twitter, kafka, object store, etc. We need a way to add these types of jars to the server environment at runtime. We can't continue to include everything that users will need. The more we include the bigger the risk for runtime conflicts with other spark dependencies.

[Brian-P-Burns]

This is fairly simple to do with toree. We can use the AddJar magic. The issue is how do make an api on the client side that makes sense to a nodejs user. We are exposing something that is server specific and Java specific to the nodejs client.

[doronrosenberg]

I would call it addSparkJar(), make it clearer its a Spark thing, and hey Spark is java. We make it return a promise and also allow you to pass in an array of jars.

[doronrosenberg]

I also figure we add this method to SparkContext and SparkSession

[Brian-P-Burns]

In the dev branch on the client side we now have SparkContext::addSparkJar. On the server side we have Utils::addJar. We just need to ensure the new jar is added to the SparkContext to ensure it's downloadable from the executors.

Note Toree's AddJar magic doesn't suit our needs. When EclairJS is running as a toree magic we are on a different classloader than the scala interpreter. The AddJar magic will only make the resource visible to the scala interpreter not eclairjs. We therefore add the resource to spark's MutableURLClassLoader which is the class loader responsible for loading the spark driver.