JobProxy

JobProxy is (should be) a proxy between an application that needs access to compute resources like starting a docker container or just starting a simple shell script and the various existing resource providing frameworks.

Motivation

While developing an application that needs access to compute resources from a running Mesos, we were searching for a Mesos framework that fullfills our constraints (which were very simple at this point). However we found that there are exists more than one framework that can do the job, but which is right/best one for our application ? A proxy as abstraction layer between our application and the frameworks seems to be very helpful. The decision which framework to be used is then indepened from writting the application ...

Requirements:

• Java 8

User Guide

Installation:

Download the latest jar with the following command:

wget https://jitpack.io/com/github/BiBiServ/jobproxy/JobProxyServerCLI/RELEASE/JobProxyServerCLI-RELEASE.jar

where

• RELEASE is an identifier you can find on release page i.e: 0.1.0.alpha.2.

Usage:

See JobProxyServerCLI for usage example.

List of implementing Frameworks:

• Mesos Chronos (https://mesos.github.io/chronos/) version 2.4.0
• DRMAA (https://www.drmaa.org/) using DRMAA V1.0 API
• JavaDocker (https://github.com/spotify/docker-client)
• Kubernetes (https://kubernetes.io)

Developer Guide

Development-Guidelines

https://github.com/BiBiServ/Development-Guidelines

Release/Development Branch Workflow

There are two branches one is the master branch with latest working version of JobProxy and the other one is the development branch for the next release.

On a release, the development branch must be merged into the master branch and the list of implementing Frameworks must be extended.

Each module has its own version number e.g.: https://github.com/jkrue/jobproxy/blob/master/JobProxyServer/pom.xml#L8

If the code is updated, the corresponding module version number must be updated.

Model updates

If you update the classes in the JobProxyModel, please make sure to update the swagger specification, by running:

mvn swagger:generate

inside the JobProxyModel directory. This command will update the swagger.yaml inside the doc/REST directory. Then please update the documentation (see below How to extend our REST documentation?).

How to use jobProxy as a dependency?

Just go to this site and follow the instructions. Please keep in mind that you can import single subproject artifacts like JobProxyServer and not the whole JobProxy parent artifact.

How to build to build JobProxy?

1. Clone this project.

2. Compile this project with Maven.

mvn clean compile

How to package JobProxy?

Run the following command for packaging.

mvn package

You will find the jar inside the JobProxy/JobProxyServerCLI/target folder.

REST

We are using Swagger for our REST API. Our swagger.yaml can be found inside our doc directory altogether with our latest REST API documentation.

How to extend our REST documentation?

We are generating markup out of our swagger yaml by using the swagger2markup-cli tool. With the following command:

java -jar swagger2markup-cli-1.3.1.jar convert -c config.properties -i swagger.yaml -f api

where

• doc.md is the output document

• config.properties is the configuration file

How to build your own jobproxy client?

Just use the swagger code generator with our swagger.yaml to produce jobproxy client code in your favorite language.

The following call produces an java client from swagger specification:

swagger-codegen-cli.sh generate -i doc/REST/swagger.yaml -l java 

How to version the project?

We decided that all modules should have the same version as the parent module. By using the below command in the project root you can update all child modules at once.

mvn  versions:set -DnewVersion=<version>

where

• version = 2.1.0.alpha.2

Vagrant test environments

Vagrant is a handy tool which helps to setup "quasi"-real test environment for testing JobProxy's Modules.

Installing requirements for Vagrant environment

If you want to test JobProxy Modules you will need the following tools:

Tool	Installation command for ubuntu
VirtualBox	$ sudo apt-get install virtualbox virtualbox-dkms
Vagrant	You can find package for Ubuntu/Debian here
Vagrant Hostmanager	$ vagrant plugin install vagrant-hostmanager
GNU Parallel	$ sudo apt-get install parallel

Starting the environment

After all prerequisites are done you should be able to start the environment.

Important Note

since Vagrant is not able to start multiple machines in parallel due to instable behavior of VirtualBox (s. here ), the start via $ vagrant up takes long time ans is therefore not used.

To start the Vagrant environment navigate to the subfolder containing the Vagrant file and run

$ ./parallel_provision.sh

This script lets start all machines sequentally without configuration and performs configuration in parallel after all machines are boot up.

Example: JobProxyChronos

$ cd JobProxyChronos/testing
$ ./parallel_provision.sh

After the script is done you will get 1 Master node running ZooKepper and Chronos and 3 slave Mesos nodes. You'll also be able to access Mesos framework at http://10.0.0.2:5050 and Chronos Web GUI at http://10.0.0.2:8081