In order to solve the problem of inter-job dependencies. We need many VCJobs to cooperate each other and orchestrate them manually or by another Job Orchestration Platform to get the job done finally.We present an new way of orchestrating VCJobs called JobFlow. We proposed two concepts to running multiple batch jobs automatically named JobTemplate and JobFlow so end users can easily declare their jobs and run them using complex controlling primitives, for example, sequential or parallel executing, if-then-else statement, switch-case statement, loop executing and so on.
JobFlow helps migrating AI, BigData, HPC workloads to the cloud-native world. Though there are already some workload flow engines, they are not designed for batch job workloads. Those jobs typically have a complex running dependencies and take long time to run, for example days or weeks. JobFlow helps the end users to declare their jobs as an jobTemplate and then reuse them accordingly. Also, JobFlow orchestrating those jobs using complex controlling primitives and launch those jobs automatically. This can significantly reduce the time consumption of an complex job and improve resource utilization. Finally, JobFlow is not an generally purposed workflow engine, it knows the details of VCJobs. End user can have a better understanding of their jobs, for example, job's running state, beginning and ending timestamps, the next jobs to run, pod-failure-ratio and so on.
- Define the API of JobFlow
- Define the behaviour of JobFlow
- Start sequence between multiple jobs
- Dependency completion state of the job start sequence
- DAG-based job dependency startup
- Supports other job
- Achieve vcjobs level gang
- Some jobs need to depend on the completion of the previous job or other status when running, etc. Otherwise, the correct result cannot be calculated.
- Sometimes inter-job dependencies also require diverse dependency types, such as conditional dependencies, circular dependencies, probes, and so on.
The blue part is the component of k8s itself, the green and brown are the components of volcano, and the yellow is the crd resource of volcano.
jobflow job submission complete process:
-
After passing the Admission. kubectl will create JobTemplate and JobFlow (Volcano CRD) objects in kube-apiserver.
-
The JobFlowController uses the JobTemplate as a template according to the configuration of the JobFlow, and creates the corresponding VcJob according to the flow dependency rules.
-
After VcJob is created, VcJobController creates corresponding Pods and podgroups according to the configuration of VcJob.
-
After Pod and PodGroup are created, vc-scheduler will go to kube-apiserver to get Pod/PodGroup and node information.
-
After obtaining the information, vc-scheduler will select the appropriate node for each Pod according to its configured scheduling policy.
-
After assigning nodes to Pods, kubelet will get the Pod's configuration from kube-apiserver and start the corresponding containers.
Create a JobFlow check
1、There cannot be a template with the same name in a JobFlow dependency
Such as: A->B->A->C A appears twice
2、Closed loops cannot occur in JobFlow
E.g:A -> B -> C
^ /
| /
< - D
Create a JobTemplte check (following the vcjob parameter specification)
E.g: job minAvailable must be greater than or equal to zero
job maxRetry must be greater than or equal to zero
tasks cannot be empty, and cannot have tasks with the same name
The number of task replicas cannot be less than zero
task minAvailable cannot be greater than task replicas...
JobFlow defines the running flow of a set of jobs. Fields in JobFlow define how jobs are orchestrated.
JobFlow is abbreviated as jf, and the resource can be viewed through kubectl get jf
JobFlow aims to realize job-dependent operation between vcjobs in volcano. According to the dependency between vcjob, vcjob is issued.
The top-level attributes of a jobflow define its apiVersion, kind, metadata and spec.
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
apiVersion |
string |
Y | flow.volcano.sh/v1alpha1 |
A string that identifies the version of the schema the object should have. The core types uses flow.volcano.sh/v1alpha1 in this version of documentation. |
kind |
string |
Y | JobFlow |
Must be JobFlow |
metadata |
Metadata |
Y | Information about the JobFlow resource. | |
spec |
Spec |
Y | A specification for the JobFlow resource attributes. | |
status |
Status |
Y | A specification for the JobFlow status attributes. |
Metadata provides basic information about the JobFlow.
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
name |
string |
Y | A name for the schematic. name is subject to the restrictions listed beneath this table. |
|
namespace |
string |
Y | A namespace for the schematic. namespace is subject to the restrictions listed beneath this table. |
|
labels |
map[string]string |
N | A set of string key/value pairs used as arbitrary labels on this component. Labels follow the Kubernetes specification. | |
annotations |
map[string]string |
N | A set of string key/value pairs used as arbitrary descriptive text associated with this object. Annotations follows the Kubernetes specification. |
The specification of cloud-native services defines service metadata, version list, service capabilities and plugins.
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
flows |
Flow array |
Y | Describes the dependencies between vcjobs. | |
jobRetainPolicy |
string |
Y | retain | After JobFlow succeed, keep the generated job. Otherwise, delete it. |
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
name |
string |
Y | JobTemplate name | |
dependsOn |
DependsOn |
Y | JobTemplate dependencies | |
patch |
Patch |
N | Patch JobTemplate |
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
targets |
string array |
Y | All jobtemplate names that JobTemplate depends on | |
probe |
Probe |
N | Probe Type Dependency | |
strategy |
string |
Y | all | Whether the dependencies need to be all satisfied |
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
spec |
spec |
Y | Patch the contents of the jobtemplate's spec |
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
httpGetList |
HttpGet array |
N | HttpGet type dependencies | |
tcpSocketList |
TcpSocket array |
N | TcpSocket type dependencies | |
taskStatusList |
TaskStatus array |
N | TaskStatus type dependencies |
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
TaskName |
string |
Y | The name of the task under vcjob | |
Path |
Probe |
Y | The path of httpget | |
Port |
int |
Y | The port of httpget | |
httpHeader |
HTTPHeader |
N | The httpHeader of httpget |
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
TaskName |
string |
Y | The name of the task under vcjob | |
Port |
int |
Y | The port of TcpSocket |
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
TaskName |
string |
Y | The name of the task under vcjob | |
Phase |
string |
Y | The phase of task |
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
pendingJobs |
string array |
N | Vcjobs in pending state | |
runningJobs |
string array |
N | Vcjobs in running state | |
failedJobs |
string array |
N | Vcjobs in failed state | |
completedJobs |
string array |
N | Vcjobs in completed and completing state | |
terminatedJobs |
string array |
N | Vcjobs in terminated and terminating state | |
unKnowJobs |
string array |
N | Vcjobs in pending state | |
jobStatusList |
JobStatus array |
N | Status information of all split vcjobs | |
conditions |
map[string]Condition |
N | It is used to describe the current state, creation time, completion time and information of all vcjobs. The vcjob state here additionally adds the waiting state to describe the vcjob whose dependencies do not meet the requirements. | |
state |
State |
N | State of JobFlow |
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
name |
string |
N | Name of vcjob | |
state |
string |
N | State of vcJob | |
startTimestamp |
Time |
N | StartTimestamp of vcjob | |
endTimestamp |
Time |
N | EndTimestamp of vcjob | |
restartCount |
int32 |
N | RestartCount of vcjob | |
runningHistories |
JobRunningHistory array |
N | Historical information of various states of vcjob |
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
phase |
string |
N | phase of vcjob | |
createTime |
Time |
N | CreateTime of vcjob | |
runningDuration |
Duration |
N | RunningDuration of vcjob | |
taskStatusCount |
map[string]TaskState |
N | The number of tasks in different states |
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
phase |
string |
N | Succeed: All vcjobs have reached completed state。 Terminating: Jobflow is deleting。 Failed: A vcjob in the flow is in the failed state, so the vcjob in the flow cannot continue to be delivered。 Running: Flow contains vcjob in Running state。 Pending: The flow contains no vcjob in the Running state。 |
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
startTimestamp |
Time |
N | The start time of a certain state of the vcjob | |
endTimestamp |
Time |
N | The end time of a certain state of the vcjob | |
state |
string |
N | Vcjob status |
JobFlow supports the functionality of the JobTemplate patch. The example in JobFlow is as follows:
apiVersion: flow.volcano.sh/v1alpha1
kind: JobFlow
metadata:
name: test
namespace: default
spec:
jobRetainPolicy: delete
flows:
- name: a
patch:
spec:
tasks:
- name: "default-nginx"
template:
spec:
containers:
- name: nginx
command:
- sh
- -c
- sleep 10s
Here is an example of jobflow:
- JobTemplate is the template of vcjob, after JobTemplate is created, it will not be processed by vc-controller like vcjob, it will wait to be referenced by JobFlow.
- JobFlow can reference multiple jobtemplates
- A jobtemplate can be referenced by multiple jobflows
- JobTemplate can be converted to and from vcjob.
- Jobtemplate is abbreviated as jt, and the resource can be viewed through kubectl get jt
- The difference between jobtemplate and vcjob is that jobtemplate will not be issued by the job controller, and jobflow can directly reference the name of the JobTemplate to implement the issuance of vcjob.
- JobFlow supports making changes to jobtemplate when referencing jobtemplate
The top-level attributes of a jobtemplate define its apiVersion, kind, metadata and spec.
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
apiVersion |
string |
Y | flow.volcano.sh/v1alpha1 |
A string that identifies the version of the schema the object should have. The core types uses flow.volcano.sh/v1alpha1 in this version of documentation. |
kind |
string |
Y | JobTemplate |
Must be JobTemplate |
metadata |
Metadata |
Y | Information about the JobTemplate resource. | |
spec |
Spec |
Y | A specification for the JobTemplate resource attributes. | |
status |
[Status ](# JobTemplateStatus) |
Y | A specification for the JobTemplate status attributes. |
Metadata provides basic information about the JobTemplate.
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
name |
string |
Y | A name for the schematic. name is subject to the restrictions listed beneath this table. |
|
namespace |
string |
Y | A namespace for the schematic. namespace is subject to the restrictions listed beneath this table. |
|
labels |
map[string]string |
N | A set of string key/value pairs used as arbitrary labels on this component. Labels follow the Kubernetes specification. | |
annotations |
map[string]string |
N | A set of string key/value pairs used as arbitrary descriptive text associated with this object. Annotations follows the Kubernetes specification. |
The spec of jobtemplate directly follows the spec of vcjob.
Attribute | Type | Required | Default Value | Description |
---|---|---|---|---|
jobDependsOnList |
string array |
Y | Vcjobs created by this jobtemplate as a template. |
You can view the sample file of JobTemplate
https://www.bilibili.com/video/BV1c44y1Y7FX
- Create the jobTemplate that needs to be used
- Create a jobflow. The flow field of the jobflow is filled with the corresponding jobtemplate used to create a vcjob.
- The field jobRetainPolicy indicates whether to delete the vcjob created by the jobflow after the jobflow succeeds. (delete/retain) default is retain.
- Create JobFlow and JobTemplate CRD
- Support sequential start of vcjob
- Support vcjob to depend on other vcjobs to start
- Support the conversion of vcjob and JobTemplate to each other
- Supports viewing of the running status of JobFlow
- JobFlow supports making changes to jobtemplate when referencing jobtemplate
if
statementsswitch
statementsfor
statements- Support job failure retry in JobFlow
- Integration with volcano-scheduler
- Support for scheduling plugins at JobFlow level